![\"\"](\"https:\/\/geko2.factoryfy.com\/wp-content\/uploads\/elk-useragent.png\")
<\/p>\nElasticSearch service has become one of the preferred services where sysops department store logs and metrics information.<\/p>\n
Recently, we faced with a problem that we already solved related with the user_agent field we get from nginx json logs.<\/p>\n
When a new request transaction goes to our web services we want to get some certain relevant information about the agent who is trying to get that information.<\/p>\n
Example:<\/p>\n
<\/p>\n
In the past, elastic search comes with an specific plugin that generate those fields with all the agent information. And even the AWS elasticsearch services has come with this plugin.<\/p>\n
Usefull links:<\/p>\n
https:\/\/docs.amazonaws.cn\/en_us\/elasticsearch-service\/latest\/developerguide\/aes-supported-plugins.html<\/p>\n
https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/plugins\/master\/ingest-user-agent.html<\/p>\n
But now, the plugin has been replaced by a specific processor<\/strong> inside Elasticsearch engine.<\/p>\n <\/p>\n The quickest answer is just using a pipeline that calls our user agent processor inside the ingest controller using the next schema:<\/p>\n <\/p>\n In our case, we use fluentbit<\/strong> inside our customer web servers to collect the json file that nginx generates and then all the agents send that info by streaming to a fluentd agrgregator<\/strong> service where we had configured the OUTPUT plugins<\/strong> of several databases and storage collectors. In our case, elasticsearch.<\/p>\n Inside the Elasticsearch we create a new pipeline<\/strong> and just need to tell the name of the user agent field we get from json:<\/p>\n So now we have created the pipeline<\/strong> telling the field name of our nginx user agent<\/strong>. Now how we connect pipeline with the logs ingest from aggregator?<\/p>\n Easy, we just need to specify the pipeline ingest<\/strong> inside our tdagent<\/strong> configuration in the match rule like this<\/p>\n \/etc\/td-agent\/td-agent.conf:<\/strong><\/p>\n <\/p>\n As you can see we configure the ingest using pipeline attribute with the pipeline name we already created before, in our case nginx:<\/p>\n pipeline nginx<\/p>\n Just adding the pipeline with the user agent processor, now our ingest will add for every row in our elasticsearch database the user agent information in the fly.<\/p>\n <\/p>\n Advice!: After few weeks of test in real production environment with the pipeline method, we can say that this processor needs too much CPU to process in real time and build the transformations. So we decided to configure a filter inside fluentd to use ua plugin in aggregator layer and it works much better now!<\/span><\/p>\nThen, how we can make it work with the logs ingest?<\/strong><\/h3>\n
![\"\"](\"https:\/\/geko2.factoryfy.com\/wp-content\/uploads\/fluent-elk.png\")
<\/p>\nPUT<\/span> _ingest<\/span>\/<\/span>pipeline<\/span>\/<\/span>nginx<\/span>\r\n{<\/span>\r\n \"description\"<\/span> :<\/span> \"Add user agent info from nginx user agent\"<\/span>,<\/span>\r\n \"processors\"<\/span> :<\/span> [<\/span>\r\n {<\/span>\r\n \"user_agent\"<\/span> :<\/span> {<\/span>\r\n \"field\"<\/span> :<\/span> \"nginx-agent-field\"<\/span>\r\n }<\/span>\r\n }<\/span>\r\n ]<\/span>\r\n}<\/span><\/pre>\n @type \"elasticsearch\"\r\n host elasticsearchtest.eu-west-1.es.amazonaws.com\r\n port 443\r\n scheme https\r\n ssl_verify true\r\n ssl_version TLSv1_2\r\n type_name \"access_log\"\r\n logstash_prefix \"nginx\"\r\n logstash_format true\r\n include_tag_key true\r\n pipeline nginx\r\n tag_key \"@log_name\"\r\n flush_interval 10s\r\n\r\n chunk_limit_size 4MB\r\n total_limit_size 512MB\r\n flush_interval 30s\r\n flush_thread_count 2\r\n queue_limit_length 15\r\n retry_max_interval 30\r\n retry_forever true\r\n\r\n<\/pre>\n