When you use Filebeat modules with Logstash, you can use the ingest pipelinesprovided by Filebeat to parse the data. You need to load the pipelinesinto Elasticsearch and configure Logstash to use them.

To load the ingest pipelines:

On the system where Filebeat is installed, run the setup command with the--pipelines option specified to load ingest pipelines for specific modules.For example, the following command loads ingest pipelines for the system andnginx modules:

filebeat setup --pipelines --modules nginx,system

A connection to Elasticsearch is required for this setup step because Filebeat needs toload the ingest pipelines into Elasticsearch. If necessary, you can temporarily disableyour configured output and enable the Elasticsearch output before running the command.

To configure Logstash to use the pipelines:

On the system where Logstash is installed, create a Logstash pipeline configurationthat reads from a Logstash input, such as Beats or Kafka, and sends events to anElasticsearch output. Set the pipeline option in the Elasticsearch output to%{[@metadata][pipeline]} to use the ingest pipelines that you loadedpreviously.

Here’s an example configuration that reads data from the Beats input and usesFilebeat ingest pipelines to parse data collected by modules:

input {  beats {    port => 5044  }}output {  if [@metadata][pipeline] {    elasticsearch {      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"      manage_template => false      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"      pipeline => "%{[@metadata][pipeline]}"       user => "elastic"      password => "secret"    }  } else {    elasticsearch {      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"      manage_template => false      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"      user => "elastic"      password => "secret"    }  }}

See the Filebeat Modulesdocumentation for more information about setting up and running modules.

For a full example, see Example: Set up Filebeat modules to work with Kafka and Logstash.