« Centralized Pipeline Management Using Elastic Cloud »

Working with Logstash Modules

Logstash modules provide a quick, end-to-end solution for ingesting data andvisualizing it with purpose-built dashboards.

These modules are available:

Each module comes pre-packaged with Logstash configurations, Kibana dashboards,and other meta files that make it easier for you to set up the Elastic Stack forspecific use cases or data sources.

You can think of modules as providing three essential functions that make iteasier for you to get started. When you run a module, it will:

Create the Elasticsearch index.
Set up the Kibana dashboards, including the index pattern, searches, andvisualizations required to visualize your data in Kibana.
Run the Logstash pipeline with the configurations required to read and parsethe data.

Running modules

To run a module and set up dashboards, you specify the following options:

bin/logstash --modules MODULE_NAME --setup [-M "CONFIG_SETTING=VALUE"]

Where:

--modules runs the Logstash module specified by MODULE_NAME.
-M "CONFIG_SETTING=VALUE" is optional and overrides the specifiedconfiguration setting. You can specify multiple overrides. Each override muststart with -M. See Specify module settings at the command line for more info.
--setup creates an index pattern in Elasticsearch and imports Kibanadashboards and visualizations. Running --setup is a one-time setup step. Omitthis option for subsequent runs of the module to avoid overwriting existingKibana dashboards.

For example, the following command runs the Netflow module with the defaultsettings, and sets up the netflow index pattern and dashboards:

bin/logstash --modules netflow --setup

The following command runs the Netflow module and overrides the Elasticsearchhost setting. Here it’s assumed that you’ve already run the setup step.

bin/logstash --modules netflow -M "netflow.var.elasticsearch.host=es.mycloud.com"

Configuring modules

To configure a module, you can eitherspecify configuration settings in thelogstash.yml settings file, or use command-line overrides tospecify settings at the command line.

Specify module settings in `logstash.yml`

To specify module settings in the logstash.ymlsettings file file, you add a module definition tothe modules array. Each module definition begins with a dash (-) and is followedby name: module_name then a series of name/value pairs that specify modulesettings. For example:

modules:- name: netflow  var.elasticsearch.hosts: "es.mycloud.com"  var.elasticsearch.username: "foo"  var.elasticsearch.password: "password"  var.kibana.host: "kb.mycloud.com"  var.kibana.username: "foo"  var.kibana.password: "password"  var.input.tcp.port: 5606

For a list of available module settings, see the documentation for the module.

Specify module settings at the command line

You can override module settings by specifying one or more configurationoverrides when you start Logstash. To specify an override, you use the -Mcommand line option:

-M MODULE_NAME.var.PLUGINTYPE1.PLUGINNAME1.KEY1=VALUE

Notice that the fully-qualified setting name includes the module name.

You can specify multiple overrides. Each override must start with -M.

The following command runs the Netflow module and overrides both theElasticsearch host setting and the udp.port setting:

bin/logstash --modules netflow -M "netflow.var.input.udp.port=3555" -M "netflow.var.elasticsearch.hosts=my-es-cloud"

Any settings defined in the command line are ephemeral and will not persist acrosssubsequent runs of Logstash. If you want to persist a configuration, you need toset it in the logstash.yml settings file.