This section deals with Logstash, including an explanation of its internal partsat a high level. X-Pack monitoring for Logstash represents a total of two pieces:
These pieces are created when X-Pack monitoring for Logstash is enabled, and theylive outside of the default Logstash pipeline in a dedicated monitoringpipeline. This configuration means that all data and processing has a minimalimpact on ordinary Logstash processing. As a secondary benefit of existing in aseparate pipeline, existing Logstash features, such as theelasticsearch output, can be reused tobenefit from its retry policies.
The elasticsearch output that is used by X-Pack monitoring for Logstash isconfigured exclusively via settings found in logstash.yml. It is notconfigured by using anything from the Logstash configurations that might also beusing their own separate elasticsearch outputs.
The Elasticsearch cluster that is configured for use with X-Pack monitoring for Logstash isexpected to be the production cluster. This configuration enables the productionElasticsearch cluster to add metadata (for example, its cluster UUID) to the Logstashmonitoring data then route it to the monitoring clusters. For more informationabout typical monitoring architectures, seeHow Monitoring Works.
Collectors, as their name implies, collect things. In X-Pack monitoring for Logstash,collectors are just Inputs in the same way that ordinary Logstashconfigurations provide inputs.
Like X-Pack monitoring for Elasticsearch, each collector can create zero or more monitoringdocuments. As it is currently implemented, each Logstash node runs two types ofcollectors: one for node stats and one for pipeline stats.
| Collector | Data Types | Description |
|---|---|---|
Node Stats |
|
Gathers details about the running node, such as memory utilization and CPUusage (for example, |
Pipeline Stats |
|
Gathers details about the node’s running pipelines, which powers theMonitoring Pipeline UI. |
Per collection interval, which defaults to 10 seconds (10s), each collector isrun. The failure of an individual collector does not impact any other collector.Each collector, as an ordinary Logstash input, creates a separate Logstash eventin its isolated monitoring pipeline. The Logstash output then sends the data.
The collection interval can be configured dynamically and you can also disabledata collection. For more information about the configuration options for thecollectors, see Monitoring Settings.
Unlike X-Pack monitoring for Elasticsearch and Kibana, there is noxpack.monitoring.collection.enabled setting on Logstash. You must use thexpack.monitoring.enabled setting to enable and disable data collection.
If gaps exist in the monitoring charts in Kibana, it is typically because eithera collector failed or the monitoring cluster did not receive the data (forexample, it was being restarted). In the event that a collector fails, a loggederror should exist on the node that attempted to perform the collection.
Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline isto send events to outputs. In the case of X-Pack monitoring for Logstash, the outputis always an elasticsearch output. However, unlike ordinary Logstash pipelines,the output is configured within the logstash.yml settings file via thexpack.monitoring.elasticsearch.* settings.
Other than its unique manner of configuration, this elasticsearch outputbehaves like all elasticsearch outputs, including its ability to pause datacollection when issues exist with the output.
It is critical that all Logstash nodes share the same setup.Otherwise, monitoring data might be routed in different ways or to different places.
If a Logstash node does not explicitly define an X-Pack monitoring output setting,the following default configuration is used:
xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]
All data produced by X-Pack monitoring for Logstash is indexed in the monitoringcluster by using the .monitoring-logstash template, which is managed by theexporters within Elasticsearch.
If you are working with a cluster that has X-Pack security enabled, extra steps arenecessary to properly configure Logstash. For more information, seeX-Pack monitoring.
When discussing security relative to the elasticsearch output, itis critical to remember that all users are managed on the production cluster,which is identified in the xpack.monitoring.elasticsearch.hosts setting.This is particularly important to remember when you move from developmentenvironments to production environments, where you often have dedicatedmonitoring clusters.
For more information about the configuration options for the output, seeMonitoring Settings.