Use the pipeline viewer to visualize and monitor the behavior of complexLogstash pipeline configurations. You can see and interact with a tree viewthat illustrates the pipeline topology, data flow, and branching logic.

The pipeline viewer highlights CPU% and event latency in cases where the valuesare anomalous. This information helps you quickly identify processing that isdisproportionately slow.

Prerequisites

Before using the pipeline viewer:

Logstash begins shipping metrics to the monitoring cluster.

View the pipeline

To view the pipeline:

Each pipeline is identified by a pipeline ID (main by default). For eachpipeline, you see the pipeline’s throughput and the numberof nodes on which the pipeline is running during the selected time range.

Many elements in the tree are clickable.For example, you can click the plugin name to expand the detail view.

Click the arrow beside a branch name to collapse or expand it.

Notes and best practices

Use semantic IDs.Specify semantic IDs when you configure the stages in your Logstash pipeline.Otherwise, Logstash generates them for you. Semantic IDs help you identifyconfigurations that are causing bottlenecks. For example, you may have severalgrok filters running in your pipeline. If you have specified semantic IDs, youcan tell at a glance which filters are slow. Semantic IDs, such asapacheParsingGrok and cloudwatchGrok, point you to the grok filters that arecausing bottlenecks.

Outliers.Values and stats that are anomalously slow or otherwise out of line are highlighted.This doesn’t necessarily indicate a problem, but it highlights potentialbottle necks so that you can find them quickly.

Some plugins are slower than others due to the nature of the work they do. Forinstance, you may find that a grok filter that uses a complicated regexp runs alot slower than a mutate filter that simply adds a field. The grok filter mightbe highlighted in this case, though it may not be possible to further optimizeits work.

Versioning.Version information is available from the dropdown list beside the pipeline ID.Logstash generates a new version each time you modify a pipeline, andstores multiple versions of the pipeline stats. Use this information to see howchanges over time affect throughput and other metrics. Logstash does not storemultiple versions of the pipeline configurations.