The Logstash event processing pipeline has three stages: inputs → filters →outputs. Inputs generate events, filters modify them, and outputs ship themelsewhere. Inputs and outputs support codecs that enable you to encode or decodethe data as it enters or exits the pipeline without having to use a separatefilter.
Inputs
You use inputs to get data into Logstash. Some of the more commonly-used inputsare:
- file: reads from a file on the filesystem, much like the UNIX command
tail -0F
- syslog: listens on the well-known port 514 for syslog messages and parsesaccording to the RFC3164 format
- redis: reads from a redis server, using both redis channels and redis lists.Redis is often used as a "broker" in a centralized Logstash installation, whichqueues Logstash events from remote Logstash "shippers".
- beats: processes events sent by Beats.
For more information about the available inputs, seeInput Plugins.
Filters
Filters are intermediary processing devices in the Logstash pipeline. You cancombine filters with conditionals to perform an action on an event if it meetscertain criteria. Some useful filters include:
- grok: parse and structure arbitrary text. Grok is currently the best way inLogstash to parse unstructured log data into something structured and queryable.With 120 patterns built-in to Logstash, it’s more than likely you’ll find onethat meets your needs!
- mutate: perform general transformations on event fields. You can rename,remove, replace, and modify fields in your events.
- drop: drop an event completely, for example, debug events.
- clone: make a copy of an event, possibly adding or removing fields.
- geoip: add information about geographical location of IP addresses (alsodisplays amazing charts in Kibana!)
For more information about the available filters, seeFilter Plugins.
Outputs
Outputs are the final phase of the Logstash pipeline. An event can pass throughmultiple outputs, but once all output processing is complete, the event hasfinished its execution. Some commonly used outputs include:
- elasticsearch: send event data to Elasticsearch. If you’re planning to saveyour data in an efficient, convenient, and easily queryable format…Elasticsearch is the way to go. Period. Yes, we’re biased :)
- file: write event data to a file on disk.
- graphite: send event data to graphite, a popular open source tool forstoring and graphing metrics. http://graphite.readthedocs.io/en/latest/
- statsd: send event data to statsd, a service that "listens for statistics,like counters and timers, sent over UDP and sends aggregates to one or morepluggable backend services". If you’re already using statsd, this could beuseful for you!
For more information about the available outputs, seeOutput Plugins.
Codecs
Codecs are basically stream filters that can operate as part of an input oroutput. Codecs enable you to easily separate the transport of your messages fromthe serialization process. Popular codecs include json, msgpack, and plain(text).
- json: encode or decode data in the JSON format.
- multiline: merge multiple-line text events such as java exception andstacktrace messages into a single event.
For more information about the available codecs, seeCodec Plugins.