Converting Ingest Node Pipelines

After implementing ingest pipelines to parse your data, youmight decide that you want to take advantage of the richer transformationcapabilities in Logstash. For example, you may need to use Logstash instead ofingest pipelines if you want to:

  • Ingest from more inputs. Logstash can natively ingest data from many othersources like TCP, UDP, syslog, and relational databases.
  • Use multiple outputs. Ingest node was designed to only support Elasticsearchas an output, but you may want to use more than one output. For example, you maywant to archive your incoming data to S3 as well as indexing it inElasticsearch.
  • Take advantage of the richer transformation capabilities in Logstash, such asexternal lookups.
  • Use the persistent queue feature to handle spikes when ingesting data (fromBeats and other sources).

To make it easier for you to migrate your configurations, Logstash provides aningest pipeline conversion tool. The conversion tool takes the ingest pipelinedefinition as input and, when possible, creates the equivalent Logstashconfiguration as output.

See Limitations for a full list of tool limitations.

Running the tool

You’ll find the conversion tool in the bin directory of your Logstashinstallation. See Logstash Directory Layout to find the location of bin on your system.

To run the conversion tool, use the following command:

bin/ingest-convert.sh --input INPUT_FILE_URI --output OUTPUT_FILE_URI [--append-stdio]

Where:

  • INPUT_FILE_URI is a file URI that specifies the full path to the JSON filethat defines the ingest node pipeline.
  • OUTPUT_FILE_URI is the file URI of the Logstash DSL file that will begenerated by the tool.
  • --append-stdio is an optional flag that adds stdin and stdout sections tothe config instead of adding the default Elasticsearch output.

This command expects a file URI, so make sure you use forward slashes andspecify the full path to the file.

For example:

bin/ingest-convert.sh --input file:///tmp/ingest/apache.json --output file:///tmp/ingest/apache.conf

Limitations

  • Painless script conversion is not supported.
  • Only a subset of available processors aresupported for conversion. Forprocessors that are not supported, the tool produces a warning and continueswith a best-effort conversion.

Supported Processors

The following ingest node processors are currently supported for conversion bythe tool:

  • Append
  • Convert
  • Date
  • GeoIP
  • Grok
  • Gsub
  • Json
  • Lowercase
  • Rename
  • Set