For other versions, see theVersioned plugin docs.
For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github.For the list of Elastic supported plugins, please consult the Elastic Support Matrix.
Starting with Elasticsearch 5.3, there’s an HTTP settingcalled http.content_type.required
. If this option is set to true
, and youare using Logstash 2.4 through 5.2, you need to update the Elasticsearch outputplugin to version 6.2.5 or higher.
If you plan to use the Kibana webinterface, use the Elasticsearch output plugin to get your log data intoElasticsearch.
You can run Elasticsearch on your own hardware, or use ourhosted Elasticsearch Service onElastic Cloud. The Elasticsearch Service is available on both AWS and GCP.Try the Elasticsearch Servicefor free.
This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0.We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower,yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without havingto upgrade Logstash in lock-step.
You can learn more about Elasticsearch at https://www.elastic.co/products/elasticsearch
Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch’s mapping changes in version 5.0.Most importantly, the subfield for string multi-fields has changed from .raw
to .keyword
to match ES defaultbehavior.
Users installing ES 5.x and LS 5.x
This change will not affect you and you will continue to use the ES defaults.
Users upgrading from LS 2.x to LS 5.x with ES 5.x
LS will not force upgrade the template, if logstash
template already exists. This means you will still use.raw
for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data afterthe new template is installed.
The retry policy has changed significantly in the 8.1.1 release.This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experienceeither partial or total failures. The bulk API sends batches of requests to an HTTP endpoint. Error codes for the HTTPrequest are handled differently than error codes for individual documents.
HTTP requests to the bulk API are expected to return a 200 response code. All other response codes are retried indefinitely.
The following document errors are handled as follows:
Note that 409 exceptions are no longer retried. Please set a higher retry_on_conflict
value if you experience 409 exceptions.It is more performant for Elasticsearch to retry these exceptions than this plugin.
Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunatelymapping errors cannot be handled without human intervention and without lookingat the field that caused the mapping mismatch. If the DLQ is enabled, theoriginal events causing the mapping errors are stored in a file that can beprocessed at a later time. Often times, the offending field can be removed andre-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping errorhappens, the problem is logged as a warning, and the event is dropped. SeeDead Letter Queues for more information about processing events in the DLQ.
The Index Lifecycle Management feature requires plugin version 9.3.1
or higher.
This feature requires an Elasticsearch instance of 6.6.0 or higher with at least a Basic license
Logstash can use Index Lifecycle Management to automate the management of indices over time.
The use of Index Lifecycle Management is controlled by the ilm_enabled
setting. By default, this is willautomatically detect whether the Elasticsearch instance supports ILM, and will use it if it is available. ilm_enabled
can also be set to true
or false
to override the automatic detection, or disable ILM.
This will overwrite the index settings and adjust the Logstash template to write the necessary settings for the templateto support index lifecycle management, including the index policy and rollover alias to be used.
Logstash will create a rollover alias for the indices to be written to, including a pattern for how the actual indices will be named, and unless an ILM policy that already exists has been specified,a default policy will also be created. The default policy is configured to rollover an index when it reaches either 50 gigabytes in size, or is 30 days old, whichever happens first.
The default rollover alias is called logstash
, with a default pattern for the rollover index of {now/d}-00001
,which will name indices on the date that the index is rolled over, followed by an incrementing number. Note that the pattern must end with a dash and a number that will be incremented.
See the Rollover API documentation for more details on naming.
The rollover alias, ilm pattern and policy can be modified.
See config below for an example:
output { elasticsearch { ilm_rollover_alias: "custom" ilm_pattern: "000001" ilm_policy: "custom_policy" }}
Custom ILM policies must already exist on the Elasticsearch cluster before they can be used.
If the rollover alias or pattern is modified, the index template will need to be overwritten as the settings index.lifecycle.name
and index.lifecycle.rollover_alias
are automatically written to the template
If the index property is supplied in the output definition, it will be overwritten by the rollover alias.
This plugin attempts to send batches of events as a single request. However, ifa request exceeds 20MB we will break it up into multiple batch requests. If a single document exceeds 20MB it will be sent as a single request.
This plugin uses the JVM to lookup DNS entries and is subject to the value of networkaddress.cache.ttl,a global setting for the JVM.
As an example, to set your DNS TTL to 1 second you would setthe LS_JAVA_OPTS
environment variable to -Dnetworkaddress.cache.ttl=1
.
Keep in mind that a connection with keepalive enabled willnot reevaluate its DNS value while the keepalive is in effect.
This plugin supports request and response compression. Response compression is enabled by default andfor Elasticsearch versions 5.0 and later, the user doesn’t have to set any configs in Elasticsearch forit to send back compressed response. For versions before 5.0, http.compression
must be set to true
inElasticsearch to take advantage of response compression when using this plugin
For requests compression, regardless of the Elasticsearch version, users have to enable http_compression
setting in their Logstash config file.
This plugin supports the following configuration options plus the Common Options described later.
Setting | Input type | Required |
---|---|---|
No |
||
No |
||
a valid filesystem path |
No |
|
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
string, one of |
No |
|
No |
||
No |
||
No |
||
No |
||
a valid filesystem path |
No |
|
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
string, one of |
No |
|
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
a valid filesystem path |
No |
|
No |
||
No |
||
No |
||
a valid filesystem path |
No |
|
No |
||
No |
||
No |
||
No |
||
No |
||
string, one of |
No |
Also see Common Options for a list of options supported by alloutput plugins.
"index"
Protocol agnostic (i.e. non-http, non-java specific) configs go hereProtocol agnostic methodsThe Elasticsearch action to perform. Valid actions are:
upsert
option. NOTE: This does not work and is not supported in Elasticsearch 1.x. Please upgrade to ES 2.x or greater to use this feature with Logstash!%{[foo]}
would use the foo field for the actionFor more details on actions, check out the Elasticsearch bulk API documentation
HTTP Path to perform the _bulk requests tothis defaults to a concatenation of the path parameter and "_bulk"
The .cer or .pem file to validate the server’s certificate
false
Enable doc_as_upsert
for update mode.Create a new document with source if document_id
doesn’t exist in Elasticsearch
The document ID for the index. Useful for overwriting existing entries inElasticsearch with the same ID.
Note: This option is deprecated due to the removal of types in Elasticsearch 6.0.It will be removed in the next major version of Logstash.This sets the document type to write events to. Generally you should try to write onlysimilar events to the same type. String expansion %{foo}
works here.If you don’t set a value for this option:
[]
Set the Elasticsearch errors in the whitelist that you don’t want to log.A useful example is when you want to skip all 409 errorswhich are document_already_exists_exception
.
Pass a set of key value pairs as the headers sent in each request toan elasticsearch node. The headers will be used for any kind of request(_bulk request, template installation, health checks and sniffing).These custom headers will be overidden by settings like http_compression
.
HTTP Path where a HEAD request is sent when a backend is marked downthe request is sent in the background to see if it has come back againbefore it is once again eligible to service requests.If you have custom firewall rules you may need to change this
[//127.0.0.1]
Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts
parameter.Remember the http
protocol uses the http address (eg. 9200, not 9300). "127.0.0.1"
["127.0.0.1:9200","127.0.0.2:9200"]
["http://127.0.0.1"]
["https://127.0.0.1:9200"]
["https://127.0.0.1:9200/mypath"]
(If using a proxy on a subpath)It is important to exclude dedicated master nodes from the hosts
listto prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.
Any special characters present in the URLs here MUST be URL escaped! This means #
should be put in as %23
for instance.
false
Enable gzip compression on requests. Note that response compression is on by default for Elasticsearch v5.0 and beyond
true
, false
, auto
auto
The default setting of auto
will automatically enable the Index Lifecycle Management feature, if the Elasticsearch cluster is running Elasticsearch version 7.0.0
or higher with the ILM feature enabled, and disable it otherwise.
Setting this flag to false
will disable the Index Lifecycle Management feature, even if the Elasticsearch cluster supports ILM.Setting this flag to true
will enable Index Lifecycle Management feature, if the Elasticsearch cluster supports it. This is requiredto enable Index Lifecycle Management on a version of Elasticsearch earlier than version 7.0.0
.
This feature requires a Basic License or above to be installed on an Elasticsearch cluster version 6.6.0 or later
{now/d}-000001
Pattern used for generating indices managed by Index Lifecycle Management. The value specified in the pattern will be appended tothe write alias, and incremented automatically when a new index is created by ILM.
Date Math can be used when specifying an ilm pattern, see Rollover API docs for details
Updating the pattern will require the index template to be rewritten
The pattern must finish with a dash and a number that will be automatically incremented when indices rollover.
logstash
Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy willbe automatically installed into Elasticsearch
If this setting is specified, the policy must already exist in Elasticsearch cluster.
logstash
The rollover alias is the alias where indices managed using Index Lifecycle Management will be written to.
If both index
and ilm_rollover_alias
are specified, ilm_rollover_alias
takes precedence.
Updating the rollover alias will require the index template to be rewritten
"logstash-%{+YYYY.MM.dd}"
The index to write events to. This can be dynamic using the %{foo}
syntax.The default value will partition your indices by day so you can more easilydelete old data or only search specific date ranges.Indexes may not contain uppercase characters.For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}.LS uses Joda to format the index pattern from event timestamp.Joda formats are defined here.
The keystore used to present a certificate to the server.It can be either .jks or .p12
Set the keystore password
true
From Logstash 1.3 onwards, a template is applied to Elasticsearch duringLogstash’s startup if one with the name template_name
does not already exist.By default, the contents of this template is the default template forlogstash-%{+YYYY.MM.dd}
which always matches indices based on the patternlogstash-*
. Should you require support for other index names, or would liketo change the mappings in the template in general, a custom template can bespecified by setting template
to the path of a template file.
Setting manage_template
to false disables this feature. If you require morecontrol over template creation, (e.g. creating indices dynamically based onfield names) you should set manage_template
to false and use the RESTAPI to apply your templates manually.
Pass a set of key value pairs as the URL query string. This query string is addedto every host listed in the hosts configuration. If the hosts list containsurls that already have query strings, the one specified here will be appended.
nil
For child documents, ID of the associated parent.This can be dynamic using the %{foo}
syntax.
Password to authenticate to a secure Elasticsearch cluster
HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remapsthe root path for the Elasticsearch HTTP API lives.Note that if you use paths as components of URLs in the hosts field you maynot also set this field. That will raise an error at startup
nil
Set which ingest pipeline you wish to execute for an event. You can also use event dependent configurationhere like pipeline => "%{INGEST_PIPELINE}"
1000
While the output tries to reuse connections efficiently we have a maximum.This sets the maximum number of open connections the output will create.Setting this too low may mean frequently closing / opening connectionswhich is bad.
100
While the output tries to reuse connections efficiently we have a maximum per endpoint.This sets the maximum number of open connections per endpoint the output will create.Setting this too low may mean frequently closing / opening connectionswhich is bad.
Set the address of a forward HTTP proxy.This used to accept hashes as arguments but now only acceptsarguments of the URI type to prevent leaking credentials.
5
How frequently, in seconds, to wait between resurrection attempts.Resurrection is the process by which backend endpoints marked down are checkedto see if they have come back to life
2
Set initial interval in seconds between bulk retries. Doubled on each retry up to retry_max_interval
64
Set max interval in seconds between bulk retries.
1
The number of times Elasticsearch should internally retry an update/upserted documentSee the partial updatesfor more info
A routing override to be applied to all processed events.This can be dynamic using the %{foo}
syntax.
""
Set script name for scripted update mode
Example:
output { elasticsearch { script => "ctx._source.message = params.event.get('message')" }}
"painless"
Set the language of the used script. If not set, this defaults to painless in ES 5.0.When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to ""
(empty string).
inline
, indexed
, file
["inline"]
Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticsearch’s config directory
"event"
Set variable name passed to script (scripted update)
false
if enabled, script is in charge of creating non-existent document (scripted update)
false
This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list.For Elasticsearch 1.x and 2.x any nodes with http.enabled
(on by default) will be added to the hosts list, including master-only nodes!For Elasticsearch 5.x and 6.x any nodes with http.enabled
(on by default) will be added to the hosts list, excluding master-only nodes.
5
How long to wait, in seconds, between sniffing attempts
HTTP Path to be used for the sniffing requeststhe default value is computed by concatenating the path value and "_nodes/http"if sniffing_path is set it will be used as an absolute pathdo not use full URL here, only paths, e.g. "/sniff/_nodes/http"
Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever schemeis specified in the URLs listed in hosts. If no explicit protocol is specified plain HTTP will be used.If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in hosts
true
Option to validate the server’s certificate. Disabling this severely compromises security.For more information on disabling certificate verification please readhttps://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
You can set the path to your own template here, if you so desire.If not set, the included template will be used.
"logstash"
This configuration option defines how the template is named inside Elasticsearch.Note that if you have used the template management features and subsequentlychange this, you will need to prune the old template manually, e.g.
curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>
where OldTemplateName
is whatever the former setting was.
false
The template_overwrite option will always overwrite the indicated templatein Elasticsearch with either the one indicated by template or the included one.This option is set to false by default. If you always want to stay up to datewith the template provided by Logstash, this option could be very useful to you.Likewise, if you have your own template file managed by puppet, for example, andyou wanted to be able to update it regularly, this option could help there as well.
Please note that if you are using your own customized version of the Logstashtemplate (logstash), setting this to true will make Logstash to overwritethe "logstash" template (i.e. removing all customized settings)
60
Set the timeout, in seconds, for network operations and requests sent Elasticsearch. Ifa timeout occurs, the request will be retried.
The truststore to validate the server’s certificate.It can be either .jks or .p12.Use either :truststore
or :cacert
.
Set the truststore password
""
Set upsert content for update mode.Create a new document with this parameter as json string if document_id
doesn’t exists
Username to authenticate to a secure Elasticsearch cluster
10000
How long to wait before checking if the connection is stale before executing a request on a connection using keepalive.You may want to set this lower, if you get connection errors regularlyQuoting the Apache commons docs (this client is based Apache Commmons):Defines period of inactivity in milliseconds after which persistent connections mustbe re-validated prior to being leased to the consumer. Non-positive value passed tothis method disables connection validation. This check helps detect connections thathave become stale (half-closed) while kept inactive in the pool.See these docs for more info
The version to use for indexing. Use sprintf syntax like %{my_version}
to use a field value here.See https://www.elastic.co/blog/elasticsearch-versioning-support.
internal
, external
, external_gt
, external_gte
, force
The version_type to use for indexing.See https://www.elastic.co/blog/elasticsearch-versioning-support.See also https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#_version_types
The following configuration options are supported by all output plugins:
Setting | Input type | Required |
---|---|---|
No |
||
No |
true
Disable or enable metric logging for this specific plugin instance.By default we record all the metrics we can, but you can disable metrics collectionfor a specific plugin.
Add a unique ID
to the plugin configuration. If no ID is specified, Logstash will generate one.It is strongly recommended to set this ID in your configuration. This is particularly usefulwhen you have two or more plugins of the same type. For example, if you have 2 elasticsearch outputs.Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.
output { elasticsearch { id => "my_plugin_id" }}