The instructions below assume that you have Logstash, Elasticsearch, and Kibana running locally.You can also run Logstash, Elasticsearch, and Kibana on separate hosts.

The Elastic Stack version 6.4 (or later) is required for this module.

The Azure module uses the azure_event_hubs input plugin to consume logs andmetrics from your Azure environment. It is installed by default with Logstash 6.4(or later). Basic understanding of the plugin and options is helpful when youset up the Azure module.See the azure_event_hubs inputplugin documentation for more information.

Elastic products are available to download andeasy to install.

Azure Monitor should be configured to stream logs to one or more Event Hubs.Logstash will need to access these Event Hubs instances to consume your Azure logs and metrics.See Microsoft Azure resources at the end of this topic for links to Microsoft Azure documentation.

The configuration in the logstash.yml file is shared between Event Hubs.Basic configuration is recommended for most use cases

modules:  - name: azure    var.elasticsearch.hosts: localhost:9200    var.kibana.host: localhost:5601    var.input.azure_event_hubs.consumer_group: "logstash"     var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..."     var.input.azure_event_hubs.threads: 9     var.input.azure_event_hubs.event_hub_connections:      - "Endpoint=sb://...EntityPath=insights-operational-logs"       - "Endpoint=sb://...EntityPath=insights-metrics-pt1m"       - "Endpoint=sb://...EntityPath=insights-logs-blocks"      - "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"      - "Endpoint=sb://...EntityPath=insights-logs-errors"      - "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"      - "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"      - "Endpoint=sb://...EntityPath=insights-logs-timeouts"

The basic configuration requires the var.input.azure_event_hubs. prefixbefore a configuration option.Notice the notation for the threads option.

Advanced configuration in the logstash.yml file supports Event Hub specificoptions. Advanced configuration is available for more granular tuning ofthreading and Blob Storage usage across multiple Event Hubs. Advancedconfiguration is not necessary or recommended for most use cases. Use it only ifit is required for your deployment scenario.

You must define the header array with name in the first position. You candefine other options in any order. The per Event Hub configuration takesprecedence. Any values not defined per Event Hub use the global config value.

In this example threads, consumer_group, and storage_connection will beapplied to each of the configured Event Hubs. Note that decorate_events isdefined in both the global and per Event Hub configuration. The per Event Hubconfiguration takes precedence, and the global configuration is effectivelyignored when the per Event Hub setting is present.

modules:  - name: azure    var.elasticsearch.hosts: localhost:9200    var.kibana.host: localhost:5601    var.input.azure_event_hubs.decorate_events: true     var.input.azure_event_hubs.threads: 9     var.input.azure_event_hubs.consumer_group: "logstash"    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..."    var.input.azure_event_hubs.event_hubs:      - ["name",                                    "initial_position",  "storage_container",  "decorate_events",  "event_hub_connection"]                                         - ["insights-operational-logs",                 "TAIL",              "activity-logs1",    "true",             "Endpoint=sb://...EntityPath=insights-operational-logs"]      - ["insights-operational-logs",                 "TAIL",              "activity_logs2",   "true",             "Endpoint=sb://...EntityPath=insights-operational-logs"]      - ["insights-metrics-pt1m",                     "TAIL",              "dbmetrics",         "true",             "Endpoint=sb://...EntityPath=insights-metrics-pt1m"]      - ["insights-logs-blocks",                      "TAIL",              "dbblocks",          "true",             "Endpoint=sb://...EntityPath=insights-logs-blocks"]      - ["insights-logs-databasewaitstatistics",      "TAIL",              "dbwaitstats",       "false",            "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"]      - ["insights-logs-errors",                      "HEAD",              "dberrors",          "true",             "Endpoint=sb://...EntityPath=insights-logs-errors"      - ["insights-logs-querystoreruntimestatistics", "TAIL",              "dbstoreruntime",    "true",             "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"]      - ["insights-logs-querystorewaitstatistics",    "TAIL",              "dbstorewaitstats",  "true",             "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"]      - ["insights-logs-timeouts",                    "TAIL",              "dbtimeouts",        "true",             "Endpoint=sb://...EntityPath=insights-logs-timeouts"]

The advanced configuration doesn’t require a prefix before a per Event Hubconfiguration option. Notice the notation for the initial_position option.

An Azure Blob Storageaccount is an essential part of Azure-to-Logstash configuration.It is required for users who want to scale out multiple Logstash instances to consume from Event Hubs.

A Blob Storage account is a central location that enables multiple instances ofLogstash to work together to process events. It records theoffset (location) of processed events. On restart, Logstash resumes processingexactly where it left off.

Configuration notes:

Sample Blob Storage connection string:

DefaultEndpointsProtocol=https;AccountName=logstash;AccountKey=ETOPnkd/hDAWidkEpPZDiXffQPku/SZdXhPSLnfqdRTalssdEuPkZwIcouzXjCLb/xPZjzhmHfwRCGo0SBSw==;EndpointSuffix=core.windows.net

Find the connection string to Blob Storage here:Azure Portal-> Blob Storage account -> Access keys.

Here are some guidelines to help you achieve a successful deployment, and avoiddata conflicts that can cause lost events.

Run this command from the Logstash directory:

bin/logstash --setup

The --modules azure option starts a Logstash pipeline for ingestion from AzureEvent Hubs. The --setup option creates an azure-* index pattern inElasticsearch and imports Kibana dashboards and visualizations.

Run this command from the Logstash directory:

bin/logstash

The --setup option is intended only for first-time setup. If you include--setup on subsequent runs, your existing Kibana dashboards will beoverwritten.

Defines the per Event Hubs configuration for the advanced configuration.

The advanced configuration uses event_hub_connection instead of event_hub_connections.The event_hub_connection option is defined per Event Hub.

List of connection strings that identifies the Event Hubs to be read. Connectionstrings include the EntityPath for the Event Hub.

Interval in seconds to write checkpoints during batch processing. Checkpointstell Logstash where to resume processing after a restart. Checkpoints areautomatically written at the end of each batch, regardless of this setting.

Writing checkpoints too frequently can slow down processing unnecessarily.

Consumer group used to read the Event Hub(s). Create a consumer groupspecifically for Logstash. Then ensure that all instances of Logstash use thatconsumer group so that they can work together properly.

Adds metadata about the Event Hub, including Event Hub name, consumer_group,processor_host, partition, offset, sequence, timestamp, and event_size.

When first reading from an Event Hub, start from this position:

If storage_connection is set, the initial_position value is used onlythe first time Logstash reads from the Event Hub.

Number of seconds to look back to find the initial position for pre-existingevents. This option is used only if initial_position is set to look_back. Ifstorage_connection is set, this configuration applies only the first time Logstashreads from the Event Hub.

Maximum number of events retrieved and processed together. A checkpoint iscreated after each batch. Increasing this value may help with performance, butrequires more memory.

Connection string for blob account storage. Blob account storage persists theoffsets between restarts, and ensures that multiple instances of Logstashprocess different partitions.When this value is set, restarts resume where processing left off.When this value is not set, the initial_position value is used on every restart.

We strongly recommend that you define this value for production environments.

Name of the storage container used to persist offsets and allow multiple instances of Logstashto work together.

To avoid overwriting offsets, you can use different storage containers. This isparticularly important if you are monitoring two Event Hubs with the same name.You can use the advanced configuration model to configure different storagecontainers.

Total number of threads used to process events. The value you set here appliesto all Event Hubs. Even with advanced configuration, this value is a globalsetting, and can’t be set per event hub.

The number of threads should be the number of Event Hubs plus one or more.See Best practices for more information.

Common options

The following configuration options are supported by all modules:

Many of the logs contain a properties top level field. This is often where themost interesting data lives. There is not a fixed schema between log types forproperties fields coming from different sources.

For example, one log may haveproperties.type where one log sets this a String type and another sets this anInteger type. To avoid mapping errors, the original properties field is moved to<azure.group>_<azure_category>_properties.<original_key>.For exampleproperties.type may end up as sql_diagnostics_Errors_properties.type oractivity_log_Security_properties.type depending on the group/category wherethe event originated.