The Elastic Stack is used for tons of use cases, from operational log andmetrics analytics, to enterprise and application search. Making sure your datagets scalably, durably, and securely transported to Elasticsearch is extremelyimportant, especially for mission critical environments.

The goal of this document is to highlight the most common architecture patternsfor Logstash and how to effectively scale as your deployment grows. The focuswill be around the operational log, metrics, and security analytics use casesbecause they tend to require larger scale deployments. The deploying and scalingrecommendations provided here may vary based on your own requirements.

Getting Started

For first time users, if you simply want to tail a log file to grasp the powerof the Elastic Stack, we recommend tryingFilebeat Modules. Filebeat Modulesenable you to quickly collect, parse, and index popular log types and viewpre-built Kibana dashboards within minutes.Metricbeat Modules provide a similarexperience, but with metrics data. In this context, Beats will ship datadirectly to Elasticsearch where Ingest Nodes will processand index your data.

Introducing Logstash

What are the main benefits for integrating Logstash into your architecture?

Scaling Ingest

Beats and Logstash make ingest awesome. Together, they provide a comprehensivesolution that is scalable and resilient. What can you expect?

Beats and Logstash

Beats run across thousands of edge host servers, collecting, tailing, andshipping logs to Logstash. Logstash serves as the centralized streamingengine for data unification and enrichment. TheBeats input plugin exposes a secure,acknowledgement-based endpoint for Beats to send data to Logstash.

Scalability

Logstash is horizontally scalable and can form groups of nodes running the samepipeline. Logstash’s adaptive buffering capabilities will facilitate smoothstreaming even through variable throughput loads. If the Logstash layer becomesan ingestion bottleneck, simply add more nodes to scale out. Here are a fewgeneral recommendations:

Resiliency

When using Filebeat orWinlogbeat for log collectionwithin this ingest flow, at-least-once delivery is guaranteed. Both thecommunication protocols, from Filebeat or Winlogbeat to Logstash, and fromLogstash to Elasticsearch, are synchronous and support acknowledgements. Theother Beats don’t yet have support for acknowledgements.

Logstash persistent queues provide protection across node failures. Fordisk-level resiliency in Logstash, it’s important to ensure disk redundancy.For on-premise deployments, it’s recommended that you configure RAID. Whenrunning in the cloud or a containerized environment, it’s recommended that youuse persistent disks with replication strategies that reflect your data SLAs.

Processing

Logstash will commonly extract fields with grok ordissect, augmentgeographical info, and can further enrich events withfile, database,or Elasticsearch lookup datasets. Be awarethat processing complexity can affect overall throughput and CPU utilization.Make sure to check out the other available filter plugins.

Secure Transport

Enterprise-grade security is available across the entire delivery chain.

Monitoring

When running Logstash 5.2 or greater,the Monitoring UI providesdeep visibility into your deployment metrics, helping observe performance andalleviate bottlenecks as you scale. Monitoring is an X-Pack feature under theBasic License and is therefore free to use. To get started, seeMonitoring Logstash.

If external monitoring is preferred, there are Monitoring APIsthat return point-in-time metrics snapshots.

Adding Other Popular Sources

Users may have other mechanisms of collecting logging data, and it’s easy tointegrate and centralize them into the Elastic Stack. Let’s walk through a fewscenarios:

TCP, UDP, and HTTP Protocols

The TCP, UDP, and HTTP protocols are common ways to feed data into Logstash.Logstash can expose endpoint listeners with the respectiveTCP, UDP, andHTTP input plugins. The data sources enumerated beloware typically ingested through one of these three protocols.

For high availability scenarios, a third-party hardware or software loadbalancer, like HAProxy, should be added to fan out traffic to a group ofLogstash nodes.

Network and Security Data

Although Beats may already satisfy your data ingest use case, network andsecurity datasets come in a variety of forms. Let’s touch on a few otheringestion points.

Centralized Syslog Servers

Existing syslog server technologies like rsyslog and syslog-ng generally sendsyslog over to Logstash TCP or UDP endpoints for extraction, processing, andpersistence. If the data format conforms to RFC3164, it can be fed directlyto the Logstash syslog input.

Infrastructure & Application Data and IoT

Infrastructure and application metrics can be collected withMetricbeat, but applicationscan also send webhooks to a Logstash HTTP input or have metrics polled from anHTTP endpoint with the HTTP poller input plugin.

For applications that log with log4j2, it’s recommended to use theSocketAppender to send JSON to the Logstash TCP input. Alternatively, log4j2can also log to a file for collection with FIlebeat. Usage of the log4j1SocketAppender is not recommended.

IoT devices like Raspberry Pis, smartphones, and connected vehicles often sendtelemetry data through one of these protocols.

Integrating with Messaging Queues

If you are leveraging message queuing technologies as part of your existinginfrastructure, getting that data into the Elastic Stack is easy. For existingusers who are utilizing an external queuing layer like Redis or RabbitMQ justfor data buffering with Logstash, it’s recommended to use Logstash persistentqueues instead of an external queuing layer. This will help with overall easeof management by removing an unnecessary layer of complexity in your ingestarchitecture.

For users who want to integrate data from existing Kafka deployments or requirethe underlying usage of ephemeral storage, Kafka can serve as a data hub whereBeats can persist to and Logstash nodes can consume from.

The other TCP, UDP, and HTTP sources can persist to Kafka with Logstash as aconduit to achieve high availability in lieu of a load balancer. A group ofLogstash nodes can then consume from topics with theKafka input to further transform and enrich the data intransit.

Resiliency and Recovery

When Logstash consumes from Kafka, persistent queues should be enabled and willadd transport resiliency to mitigate the need for reprocessing during Logstashnode failures. In this context, it’s recommended to use the default persistentqueue disk allocation size queue.max_bytes: 1GB.

If Kafka is configured to retain data for an extended period of time, data canbe reprocessed from Kafka in the case of disaster recovery and reconciliation.

Other Messaging Queue Integrations

Although an additional queuing layer is not required, Logstash can consume froma myriad of other message queuing technologies likeRabbitMQ and Redis. It alsosupports ingestion from hosted queuing services likePub/Sub, Kinesis, andSQS.