The information you need to manage often comes from several disparate sources, and use cases can require multipledestinations for your data. Your Logstash pipeline can use multiple input and output plugins to handle theserequirements.
In this section, you create a Logstash pipeline that takes input from a Twitter feed and the Filebeat client, thensends the information to an Elasticsearch cluster as well as writing the information directly to a file.
To add a Twitter feed, you use the twitter
input plugin. Toconfigure the plugin, you need several pieces of information:
Visit https://dev.twitter.com/apps to set up a Twitter account and generate your consumerkey and secret, as well as your access token and secret. See the docs for the twitter
input plugin if you’re not sure how to generate these keys.
Like you did earlier when you worked on Parsing Logs with Logstash, create a config file (called second-pipeline.conf
) thatcontains the skeleton of a configuration pipeline. If you want, you can reuse the file you created earlier, but makesure you pass in the correct config file name when you run Logstash.
Add the following lines to the input
section of the second-pipeline.conf
file, substituting your values for theplaceholder values shown here:
twitter { consumer_key => "enter_your_consumer_key_here" consumer_secret => "enter_your_secret_here" keywords => ["cloud"] oauth_token => "enter_your_access_token_here" oauth_token_secret => "enter_your_access_token_secret_here" }
As you learned earlier in Configuring Filebeat to Send Log Lines to Logstash, the Filebeatclient is a lightweight, resource-friendly tool that collects logs from files on the server and forwards these logs to yourLogstash instance for processing.
After installing Filebeat, you need to configure it. Open the filebeat.yml
file located in your Filebeat installationdirectory, and replace the contents with the following lines. Make sure paths
points to your syslog:
filebeat.inputs:- type: log paths: - /var/log/*.log fields: type: syslog output.logstash: hosts: ["localhost:5044"]
Absolute path to the file or files that Filebeat processes. |
|
Adds a field called |
Save your changes.
To keep the configuration simple, you won’t specify TLS/SSL settings as you would in a real worldscenario.
Configure your Logstash instance to use the Filebeat input plugin by adding the following lines to the input
sectionof the second-pipeline.conf
file:
beats { port => "5044" }
You can configure your Logstash pipeline to write data directly to a file with thefile
output plugin.
Configure your Logstash instance to use the file
output plugin by adding the following lines to the output
sectionof the second-pipeline.conf
file:
file { path => "/path/to/target/file" }
Writing to multiple Elasticsearch nodes lightens the resource demands on a given Elasticsearch node, as well asproviding redundant points of entry into the cluster when a particular node is unavailable.
To configure your Logstash instance to write to multiple Elasticsearch nodes, edit the output
section of the second-pipeline.conf
file to read:
output { elasticsearch { hosts => ["IP Address 1:port1", "IP Address 2:port2", "IP Address 3"] }}
Use the IP addresses of three non-master nodes in your Elasticsearch cluster in the host line. When the hosts
parameter lists multiple IP addresses, Logstash load-balances requests across the list of addresses. Also note thatthe default port for Elasticsearch is 9200
and can be omitted in the configuration above.
At this point, your second-pipeline.conf
file looks like this:
input { twitter { consumer_key => "enter_your_consumer_key_here" consumer_secret => "enter_your_secret_here" keywords => ["cloud"] oauth_token => "enter_your_access_token_here" oauth_token_secret => "enter_your_access_token_secret_here" } beats { port => "5044" }}output { elasticsearch { hosts => ["IP Address 1:port1", "IP Address 2:port2", "IP Address 3"] } file { path => "/path/to/target/file" }}
Logstash is consuming data from the Twitter feed you configured, receiving data from Filebeat, andindexing this information to three nodes in an Elasticsearch cluster as well as writing to a file.
At the data source machine, run Filebeat with the following command:
sudo ./filebeat -e -c filebeat.yml -d "publish"
Filebeat will attempt to connect on port 5044. Until Logstash starts with an active Beats plugin, therewon’t be any answer on that port, so any messages you see regarding failure to connect on that port are normal for now.
To verify your configuration, run the following command:
bin/logstash -f second-pipeline.conf --config.test_and_exit
The --config.test_and_exit
option parses your configuration file and reports any errors. When the configuration filepasses the configuration test, start Logstash with the following command:
bin/logstash -f second-pipeline.conf
Use the grep
utility to search in the target file to verify that information is present:
grep syslog /path/to/target/file
Run an Elasticsearch query to find the same information in the Elasticsearch cluster:
curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=fields.type:syslog'
Replace $DATE with the current date, in YYYY.MM.DD format.
To see data from the Twitter feed, try this query:
curl -XGET 'http://localhost:9200/logstash-$DATE/_search?pretty&q=client:iphone'
Again, remember to replace $DATE with the current date, in YYYY.MM.DD format.