Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin:

You can now create your own Logstash plugin in seconds! The generate subcommand of bin/logstash-plugin creates the foundationfor a new Logstash plugin with templatized files. It creates the correct directory structure, gemspec files, and dependencies so youcan start adding custom code to process data with Logstash.

For more information, see Generating Plugins

Alternatively, you can use the examples repo we host on github.com

Your file structure should look like this:

$ tree logstash-filter-mypluginname├── Gemfile├── LICENSE├── README.md├── Rakefile├── lib│   └── logstash│       └── filters│           └── mypluginname.rb├── logstash-filter-mypluginname.gemspec└── spec    └── filters        └── mypluginname_spec.rb

For more information about the Ruby gem file structure and an excellentwalkthrough of the Ruby gem creation process, seehttp://timelessrepo.com/making-ruby-gems

Before we dive into the details, open up the plugin file in your favorite text editorand take a look.

# encoding: utf-8require "logstash/filters/base"require "logstash/namespace"# Add any asciidoc formatted documentation here# This example filter will replace the contents of the default# message field with whatever you specify in the configuration.## It is only intended to be used as an example.class LogStash::Filters::Example < LogStash::Filters::Base  # Setting the config_name here is required. This is how you  # configure this filter from your Logstash config.  #  # filter {  #   example { message => "My message..." }  # }  config_name "example"  # Replace the message with this value.  config :message, :validate => :string, :default => "Hello World!"  public  def register    # Add instance variables  end # def register  public  def filter(event)    if @message      # Replace the event message with our message as configured in the      # config file.      event.set("message", @message)    end    # filter_matched should go in the last line of our successful code    filter_matched(event)  end # def filterend # class LogStash::Filters::Example

It seems like a small thing, but remember to specify the encoding at thebeginning of your plugin code:

# encoding: utf-8

Logstash depends on things being in UTF-8, so we put this here to tell the Rubyinterpreter that we’re going to be using the UTF-8 encoding.

Logstash filter plugins require parent classes defined inlogstash/filters/base and logstash/namespace:

require "logstash/filters/base"require "logstash/namespace"

Of course, the plugin you build may depend on other code, or even gems. Just putthem here along with these Logstash dependencies.

The filter plugin class should be a subclass ofLogStash::Filters::Base:

class LogStash::Filters::Example < LogStash::Filters::Base

The class name should closely mirror the plugin name, for example:

LogStash::Filters::Example

  config_name "example"

This is the name your plugin will call inside the filter configurationblock.

If you set config_name "example" in your plugin code,the corresponding Logstash configuration block would need to look like this:

  public  def register  end # def register

The Logstash register method is like an initialize method. It was originallycreated to enforce having super called, preventing headaches for newbies.(Note: It may go away in favor of initialize, in conjunction with someenforced testing to ensure super is called.)

public means the method can be called anywhere, not just within the class.This is the default behavior for methods in Ruby, but it is specified explicitlyhere anyway.

You can also assign instance variables here (variables prepended by @).Configuration variables are now in scope as instance variables, like @message

  public  def filter(event)    if @message      # Replace the event message with our message as configured in the      # config file.      event.set("message", @message)    end  # filter_matched should go in the last line of our successful code  filter_matched(event)end # def filter

The plugin’s filter method is where the actual filtering work takes place!Inside the filter method you can refer to the event data using the Eventobject. Event is the main object that encapsulates data flow internally in Logstashand provides an API for the plugin developers to interact with theevent’s content.

The filter method should also handle any event dependent configuration byexplicitly calling the sprintf method available in Event class. For example:

field_foo = event.sprintf(field)

Note that configuration variables are now in scope as instance variables, like@message

  filter_matched(event)

Calling the filter_matched method upon successful execution of the plugin willensure that any fields or tags added through the Logstash configuration for thisfilter will be handled correctly. For example, any add_field, remove_field,add_tag and/or remove_tag actions will be performed at this time.

Event methods such as event.cancel are now available to control the workflowof the event being processed.

A require statement in Ruby is used to include necessary code. In some casesyour plugin may require additional files. For example, the collectd pluginusesthe types.db file provided by collectd. In the main directory of your plugin,a file called vendor.json is where these files are described.

The vendor.json file contains an array of JSON objects, each describing a filedependency. This example comes from thecollectdcodec plugin:

[{        "sha1": "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e",        "url": "http://collectd.org/files/collectd-5.4.0.tar.gz",        "files": [ "/src/types.db" ]}]

Another example of the vendor.json file is thegeoip filter

The process used to download these dependencies is to call rake vendor. Thiswill be discussed further in the testing section of this document.

Another kind of external dependency is on jar files. This will be describedin the "Add a gemspec file" section.

Gemfiles allow Ruby’s Bundler to maintain the dependencies for your plugin.Currently, all we’ll need is the Logstash gem, for testing, but if you requireother gems, you should add them in here.

See Bundler’s Gemfile page for more details.

source 'https://rubygems.org'gemspecgem "logstash", :github => "elastic/logstash", :branch => "7.0"

At the bottom of the gemspec file is a section with a comment:Gem dependencies. This is where any other needed gems must be mentioned. Ifa gem is necessary for your plugin to function, it is a runtime dependency. Ifa gem are only used for testing, then it would be a development dependency.

You can also have versioning requirements for your dependencies—including otherLogstash plugins:

  # Gem dependencies  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"  s.add_development_dependency 'logstash-devutils'

This gemspec has a runtime dependency on the logstash-core-plugin-api and requires thatit have a version number greater than or equal to version 1.60 and less than or equal to version 2.99.

All plugins have a runtime dependency on the logstash-core-plugin-api gem, anda development dependency on logstash-devutils.

In some cases, such as theElasticsearch output plugin,your code may depend on a jar file. In cases such as this, the dependency isadded in the gemspec file in this manner:

  # Jar dependencies  s.requirements << "jar 'org.elasticsearch:elasticsearch', '5.0.0'"  s.add_runtime_dependency 'jar-dependencies'

With these both defined, the install process will search for the required jarfile at http://mvnrepository.com and download the specified version.

You already have all the necessary ingredients, so let’s go ahead and run thebuild command:

gem build logstash-filter-example.gemspec

That’s it! Your gem should be built and be in the same path with the name

logstash-filter-mypluginname-0.1.0.gem

The s.version number from your gemspec file will provide the gem version, inthis case, 0.1.0.

You should test install your plugin into a clean installation of Logstash.Download the latest version from theLogstash downloads page.

Your results will depend on what your filter plugin is designed to do.

bin/logstash -e 'input { stdin{} } filter { example {} } output {stdout { codec => rubydebug }}'

Test your filter by sending input through stdin and output (after filtering)through stdout with the rubydebug codec, which enhances readability.

In the case of the example filter plugin, any text you send will bereplaced by the contents of the message configuration parameter, the defaultvalue being "Hello World!":

Testing 1, 2, 3{       "message" => "Hello World!",      "@version" => "1",    "@timestamp" => "2015-01-27T19:17:18.932Z",          "host" => "cadenza"}

Feel free to experiment and test this by changing the message parameter:

bin/logstash -e 'input { stdin{} } filter { example { message => "This is a new message!"} } output {stdout { codec => rubydebug }}'

Congratulations! You’ve built, deployed and successfully run a Logstashfilter.

Logstash and all its plugins are licensed underApache License, version 2 ("ALv2").If you make your plugin publicly available via RubyGems.org,please make sure to have this line in your gemspec:

To begin, you’ll need an account on RubyGems.org

After creating an account,obtain an APIkey from RubyGems.org. By default, RubyGems uses the file ~/.gem/credentialsto store your API key. These credentials will be used to publish the gem.Replace username and password with the credentials you created atRubyGems.org:

curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentialschmod 0600 ~/.gem/credentials

Before proceeding, make sure you have the right version in your gemspec fileand commit your changes.

To publish version 0.1.0 of your new logstash gem:

bundle installbundle exec rake vendorbundle exec rspecbundle exec rake publish_gem

Executing rake publish_gem:

1. Reads the version from the gemspec file (s.version = '0.1.0')
2. Checks in your local repository if a tag exists for that version. If the tagalready exists, it aborts the process. Otherwise, it creates a new version tagin your local repository.
3. Builds the gem
4. Publishes the gem to RubyGems.org

That’s it! Your plugin is published! Logstash users can now install your pluginby running:

bin/logstash-plugin install logstash-filter-mypluginname

Some of the many benefits of having your plugin in the logstash-pluginsrepository are:

To begin migrating your plugin to logstash-plugins, simply create a newissue inthe Logstash repository. When the acceptance guidelines are completed, we willfacilitate the move to the logstash-plugins organization using the recommendedgithub process.