This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

To develop a new Java filter for Logstash, you write a new Java class thatconforms to the Logstash Java Filters API, package it, and install it with thelogstash-plugin utility. We’ll go through each of those steps.

Set up your environment

Copy the example repo

Start by copying the example filter plugin. The plugin API is currently part of theLogstash codebase so you must have a local copy of that available. You canobtain a copy of the Logstash codebase with the following git command:

git clone --branch <branch_name> --single-branch https://github.com/elastic/logstash.git <target_folder>

The branch_name should correspond to the version of Logstash containing thepreferred revision of the Java plugin API.

The beta version of the Java plugin API is available in the 6.7branch of the Logstash codebase.

Specify the target_folder for your local copy of the Logstash codebase. If youdo not specify target_folder, it defaults to a new folder called logstashunder your current folder.

Generate the .jar file

After you have obtained a copy of the appropriate revision of the Logstashcodebase, you need to compile it to generate the .jar file containing the Javaplugin API. From the root directory of your Logstash codebase ($LS_HOME), youcan compile it with ./gradlew assemble (or gradlew.bat assemble if you’rerunning on Windows). This should produce the$LS_HOME/logstash-core/build/libs/logstash-core-x.y.z.jar where x, y, andz refer to the version of Logstash.

After you have successfully compiled Logstash, you need to tell your Java pluginwhere to find the logstash-core-x.y.z.jar file. Create a new file namedgradle.properties in the root folder of your plugin project. That file shouldhave a single line:

LOGSTASH_CORE_PATH=<target_folder>/logstash-core

where target_folder is the root folder of your local copy of the Logstash codebase.

Code the plugin

The example filter plugin allows one to configure a field in each event thatwill be reversed. For example, if the filter were configured to reverse theday_of_week field, an event with day_of_week: "Monday" would be transformedto day_of_week: "yadnoM". Let’s look at the main class in that example filter:

@LogstashPlugin(name = "java_filter_example")public class JavaFilterExample implements Filter {    public static final PluginConfigSpec<String> SOURCE_CONFIG =            PluginConfigSpec.stringSetting("source", "message");    private String id;    private String sourceField;    public JavaFilterExample(String id, Configuration config, Context context) {        this.id = id;        this.sourceField = config.get(SOURCE_CONFIG);    }    @Override    public Collection<Event> filter(Collection<Event> events, FilterMatchListener matchListener) {        for (Event e : events) {            Object f = e.getField(sourceField);            if (f instanceof String) {                e.setField(sourceField, StringUtils.reverse((String)f));                matchListener.filterMatched(e);            }        }        return events;    }    @Override    public Collection<PluginConfigSpec<?>> configSchema() {        return Collections.singletonList(SOURCE_CONFIG);    }    @Override    public String getId() {        return this.id;    }}

Let’s step through and examine each part of that class.

Class declaration

@LogstashPlugin(name = "java_filter_example")public class JavaFilterExample implements Filter {

Notes about the class declaration:

Plugin settings

The snippet below contains both the setting definition and the method referencing it:

public static final PluginConfigSpec<String> SOURCE_CONFIG =        PluginConfigSpec.stringSetting("source", "message");@Overridepublic Collection<PluginConfigSpec<?>> configSchema() {    return Collections.singletonList(SOURCE_CONFIG);}

The PluginConfigSpec class allows developers to specify the settings that a plugin supports complete with settingname, data type, deprecation status, required status, and default value. In this example, the source setting definesthe name of the field in each event that will be reversed. It is not a required setting and if it is not explicitlyset, its default value will be message.

The configSchema method must return a list of all settings that the plugin supports. In a future phase of theJava plugin project, the Logstash execution engine will validate that all required settings are present and thatno unsupported settings are present.

Constructor and initialization

private String id;private String sourceField;public JavaFilterExample(String id, Configuration config, Context context) {    this.id = id;    this.sourceField = config.get(SOURCE_CONFIG);}

All Java filter plugins must have a constructor taking a String id and aConfiguration and Context argument. This is the constructor that will beused to instantiate them at runtime. The retrieval and validation of all pluginsettings should occur in this constructor. In this example, the name of thefield to be reversed in each event is retrieved from its setting and stored ina local variable so that it can be used later in the filter method.

Any additional initialization may occur in the constructor as well. If there areany unrecoverable errors encountered in the configuration or initialization ofthe filter plugin, a descriptive exception should be thrown. The exception willbe logged and will prevent Logstash from starting.

Filter method

@Overridepublic Collection<Event> filter(Collection<Event> events, FilterMatchListener matchListener) {    for (Event e : events) {        Object f = e.getField(sourceField);        if (f instanceof String) {            e.setField(sourceField, StringUtils.reverse((String)f));            matchListener.filterMatched(e);        }    }    return events;

Finally, we come to the filter method that is invoked by the Logstashexecution engine on batches of events as they flow through the event processingpipeline. The events to be filtered are supplied in the events argument andthe method should return a collection of filtered events. Filters may perform avariety of actions on events as they flow through the pipeline including:

In the example above, the value of the source field is retrieved from eachevent and reversed if it is a string value. Because each event is mutated inplace, the incoming events collection can be returned.

The matchListener is the mechanism by which filters indicate which events"match". The common actions for filters such as add_field and add_tag areapplied only to events that are designated as "matching". Some filters such asthe grokfilterhave a clear definition for what constitutes a matching event and will notifythe listener only for matching events. Other filters such as theUUIDfilterhave no specific match criteria and should notify the listener for every eventfiltered. In this example, the filter notifies the match listener for any eventthat had a String value in its source field and was therefore able to bereversed.

getId method

@Overridepublic String getId() {    return id;}

For filter plugins, the getId method should always return the id that was provided to the plugin through itsconstructor at instantiation time.

Unit tests

Lastly, but certainly not least importantly, unit tests are strongly encouraged.The example filter plugin includes anexampleunit test that you can use as a template for your own.

Package and deploy

Java plugins are packaged as Ruby gems for dependency management andinteroperability with Ruby plugins.

One of the goals for Java plugin support is to eliminate the need for anyknowledge of Ruby or its toolchain for Java plugin development. Future phases ofthe Java plugin project will automate the packaging of Java plugins as Ruby gemsso no direct knowledge of or interaction with Ruby will be required. In thecurrent phase, Java plugins must still be manually packaged as Ruby gemsand installed with the logstash-plugin utility.

Compile to JAR file

The Java plugin should be compiled and assembled into a fat jar with thevendor task in the Gradle build file. This will package all Java dependenciesinto a single jar and write it to the correct folder for later packaging into aRuby gem.

Manually package as Ruby gem

Several Ruby source files are required to package the jar file as aRuby gem. These Ruby files are used only at Logstash startup time to identifythe Java plugin and are not used during runtime event processing.

These Ruby source files will be automatically generated in a future release.

logstash-filter-<filter-name>.gemspec

Gem::Specification.new do |s|  s.name            = 'logstash-filter-java_filter_example'  s.version         = PLUGIN_VERSION  s.licenses        = ['Apache-2.0']  s.summary         = "Example filter using Java plugin API"  s.description     = ""  s.authors         = ['Elasticsearch']  s.email           = 'info@elastic.co'  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"  s.require_paths = ['lib', 'vendor/jar-dependencies']  # Files  s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]  # Special flag to let us know this is actually a logstash plugin  s.metadata = { 'logstash_plugin' => 'true', 'logstash_group' => 'filter'}  # Gem dependencies  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"  s.add_runtime_dependency 'jar-dependencies'  s.add_development_dependency 'logstash-devutils'end

You can use this file with the following modifications:

lib/logstash/filters/<filter-name>.rb

# encoding: utf-8require "logstash/filters/base"require "logstash/namespace"require "logstash-filter-java_filter_example_jars"require "java"class LogStash::filters::JavaFilterExample < LogStash::Filters::Base  config_name "java_filter_example"  def self.javaClass() org.logstash.javaapi.JavaFilterExample.java_class; endend

Modify these items in the file above:

lib/logstash-filter-<filter-name>_jars.rb

require 'jar_dependencies'require_jar('org.logstash.javaapi', 'logstash-filter-java_filter_example', '0.0.1')

In the file above:

After you have created the previous files and the plugin JAR file, build the gem using thefollowing command:

gem build logstash-filter-<filter-name>.gemspec

Installing the Java plugin in Logstash

After you have packaged your Java plugin as a Ruby gem, you can install it inLogstash with this command:

bin/logstash-plugin install --no-verify --local /path/to/javaPlugin.gem

For Windows platforms: Substitute backslashes for forward slashes as appropriate in the command.

Run Logstash with the Java filter plugin

The following is a minimal Logstash configuration that can be used to test thatthe Java filter plugin is correctly installed and functioning.

input {  generator { message => "Hello world!" count => 1 }}filter {  java_filter_example {}}output {  stdout { codec => rubydebug }}

Copy the above Logstash configuration to a file such as java_filter.conf.Start Logstash with:

bin/logstash --java-execution -f /path/to/java_filter.conf

The --java-execution flag to enable the Java execution engine isrequired as Java plugins are not supported in the Ruby execution engine.

The expected Logstash output (excluding initialization) with the configurationabove is:

{      "sequence" => 0,      "@version" => "1",       "message" => "!dlrow olleH",    "@timestamp" => yyyy-MM-ddThh:mm:ss.SSSZ,          "host" => "<yourHostName>"}

Feedback

If you have any feedback on Java plugin support in Logstash, please comment on ourmain Github issue or post in theLogstash forum.