This section discusses the information you need to know to successfully contribute a patch to a Logstash plugin.
Each plugin defines its own configuration options. These control the behaviour of the plugin to some degree. Configurationoption definitions commonly include:
Plugins are subclasses of a Logstash base class. A plugin’s base class defines common configuration and methods.
Input plugins ingest data from an external source. Input plugins are always associated with a codec. An input pluginalways has an associated codec plugin. Input and codec plugins operate in conjunction to create a Logstash event and addthat event to the processing queue. An input codec is a subclass of the LogStash::Inputs::Base
class.
Table 2. Input API
|
Required. This API sets up resources for the plugin, typically the connection to theexternal source. |
|
Required. This API fetches or listens for source data, typically looping until stopped. Must handleerrors inside the loop. Pushes any created events to the queue object specified in the method argument. Some inputs mayreceive batched data to minimize the external call overhead. |
|
Optional. Stops external connections and cleans up. |
Codec plugins decode input data that has a specific structure, such as JSON input data. A codec plugin is a subclass ofLogStash::Codecs::Base
.
Table 3. Codec API
|
Identical to the API of the same name for input plugins. |
|
Must be implemented. Used to create an Event from the raw data given in the methodargument. Must handle errors. The caller must provide a Ruby block. The block is called with the created Event. |
|
Required. Used to create a structured data object from the given Event. May handleerrors. This method calls a block that was previously stored as @on_event with two arguments: the original event and thedata object. |
A mechanism to change, mutate or merge one or more Events. A filter plugin is a subclass of the LogStash::Filters::Base
class.
Table 4. Filter API
|
Identical to the API of the same name for input plugins. |
|
Required. May handle errors. Used to apply a mutation function to the given event. |
A mechanism to send an event to an external destination. This process may require serialization. An output plugin is asubclass of the LogStash::Outputs::Base
class.
Table 5. Output API
|
Identical to the API of the same name for input plugins. |
|
Required. Must handle errors. Used to prepare the given event for transmission tothe external destination. Some outputs may buffer the prepared events to batch transmit to the destination. |
A bug or feature is identified. An issue is created in the plugin repository. A patch is created and a pull request (PR)is submitted. After review and possible rework the PR is merged and the plugin is published.
The Community Maintainer Guide explains, in more detail, the process of getting a patch accepted,merged and published. The Community Maintainer Guide also details the roles that contributors and maintainers areexpected to perform.
Test Driven Development, colloquially known as TDD, describes a methodology for using tests to guide evolution of sourcecode. For our purposes, we are only going to use a part of it, that is, before writing the fix - we create tests thatillustrate the bug by failing. We stop when we have written enough code to make the tests pass and submit the fix andtests as a patch. It is not necessary to write the tests before the fix, but it is very easy to write a passing testafterwards that may not actually verify that the fault is really fixed especially if the fault can be triggered viamultiple execution paths or varying input data.
Logstash uses Rspec, a Ruby testing framework, to define and run the test suite. What follows is a summary of varioussources.
Rspec Example
1 # encoding: utf-8 2 require "logstash/devutils/rspec/spec_helper" 3 require "logstash/plugin" 4 5 describe "outputs/riemann" do 6 describe "#register" do 7 let(:output) do 8 LogStash::Plugin.lookup("output", "riemann").new(configuration) 9 end1011 context "when no protocol is specified" do12 let(:configuration) { Hash.new }1314 it "the method completes without error" do15 expect {output.register}.not_to raise_error16 end17 end1819 context "when a bad protocol is specified" do20 let(:configuration) { {"protocol" => "fake"} }2122 it "the method fails with error" do23 expect {output.register}.to raise_error24 end25 end2627 context "when the tcp protocol is specified" do28 let(:configuration) { {"protocol" => "tcp"} }2930 it "the method completes without error" do31 expect {output.register}.not_to raise_error32 end33 end34 end3536 describe "#receive" do37 let(:output) do38 LogStash::Plugin.lookup("output", "riemann").new(configuration)39 end4041 context "when operating normally" do42 let(:configuration) { Hash.new }43 let(:event) do44 data = {"message"=>"hello", "@version"=>"1",45 "@timestamp"=>"2015-06-03T23:34:54.076Z",46 "host"=>"vagrant-ubuntu-trusty-64"}47 LogStash::Event.new(data)48 end4950 before(:example) do51 output.register52 end5354 it "should accept the event" do55 expect { output.receive event }.not_to raise_error56 end57 end58 end59 end
Describe blocks (lines 5, 6 and 36 in Example 1).
describe(string){block} -> nildescribe(Class){block} -> nil
With RSpec, we are always describing the plugin method behavior. The describe block is added in logical sections and canaccept either an existing class name or a string. The string used in line 5 is the plugin name. Line 6 is the registermethod, line 36 is the receive method. It is a RSpec convention to prefix instance methods with one hash and classmethods with one dot.
Context blocks (lines 11, 19, 27 and 41).
context(string){block} -> nil
In RSpec, context blocks define sections that group tests by a variation. The string should start with the word when
and then detail the variation. See line 11. The tests in the content block should should only be for that variation.
Let blocks (lines 7, 12, 20, 28, 37, 42 and 43).
let(symbol){block} -> nil
In RSpec, let
blocks define resources for use in the test blocks. These resources are reinitialized for every testblock. They are available as method calls inside the test block. Define let
blocks in describe
and context
blocks,which scope the let
block and any other nested blocks.You can use other let
methods defined later within the let
block body. See lines 7-9, which define the output resourceand use the configuration method, defined with different variations in lines 12, 20 and 28.
Before blocks (line 50).
before(symbol){block} -> nil - symbol is one of :suite, :context, :example, but :all and :each are synonyms for :suite and :example respectively.
In RSpec, before
blocks are used to further set up any resources that would have been initialized in a let
block.You cannot define let
blocks inside before
blocks.
You can also define after
blocks, which are typically used to clean up any setup activity performed by a before
block.
It blocks (lines 14, 22, 30 and 54).
it(string){block} -> nil
In RSpec, it
blocks set the expectations that verify the behavior of the tested code. The string should not start withit or should, but needs to express the outcome of the expectation. When put together the texts from the enclosingdescribe, context
and it
blocks should form a fairly readable sentence, as in lines 5, 6, 11 and 14:
outputs/riemann#register when no protocol is specified the method completes without error
Readable code like this make the goals of tests easy to understand.
Expect method (lines 15, 23, 31, 55).
expect(object){block} -> nil
In RSpec, the expect method verifies a statement that compares an actual result to an expected result. The expect
methodis usually paired with a call to the to
or not_to
methods. Use the block form when expecting errors or observing forchanges. The to
or not_to
methods require a matcher
object that encapsulates the expected value. The argument formof the expect
method encapsulates the actual value. When put together the whole line tests the actual against theexpected value.
Matcher methods (lines 15, 23, 31, 55).
raise_error(error class|nil) -> matcher instancebe(object) -> matcher instanceeq(object) -> matcher instanceeql(object) -> matcher instance for more see http://www.relishapp.com/rspec/rspec-expectations/docs/built-in-matchers
In RSpec, a matcher is an object generated by the equivalent method call (be, eq) that will be used to evaluate theexpected against the actual values.
This example fixes an issue in the ZeroMQ outputplugin. The issue does not require knowledge of ZeroMQ.
The activities in this example have the following prerequisites:
chruby
tool manages Ruby versions.bundler
and rake
gems installed.logstash/
.Open the following files in a text editor:
logstash-output-zeromq/lib/logstash/outputs/zeromq.rb
logstash-output-zeromq/lib/logstash/util/zeromq.rb
logstash-output-zeromq/spec/outputs/zeromq_spec.rb
According to the issue, log output in server mode must indicate bound
. Furthermore, the test file contains no tests.
Line 21 of util/zeromq.rb
reads @logger.info("0mq: #{server? ? 'connected' : 'bound'}", :address => address)
In the text editor, set the file encoding and require zeromq.rb
for the file zeromq_spec.rb
by adding the followinglines:
# encoding: utf-8require "logstash/outputs/zeromq"require "logstash/devutils/rspec/spec_helper"
The desired error message should read:
LogStash::Outputs::ZeroMQ when in server mode a 'bound' info line is logged
To properly generate this message, add a describe
block with the fully qualified class name as the argument, a contextblock, and an it
block.
describe LogStash::Outputs::ZeroMQ do context "when in server mode" do it "a 'bound' info line is logged" do end endend
To add the missing test, use an instance of the ZeroMQ output and a substitute logger. This example uses an RSpec featurecalled test doubles as the substitute logger.
Add the following lines to zeromq_spec.rb
, after describe LogStash::Outputs::ZeroMQ do
and before context "when inserver mode" do
:
let(:output) { described_class.new("mode" => "server", "topology" => "pushpull" }let(:tracer) { double("logger") }
Add the body to the it
block. Add the following five lines after the line context "when in server mode" do
:
At the end of the modifications, the relevant code section reads:
# encoding: utf-8require "logstash/outputs/zeromq"require "logstash/devutils/rspec/spec_helper"describe LogStash::Outputs::ZeroMQ do let(:output) { described_class.new("mode" => "server", "topology" => "pushpull") } let(:tracer) { double("logger") } context "when in server mode" do it "a ‘bound’ info line is logged" do allow(tracer).to receive(:debug) output.logger = tracer expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"}) output.register output.do_close end endend
To run this test:
bundle install
bundle exec rspec
Assuming all prerequisites were installed correctly, the test fails with output similar to:
Using Accessor#strict_set for specsRun options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true,:elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged (FAILED - 1)Failures: 1) LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged Failure/Error: output.register Double "logger" received :info with unexpected arguments expected: ("0mq: bound", {:address=>"tcp://127.0.0.1:2120"}) got: ("0mq: connected", {:address=>"tcp://127.0.0.1:2120"}) # ./lib/logstash/util/zeromq.rb:21:in `setup' # ./lib/logstash/outputs/zeromq.rb:92:in `register' # ./lib/logstash/outputs/zeromq.rb:91:in `register' # ./spec/outputs/zeromq_spec.rb:13:in `(root)' # /Users/guy/.gem/jruby/1.9.3/gems/rspec-wait-0.0.7/lib/rspec/wait.rb:46:in `(root)'Finished in 0.133 seconds (files took 1.28 seconds to load)1 example, 1 failureFailed examples:rspec ./spec/outputs/zeromq_spec.rb:10 # LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is loggedRandomized with seed 2568
To correct the error, open the util/zeromq.rb
file in your text editor and swap the positions of the words connected
and bound
on line 21. Line 21 now reads:
@logger.info("0mq: #{server? ? 'bound' : 'connected'}", :address => address)
Run the test again with the bundle exec rspec
command.
The test passes with output similar to:
Using Accessor#strict_set for specsRun options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is loggedFinished in 0.114 seconds (files took 1.22 seconds to load)1 example, 0 failuresRandomized with seed 45887
Commit the changes to git and Github.
Your pull request is visible from the Pull Requestssection of the original Github repository. The plugin maintainers review your work, suggest changes if necessary, andmerge and publish a new version of the plugin.