Logstash Reference [7.0] » Working with Logstash Modules » Logstash Netflow Module
«  Logstash ArcSight Module     Azure Module  »

Logstash Netflow Module

The Logstash Netflow module simplifies the collection, normalization, andvisualization of network flow data. With a single command, the module parsesnetwork flow data, indexes the events into Elasticsearch, and installs a suiteof Kibana dashboards to get you exploring your data immediately.

Logstash modules support Netflow Version 5 and 9.

What is Flow Data?

Netflow is a type of data record streamed from capable network devices. Itcontains information about connections traversing the device, and includessource IP addresses and ports, destination IP addresses and ports, types ofservice, VLANs, and other information that can be encoded into frame andprotocol headers. With Netflow data, network operators can go beyond monitoringsimply the volume of data crossing their networks. They can understand where thetraffic originated, where it is going, and what services or applications it ispart of.

Requirements

These instructions assume you have already installed Elastic Stack(Logstash, Elasticsearch, and Kibana) version 5.6 or higher. The products youneed are available to download and easy toinstall.

Getting Started

  1. Start the Logstash Netflow module by running the following command in theLogstash installation directory:

    bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=NNNN

    Where NNNN is the UDP port on which Logstash will listen for network trafficdata. If you don’t specify a port, Logstash listens on port 2055 by default.

    The --modules netflow option spins up a Netflow-aware Logstash pipelinefor ingestion.

    The --setup option creates a netflow-* index pattern in Elasticsearch andimports Kibana dashboards and visualizations. Running --setup is a one-timesetup step. Omit this option for subsequent runs of the module to avoidoverwriting existing Kibana dashboards.

    The command shown here assumes that you’re running Elasticsearch and Kibana onyour localhost. If you’re not, you need to specify additional connectionoptions. See Configuring the Module.

  2. Explore your data in Kibana:

    1. Open your browser and navigate tohttp://localhost:5601. If security is enabled, you’llneed to specify the Kibana username and password that you used when you set upsecurity.
    2. Open Netflow: Network Overview Dashboard.
    3. See Exploring Your Data for additional details on data exploration.

Exploring Your Data

Once the Logstash Netflow module starts processing events, you can immediatelybegin using the packaged Kibana dashboards to explore and visualize yournetwork flow data.

You can use the dashboards as-is, or tailor them to work better with existinguse cases and business requirements.

Example Dashboards

On the Overview dashboard, you can see a summary of basic traffic data and setup filters before you drill down to gain deeper insight into the data.

Netflow overview dashboard

For example, on the Conversation Partners dashboard, you can see the sourceand destination addresses of the client and server in any conversation.

Netflow conversation partners dashboard

On the Traffic Analysis dashboard, you can identify high volume conversationsby viewing the traffic volume in bytes.

Netflow traffic analysis dashboard

Then you can go to the Geo Location dashboard where you can visualize thelocation of destinations and sources on a heat map.

Netflow geo location dashboard

Configuring the Module

You can further refine the behavior of the Logstash Netflow module by specifyingsettings in the logstash.yml settings file, or overriding settings at thecommand line.

For example, the following configuration in the logstash.yml file setsLogstash to listen on port 9996 for network traffic data:

modules:  - name: netflow    var.input.udp.port: 9996

To specify the same settings at the command line, you use:

bin/logstash --modules netflow -M netflow.var.input.udp.port=9996

For more information about configuring modules, seeWorking with Logstash Modules.

Configuration Options

The Netflow module provides the following settings for configuring the behaviorof the module. These settings include Netflow-specific options plus commonoptions that are supported by all Logstash modules.

When you override a setting at the command line, remember to prefix the settingwith the module name, for example, netflow.var.input.udp.port instead ofvar.input.udp.port.

If you don’t specify configuration settings, Logstash uses the defaults.

Netflow Options

var.input.udp.port:
  • Value type is number
  • Default value is 2055.

Sets the UDP port on which Logstash listens for network traffic data. Although2055 is the default for this setting, some devices use ports in the range of9995 through 9998, with 9996 being the most commonly used alternative.

var.input.udp.workers:
  • Value type is number
  • Default value is 2.

Number of threads processing packets.

var.input.udp.receive_buffer_bytes:
  • Value type is number
  • Default value is 212992.

The socket receive buffer size in bytes.The operating system will use the max allowed value if receive_buffer_bytes is larger than allowed.Consult your operating system documentation if you need to increase this max allowed value.

var.input.udp.queue_size:
  • Value type is number
  • Default value is 2000.

This is the number of unprocessed UDP packets you can hold in memory beforepackets will start dropping.

Common options

The following configuration options are supported by all modules:

var.elasticsearch.hosts
  • Value type is uri
  • Default value is "localhost:9200"

Sets the host(s) of the Elasticsearch cluster. For each host, you must specifythe hostname and port. For example, "myhost:9200". If given an array,Logstash will load balance requests across the hosts specified in the hostsparameter. It is important to exclude dedicated masternodes from the hosts list to prevent Logstash from sending bulk requests to themaster nodes. So this parameter should only reference either data or clientnodes in Elasticsearch.

Any special characters present in the URLs here MUST be URL escaped! This means #should be put in as %23 for instance.

var.elasticsearch.username
  • Value type is string
  • Default value is "elastic"

The username to authenticate to a secure Elasticsearch cluster.

var.elasticsearch.password
  • Value type is string
  • Default value is "changeme"

The password to authenticate to a secure Elasticsearch cluster.

var.elasticsearch.ssl.enabled
  • Value type is boolean
  • There is no default value for this setting.

Enable SSL/TLS secured communication to the Elasticsearch cluster. Leaving thisunspecified will use whatever scheme is specified in the URLs listed in hosts.If no explicit protocol is specified, plain HTTP will be used. If SSL isexplicitly disabled here, the plugin will refuse to start if an HTTPS URL isgiven in hosts.

var.elasticsearch.ssl.verification_mode
  • Value type is string
  • Default value is "strict"

The hostname verification setting when communicating with Elasticsearch. Set todisable to turn off hostname verification. Disabling this has serious securityconcerns.

var.elasticsearch.ssl.certificate_authority
  • Value type is string
  • There is no default value for this setting

The path to an X.509 certificate to use to validate SSL certificates whencommunicating with Elasticsearch.

var.elasticsearch.ssl.certificate
  • Value type is string
  • There is no default value for this setting

The path to an X.509 certificate to use for client authentication whencommunicating with Elasticsearch.

var.elasticsearch.ssl.key
  • Value type is string
  • There is no default value for this setting

The path to the certificate key for client authentication when communicatingwith Elasticsearch.

var.kibana.host
  • Value type is string
  • Default value is "localhost:5601"

Sets the hostname and port of the Kibana instance to use for importingdashboards and visualizations. For example: "myhost:5601".

var.kibana.scheme
  • Value type is string
  • Default value is "http"

Sets the protocol to use for reaching the Kibana instance. The options are:"http" or "https". The default is "http".

var.kibana.username
  • Value type is string
  • Default value is "elastic"

The username to authenticate to a secured Kibana instance.

var.kibana.password
  • Value type is string
  • Default value is "changeme"

The password to authenticate to a secure Kibana instance.

var.kibana.ssl.enabled
  • Value type is boolean
  • Default value is false

Enable SSL/TLS secured communication to the Kibana instance.

var.kibana.ssl.verification_mode
  • Value type is string
  • Default value is "strict"

The hostname verification setting when communicating with Kibana. Set todisable to turn off hostname verification. Disabling this has serious securityconcerns.

var.kibana.ssl.certificate_authority
  • Value type is string
  • There is no default value for this setting

The path to an X.509 certificate to use to validate SSL certificates whencommunicating with Kibana.

var.kibana.ssl.certificate
  • Value type is string
  • There is no default value for this setting

The path to an X.509 certificate to use for client authentication whencommunicating with Kibana.

var.kibana.ssl.key
  • Value type is string
  • There is no default value for this setting

The path to the certificate key for client authentication when communicatingwith Kibana.

«  Logstash ArcSight Module     Azure Module  »