logstash_lwrp chef cookbook

A resource-driven Chef cookbook for installing and running Logstash on GNU/Linux systems.

Provides resources for installing, configuring and running the currently supported versions of the Logstash application.

Installation of Logstash prerequisites such as Java are out-of-scope for this cookbook

Supported Software / Platforms

Platforms

• Debian / Ubuntu
• RHEL / CentOS

Chef

• Chef 13+

Logstash

• 5.x
• 6.x

Usage

This cookbook provides resources to manage Logstash including ones to install the application, create base configuration, setup processing pipelines and managing the service. In general, a normal deployment will need to use one of each resource to have a fully functioning Logstash install.

Example

logstash_install 'example' do
  action :install
end

logstash_config 'example' do
  action :create
end

logstash_pipeline 'main' do
  instance 'example'
  config_string "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
  pipeline_workers 1
  action :create
end

logstash_service 'example' do
  xms '256M'
  xmx '256M'
  action [:start, :enable]
end

More complex examples can be found in the test cookbook recipe: https://github.com/petracvv/logstash_lwrp/blob/master/test/fixtures/cookbooks/logstash_lwrp_test/recipes/default.rb

Resources

logstash_install

logstash_install installs an instance of the Logstash application using the official tar package found on Elastic's main site. At the moment, only one instance of Logstash per node is officially supported but it is possible to have multiple instances if they are using different Logstash versions.

Actions

• :install: Installs an instance of the logstash application
• :upgrade: Upgrades an instance of the logstash application

Properties

Examples

Install a Logstash 6.4.3 instance named 'testing' to /opt/logstash_testing_6_4_3/ with a symlink at /opt/logstash_testing

logstash_install 'testing' do
  version '6.4.3'
  action :install
end

logstash_config

Installs the base Logstash configuration into the logstath.yml file. If you are using a Logstash 5.x version without multiple pipeline support, you will need to add your pipeline configuration with this resource instead of the logstash_pipeline resource.

Actions

• :create: Deploys the configuration to the logstash.yml file of the named Logstash instance

Properties

Examples

Create a logstash configuration with a custom node name for the 'testing' instance

logstash_config 'testing' do
  node_name 'logstash-test'
  action :create
end

Deploy the main pipeline for a 'testing' Logstash 5.x install. This should only be done in Logstash 5.x installs since it does not support pipelines.yml

logstash_config 'testing' do
  node_name 'logstash-5-test'
  pipeline_defaults(
    'pipeline.id' => 'main',
    'path.config' => '/etc/mycustomlogstash/'
  )
  action :create
end

logstash_pipeline

This resource deploys a pipeline to the pipelines.yml config file for Logstash versions that support it.

There are two main ways to deploy pipeline configuration in Logstash:

• Using a config.string where all your config is defined in one string in pipelines.yml
• Using a path.config where logstash looks for configuration files in the target path specified.

If supplied with a config_string property, logstash_pipeline will deploy a pipeline with the configuration in the config.string logstash property

If supplied with a config_templates property, logstash_pipeline will create a per-pipeline directory in the Logstash install location and deploy the logstash configuration files specified in the property. It will then set the path.config Logstash property to the created directory. The specified templates are always deployed from the wrapper cookbook (i.e. where the logstash_pipeline resource is defined)

Actions

• :create: Deploy a the pipelines.yml with the pipeline config specified

Properties

Examples

Deploy a short pipeline configuration for the 'testing' Logstash install.

logstash_pipeline 'string' do
  instance 'testing'
  pipeline_workers 1
  config_string "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
  action :create
end

Deploy a more complicated Logstash pipeline for the 'testing' Logstash instance using templates and variables in your wrapper cookbook

logstash_pipeline 'main' do
  instance 'testing'
  config_templates %w( input_syslog.conf.erb output_elasticsearch.conf.erb output_stdout.conf.erb)
  config_template_variables(
    'test.id' => '321',
    'test.name' => 'Variables'
  )
  action :create
end

logstash_service

This resource sets up the Logstash system service and sets the runtime Java options for the service. The service is generated using a provided upstream script; therefore, at this moment there is no way to edit the service file itself through this resource.

Actions

• :start: Creates the service and starts it
• :stop: Stops the service
• :restart: Restarts the service
• :enable: Creates the service and enables it (starts on boot)

Properties

Examples

Install and start the Logstash service for the 'testing' instance with 256M as the memory limit

logstash_service 'testing' do
  xms '256M'
  xmx '256M'
  action [:start, :enable]
end

Acknowledgements

Thanks to:

• The tomcat cookbook authors for inspiration for the logstash_install resource pattern.
• The haproxy cookbook authors for great examples of the Chef accumulator pattern

License & Authors

• Author: Mihai Petracovici (https://github.com/petracvv)

Copyright:: 2018, Mihai Petracovici

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.