elk_forwarder

Installs and configures the logstash-forwarder to forward specified logs to specified servers

Requirements

• golang cookbook

Attributes

You can also find comments in attributes/default.rb

General Attributes

Config File Attributes

The node['elk_forwarder']['config'] hash closely mimics the logstash forwarder config file format, with the only difference of the files key contains a hash instead of an array

The following table is namespaced under node['elk_forwarder']['config'] so prepend node['elk_forwarder']['config'] to the key column

The node['elk_forwarder']['config']['files'] hash is probably the most useful,
check out the Usage section for instructions on how to configure files to forward

Usage

Thats great but how do I use it.

Point to your servers

Put this in a recipe, probably in your base cookbook

servers = search(
  :node,
  "role:elk_server AND chef_environment:#{node.chef_environment}"
).map do |node|
  "#{node['fqdn']}:5043"
end

node.set['elk_forwarder']['config']['network']['servers'] = servers

Grab your Lumberjack Certificate

This one is pretty much up to you, but there is a built in recipe for this
that pulls a certificate from a data bag item. I will use that as an example
assuming that you stored your CA certificate in the ca certificate key in the
logstash data bag item in the certificates data bag

In your Attributes file

set['elk_forwarder']['certs']['ca_data_bag'] = 'certificates'
set['elk_forwarder']['certs']['ca_data_bag_item'] = 'logstash'
set['elk_forwarder']['certs']['ca_data_bag_item_key'] = 'ca certificate'

In your Recipe

include_recipe 'elk_forwarder::certs'

Configure files to forward

From your Attributes

set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log']
set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp'
set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'

Or from your recipe. This one loops through a list of apps

apps.each do |app|
  node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"]
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'rails_app'
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['app'] = app
end

As a side note all attribute modifications need to happen at compile time

Recipes

default

Installs and configures the logstash-forwarder

1. Install logstash-forwarder using the _source or _package recipes
2. Include various recipes for this cookbook:
   • elk_forwarder::configure

configure

Configures the forwarder with the ['elk_forwarder']['config'] hash

1. Creates the config file: #{node['elk_forwarder']['config_dir']}/logstash-forwarder.conf

certs

Installs SSL Certs and Keys from data bags to the paths specified in these atts:

• node['elk_forwarder']['network']['ssl ca']
• node['elk_forwarder']['network']['ssl certificate']
• node['elk_forwarder']['network']['ssl key']

The following attributes are used to determine the location of the certs/keys

default['elk_forwarder']['cert_data_bag']      = 'certificates'
default['elk_forwarder']['cert_data_bag_item'] = 'logstash'

The data bag you specify is expected to be in the following format:

{
  "id": "elk_forwarder",
  "data": {
    "ca": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n",
    "key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n"
  }
}

Usage

Include this recipe in a wrapper cookbook:

metadata.rb

depends 'elk_forwarder', '~> 1.0'

recipes/your_recipe.rb

include_recipe 'elk_forwarder::default'

Certificates

Generating and distributing SSL Certificates is out of scope for this cookbook
unless you want to try out the certs recipe which just distributes certs from a
data bag. If you want to take care of certs yourself then your wrapper cookbook
will need to configure the SSL Certificates and Keys. See the
Certificate Notes
on the logstash forwarder repo for help.

Remember to set these attributes to the filenames where you store the certs:

• node['elk_forwarder']['network']['ssl ca']
• node['elk_forwarder']['network']['ssl certificate'] (optional)
• node['elk_forwarder']['network']['ssl key'] (optional)

Contributing

1. Fork the repository on Github
2. Create a named feature branch (like add_component_x)
3. Write your change
4. Write tests for your change (if applicable)
5. Run the tests with kitchen test, ensuring they all pass
6. Submit a Pull Request using Github

License and Authors

Author:: EverTrue, inc. (devops@evertrue.com)