Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

Select Status

RSS

elk_forwarder (5) Versions 3.0.1

Installs logstash-forwarder for use in an ELK Cluster

Policyfile
Berkshelf
Knife
cookbook 'elk_forwarder', '= 3.0.1', :supermarket
cookbook 'elk_forwarder', '= 3.0.1'
knife supermarket install elk_forwarder
knife supermarket download elk_forwarder
README
Dependencies
Changelog
Quality 0%

elk_forwarder Build Status Dependency Status

Installs and configures the logstash-forwarder to forward specified logs to specified servers

Requirements

  • golang cookbook

Attributes

You can also find comments in attributes/default.rb

General Attributes

<table>
<tr>
<th>Key</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
<tr>
<td><tt>['elk_forwarder']['install_type']</tt></td>
<td>String</td>
<td>source or package</td>
<td><tt>package</tt></td>
</tr>
<tr>
<td><tt>['elk_forwarder']['config_dir']</tt></td>
<td>String</td>
<td>Where to put config</td>
<td><tt>/etc/logstash-forwarder</tt></td>
</tr>
<tr>
<td><tt>['elk_forwarder']['service_name']</tt></td>
<td>String</td>
<td>The Service Name</td>
<td><tt>logstash-forwarder</tt></td>
</tr>
<tr>
<td><tt>['elk_forwarder']['log_dir']</tt></td>
<td>String</td>
<td>Directory to log to</td>
<td><tt>/var/log/logstash-forwarder</tt></td>
</tr>
<tr>
<td><tt>['elk_forwarder']['install_dir']</tt></td>
<td>String</td>
<td>Directory to install to</td>
<td><tt>/opt/logstash-forwarder</tt></td>
</tr>
<tr>
<td><tt>['elk_forwarder']['daemon_args']</tt></td>
<td>String</td>
<td>Extra args for the forwarder</td>
<td><tt>-spool-size 5</tt></td>
</tr>
<tr>
<td><tt>['elk_forwarder']['syslog']['facility']</tt></td>
<td>String</td>
<td>The Syslog facility.</td>
<td><tt>local0</tt></td>
</tr>
</table>

Config File Attributes

The node['elk_forwarder']['config'] hash closely mimics the logstash forwarder config file format, with the only difference of the files key contains a hash instead of an array

The following table is namespaced under node['elk_forwarder']['config'] so prepend node['elk_forwarder']['config'] to the key column
<table>
<tr>
<td><tt>['network']['servers']</tt></td>
<td>Array</td>
<td>An array of logstash agent address:port values</td>
<td><tt>[]</tt></td>
</tr>
<tr>
<td><tt>['network']['ssl certificate']</tt></td>
<td>String</td>
<td>The path to find the SSL Certificate</td>
<td>/etc/pki/tls/certs/logstash-forwarder/cert.pem</td>
</tr>
<tr>
<td><tt>['network']['ssl certificate']</tt></td>
<td>String</td>
<td>The path to find the SSL Certificate</td>
<td>/etc/pki/tls/certs/logstash-forwarder/cert.pem</td>
</tr>
<tr>
<td><tt>['network']['ssl key']</tt></td>
<td>String</td>
<td>The path to find the SSL Private Key</td>
<td>/etc/pki/tls/certs/logstash-forwarder/key.pem</td>
</tr>
<tr>
<td><tt>['network']['ssl ca']</tt></td>
<td>String</td>
<td>The path to find the SSL CA Certificate</td>
<td>/etc/pki/tls/certs/logstash-forwarder/ca.pem</td>
</tr>
<tr>
<td><tt>['network']['timeout']</tt></td>
<td>String</td>
<td>Seconds to wait before connecting to next server</td>
<td>15</td>
</tr>
<tr>
<td><tt>['haproxy']['defaults']['maxconn']</tt></td>
<td>String (well, a number in quotes, really)</td>
<td>The DEFAULT maximum number of concurrent connections the frontend will accept to serve. Excess connections will be queued by the system in the socket's listen queue and will be served once a connection closes. (See: http://code.google.com/p/haproxy-docs/wiki/maxconn)</td>
<td><tt>60000</tt></td>
</tr>
<tr>
<td><tt>['files']</tt></td>
<td>Hash</td>
<td>The List of files to track and associated fields to add</td>
<td>{}</td>
</tr>
</table>

Usage

Thats great but how do I use it.

Point to your servers

Put this in a recipe, probably in your base cookbook

servers = search(
  :node,
  "role:elk_server AND chef_environment:#{node.chef_environment}"
).map do |node|
  "#{node['fqdn']}:5043"
end

node.set['elk_forwarder']['config']['network']['servers'] = servers

Grab your Lumberjack Certificate

This one is pretty much up to you, but there is a built in recipe for this
that pulls a certificate from a data bag item. I will use that as an example
assuming that you stored your CA certificate in the ca certificate key in the
logstash data bag item in the certificates data bag

In your Attributes file

set['elk_forwarder']['certs']['ca_data_bag'] = 'certificates'
set['elk_forwarder']['certs']['ca_data_bag_item'] = 'logstash'
set['elk_forwarder']['certs']['ca_data_bag_item_key'] = 'ca certificate'

In your Recipe

include_recipe 'elk_forwarder::certs'

Configure files to forward

From your Attributes

set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log']
set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp'
set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'

Or from your recipe. This one loops through a list of apps

apps.each do |app|
  node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"]
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'rails_app'
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['app'] = app
end

As a side note all attribute modifications need to happen at compile time

Recipes

default

Installs and configures the logstash-forwarder

  1. Install logstash-forwarder using the _source or _package recipes
  2. Include various recipes for this cookbook:
    • elk_forwarder::configure

configure

Configures the forwarder with the ['elk_forwarder']['config'] hash

  1. Creates the config file: #{node['elk_forwarder']['config_dir']}/logstash-forwarder.conf

certs

Installs SSL Certs and Keys from data bags to the paths specified in the configuration
from:

  • node['elk_forwarder']['network']['ssl ca']
  • node['elk_forwarder']['network']['ssl certificate']
  • node['elk_forwarder']['network']['ssl key']

The following attributes are used to determine the location of the certs/keys

# The Server's CA Certificate. This cert is required
set['elk_forwarder']['certs']['ca_data_bag'] = 'certificates'
set['elk_forwarder']['certs']['ca_data_bag_item'] = 'logstash'
set['elk_forwarder']['certs']['ca_data_bag_item_key'] = 'ca certificate'

# The Client Certificate (optional)
set['elk_forwarder']['certs']['certificate_data_bag'] = 'certificates'
set['elk_forwarder']['certs']['certificate_data_bag_item'] = 'logstash'
set['elk_forwarder']['certs']['certificate_data_bag_item_key'] = 'ca certificate'

# The Client key (optional)
set['elk_forwarder']['certs']['key_data_bag'] = 'certificates'
set['elk_forwarder']['certs']['key_data_bag_item'] = 'logstash'
set['elk_forwarder']['certs']['key_data_bag_item_key'] = 'ca certificate'

Usage

Include this recipe in a wrapper cookbook:

metadata.rb

depends 'elk_forwarder', '~> 1.0'

recipes/your_recipe.rb

include_recipe 'elk_forwarder::default'

Certificates

Generating and distributing SSL Certificates is out of scope for this cookbook. Your wrapper cookbook will
need to configure the SSL Certificates and Keys. See the
Certificate Notes
on the logstash forwarder repo for help.

However, with that said there is a certs recipe that you can use at your own risk to pull certs from a data bag.

You can tweak the Certificate locations in the [elk_forwarder]['config']['network']['ssl *'] attributes

Contributing

  1. Fork the repository on Github
  2. Create a named feature branch (like add_component_x)
  3. Write your change
  4. Write tests for your change (if applicable)
  5. Run the tests with kitchen test, ensuring they all pass
  6. Submit a Pull Request using Github

License and Authors

Author:: EverTrue, inc. (devops@evertrue.com)

Change Log for elk_forwarder

All notable changes to this project will be documented in this file.
This project adheres to Semantic Versioning.

Unreleased

Added

Changed

Removed

[3.0.1] - 2015-08-07

Fixed

  • Fix certificate miscommunication

[3.0.0] - 2015-08-07

Changed

  • Massive refactoring (much of attribute names changed)
  • Change test environment name from _default to dev
  • Stop testing on Ubuntu 12.04

[2.0.0] - 2015-06-16

Changed

  • Breaking: All Configuration to just be through attributes, no LWRP unfortunately
  • Breaking: The ['config']['files'] attribute has been changed from an array to a hash
  • Added a mocking mode for running tests

Removed

  • The logstash_forwarder_log LWRP as it was causing problematic race conditions

1.0.0 - 2015-04-27

Added

  • Initial Release

Foodcritic Metric
            

3.0.1 failed this metric

FC031: Cookbook without metadata file: /tmp/cook/9beb9f4ba90ff734e0b4c81a/elk_forwarder/metadata.rb:1
FC045: Consider setting cookbook name in metadata: /tmp/cook/9beb9f4ba90ff734e0b4c81a/elk_forwarder/metadata.rb:1