Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

Select Status

RSS

elk_forwarder (5) Versions 4.0.0

Installs logstash-forwarder for use in an ELK Cluster

Policyfile
Berkshelf
Knife
cookbook 'elk_forwarder', '~> 4.0.0', :supermarket
cookbook 'elk_forwarder', '~> 4.0.0'
knife supermarket install elk_forwarder
knife supermarket download elk_forwarder
README
Dependencies
Changelog
Quality 75%

elk_forwarder Build Status Dependency Status

Installs and configures the logstash-forwarder to forward specified logs to specified servers

Requirements

  • golang cookbook

Attributes

You can also find comments in attributes/default.rb

General Attributes

Key Type Description Default
['elk_forwarder']['install_type'] String source or package package
['elk_forwarder']['config_dir'] String Where to put config /etc/logstash-forwarder
['elk_forwarder']['service_name'] String The Service Name logstash-forwarder
['elk_forwarder']['log_dir'] String Directory to log to /var/log/logstash-forwarder
['elk_forwarder']['install_dir'] String Directory to install to /opt/logstash-forwarder
['elk_forwarder']['daemon_args'] String Extra args for the forwarder -spool-size 5
['elk_forwarder']['syslog']['facility'] String The Syslog facility. local0

Config File Attributes

The node['elk_forwarder']['config'] hash closely mimics the logstash forwarder config file format, with the only difference of the files key contains a hash instead of an array

The following table is namespaced under node['elk_forwarder']['config'] so prepend node['elk_forwarder']['config'] to the key column

Key Type Description Default
['network']['servers'] Array An array of logstash agent address:port values []
['network']['ssl certificate'] String The path to find the SSL Certificate /etc/pki/tls/certs/logstash-forwarder/cert.pem
['network']['ssl key'] String The path to find the SSL Private Key /etc/pki/tls/certs/logstash-forwarder/key.pem
['network']['ssl ca'] String The path to find the SSL CA Certificate /etc/pki/tls/certs/logstash-forwarder/ca.pem
['network']['timeout'] String Seconds to wait before connecting to next server 15
['files'] Hash The List of files to track and associated fields to add {}

The node['elk_forwarder']['config']['files'] hash is probably the most useful,
check out the Usage section for instructions on how to configure files to forward

Usage

Thats great but how do I use it.

Point to your servers

Put this in a recipe, probably in your base cookbook

servers = search(
  :node,
  "role:elk_server AND chef_environment:#{node.chef_environment}"
).map do |node|
  "#{node['fqdn']}:5043"
end

node.set['elk_forwarder']['config']['network']['servers'] = servers

Grab your Lumberjack Certificate

This one is pretty much up to you, but there is a built in recipe for this
that pulls a certificate from a data bag item. I will use that as an example
assuming that you stored your CA certificate in the ca certificate key in the
logstash data bag item in the certificates data bag

In your Attributes file

set['elk_forwarder']['certs']['ca_data_bag'] = 'certificates'
set['elk_forwarder']['certs']['ca_data_bag_item'] = 'logstash'
set['elk_forwarder']['certs']['ca_data_bag_item_key'] = 'ca certificate'

In your Recipe

include_recipe 'elk_forwarder::certs'

Configure files to forward

From your Attributes

set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log']
set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp'
set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'

Or from your recipe. This one loops through a list of apps

apps.each do |app|
  node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"]
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'rails_app'
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['app'] = app
end

As a side note all attribute modifications need to happen at compile time

Recipes

default

Installs and configures the logstash-forwarder

  1. Install logstash-forwarder using the _source or _package recipes
  2. Include various recipes for this cookbook:
    • elk_forwarder::configure

configure

Configures the forwarder with the ['elk_forwarder']['config'] hash

  1. Creates the config file: #{node['elk_forwarder']['config_dir']}/logstash-forwarder.conf

certs

Installs SSL Certs and Keys from data bags to the paths specified in these atts:

  • node['elk_forwarder']['network']['ssl ca']
  • node['elk_forwarder']['network']['ssl certificate']
  • node['elk_forwarder']['network']['ssl key']

The following attributes are used to determine the location of the certs/keys

default['elk_forwarder']['cert_data_bag']      = 'certificates'
default['elk_forwarder']['cert_data_bag_item'] = 'logstash'

The data bag you specify is expected to be in the following format:

{
  "id": "elk_forwarder",
  "data": {
    "ca": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n",
    "key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n"
  }
}

Usage

Include this recipe in a wrapper cookbook:

metadata.rb

depends 'elk_forwarder', '~> 1.0'

recipes/your_recipe.rb

include_recipe 'elk_forwarder::default'

Certificates

Generating and distributing SSL Certificates is out of scope for this cookbook
unless you want to try out the certs recipe which just distributes certs from a
data bag. If you want to take care of certs yourself then your wrapper cookbook
will need to configure the SSL Certificates and Keys. See the
Certificate Notes
on the logstash forwarder repo for help.

Remember to set these attributes to the filenames where you store the certs:

  • node['elk_forwarder']['network']['ssl ca']
  • node['elk_forwarder']['network']['ssl certificate'] (optional)
  • node['elk_forwarder']['network']['ssl key'] (optional)

Contributing

  1. Fork the repository on Github
  2. Create a named feature branch (like add_component_x)
  3. Write your change
  4. Write tests for your change (if applicable)
  5. Run the tests with kitchen test, ensuring they all pass
  6. Submit a Pull Request using Github

License and Authors

Author:: EverTrue, inc. (devops@evertrue.com)

Change Log for elk_forwarder

All notable changes to this project will be documented in this file.
This project adheres to Semantic Versioning.

Unreleased

Added

Changed

Removed

[4.0.0] - 2015-12-23

Fixed

  • Remove key and certificate code (now in evertrue/logserver-cookbook) and just install the CA cert
  • Install elk_forwarder::certs as part of configure

[3.0.2] - 2015-11-05

Fixed

  • Build path to be in file_cache_path

[3.0.1] - 2015-08-07

Fixed

  • Fix certificate miscommunication

[3.0.0] - 2015-08-07

Changed

  • Massive refactoring (much of attribute names changed)
  • Change test environment name from _default to dev
  • Stop testing on Ubuntu 12.04

[2.0.0] - 2015-06-16

Changed

  • Breaking: All Configuration to just be through attributes, no LWRP unfortunately
  • Breaking: The ['config']['files'] attribute has been changed from an array to a hash
  • Added a mocking mode for running tests

Removed

  • The logstash_forwarder_log LWRP as it was causing problematic race conditions

1.0.0 - 2015-04-27

Added

  • Initial Release

Collaborator Number Metric
            

4.0.0 passed this metric

Contributing File Metric
            

4.0.0 passed this metric

Foodcritic Metric
            

4.0.0 passed this metric

Testing File Metric
            

4.0.0 failed this metric

Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file