Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

RSS

et_elk (4) Versions 4.0.0

Configures an Elk Cluster

Berkshelf/Librarian
Policyfile
Knife
cookbook 'et_elk', '= 4.0.0'
cookbook 'et_elk', '= 4.0.0', :supermarket
knife cookbook site install et_elk
knife cookbook site download et_elk
README
Dependencies
Changelog
Quality

ELK Build Status

ELK

This cookbook is the top level wrapper for the EverTrue ELK cookbook ecosystem

Requirements

  • et_elk Ties all the peices of an ELK cluster together
    • elk_forwarder Installs and Configures Logstash forwarder
    • elk_logstash Installs and Configures a central Logstash server
    • Leverages the logstash cookbook
    • elk_elasticsearch Installs and Configures an optimized Elasticsearch for use with Logstash
    • Leverages the elasticsearch cookbook
    • elk_kibana Installs and Configures Kibana for visualizing logs
    • Leverages the kibana cookbook

How do I actually use this system

1. Forwarders

You need to apply the client recipe to all your nodes. This will install and configure the Logstash Forwarder using elk_forwarder

2. Configure Forwarders

You then need to determine what logs you want to forward and you need to come up with a unique "type" to identify them. I.E. logs from rails app log files will have their type field set to rails_app.

From your Attributes

set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log']
set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp'
set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'

Or from your recipe. This one loops through a list of apps

apps.each do |app|
  node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"]
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = app
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'
end

3. Write your pattern

We need to build a pattern(s) to add to evertrue_patterns.erb or another patterns template that you have setup. Grab a bunch of sample logs and use the Grok Constructor utility to construct a pattern that will match your logs. I usually find an existing pattern online and then tweak it to properly match the logs I am parsing.

Give your pattern a name like ET_PUPPIES_APP and add it to your patterns template.

4. Write your filter

This is pretty open ended, but here are the key requirements.

  • Checks the log's type field with an if condition
  • Groks the log with the pattern you just created
  • Parses a field containing some gross timestamp into the standard @timestamp field using the date filter. Note that the timestamp field is very different from @timestamp. timestamp is just a string extracted from the log which needs to be converted into a nice date/time.

Example:

filter {
  if [type] == "nginx_access" {
    grok {
      match => ["message", "%{ET_NGINXACCESS}"]
    }

    date {
      match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
      remove_field => [ "timestamp" ]
    }
  }
}

Once you have written your filter, save it to a new template and call the logstash_config resource with something like this in your cookbook's recipe

logstash_config 'nginx filter' do
  templates_cookbook 'yourcookbook'
  templates 'filter_nginx' => 'filter_nginx.erb'
  instance 'server'
end

And now you are shipping, recieving, groking, parsing, inspecting, mutating, indexing, and storing your logs!

Recipes

default

Installs everything on the node and configures it to be a central log server that recieves logs from the clients

  1. Install Elasticsearch, Logstash, and kibana
  2. Configure logstash with inputs and outputs using elk::server

client

Configures the node to be a client that ships logs to the central logstash server

  1. Install logstash-forwarder via elk_forwarder
  2. Configure logstash-forwarder to forward logs to central log servers
    • Central logstash servers are discovered via the node['et_elk']['logstash_discovery'] attribute
    • To Disable auto-discovery simply set node['et_elk']['logstash_discovery'] to ''

Usage

depends 'et_elk', '~> 1.0'
# To install and configure the forwarder

include_recipe 'et_elk::client'

# To install everything on the node

include_recipe 'et_elk::default'

Contributing

  1. Fork the repository on Github
  2. Create a named feature branch (like add_component_x)
  3. Write your change
  4. Write tests for your change (if applicable)
  5. Run the tests with kitchen test, ensuring they all pass
  6. Submit a Pull Request using Github

License and Authors

Author:: EverTrue, inc. (eddie.hurtig@evertrue.com)

Change Log for et_elk

All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.

Unreleased

Changed

[4.0.0] - 2016-01-07

Changed

  • Drop the logstash cookbook
  • Upgrade logstash to version 1:2.1.1-1

[3.0.3] - 2016-01-04

Changed

  • Set network.host in elasticsearch YAML to a sane value

[3.0.2] - 2016-01-04

Changed

  • Use node.name for x_processed_by field when node fqdn is not available

[3.0.1] - 2015-12-23

Changed

  • Add logstash curator cron job

[3.0.0] - 2015-12-22

Changed

  • Cookbook functionality
    • Move to Java 8
    • Remove "client" functionality completely
    • Move most "server_config" content (filters, patterns, etc) to evertrue/logserver-cookbook
    • Merge in evertrue/elk_elasticsearch-cookbook, evertrue/elk_logstash-cookbook, and evertrue/elk_kibana-cookbook
    • Move key/cert configuration to evertrue/logserver-cookbook
    • Drop kibana recipe in favor of directly including kibana_lwrp::install
    • Stop using default recipe just to install dependencies
    • Give elasticsearch recipe its own attributes file
    • Move kibana attributes to server
    • Clean up the recipe headers
    • Remove blank/false attributes from server attributes file
    • Use underscores instead of spaces for ssl-related keys
    • Move attributes affecting logstash filter files into their own namespace for easy parsing
    • Set lumberjack default codec to "plain" to match what we are using in prod
    • Automatically include x_input_processor tag in all inputs
    • Clean up a lot of useless attributes
    • Remove resources that do nothing from logstash recipe
    • Move inputs/outputs definition to this cookbook (from evertrue/logserver-cookbook)
    • Craft a helper method (generate_module_config) to simplify the rendering of the input/output templates
    • Upgrade elasticsearch cookbook to version 2 (and associated cookbook changes)
    • Upgrade kibana to v4.3.0
    • Use less specific version for kibana_lwrp cookbook
  • Test functionality
    • Modernize TravisCI config
    • Substantially clean up and improve the integration tests
    • Remove Rubygems stuff completely (deprecated in favor of ChefDK)
    • Remove unused default instace from Travis tests
    • ServerSpec tests: Use net/http+JSON instead of cURL to parse ES status response

[2.3.3] - 2015-10-15

Changed

  • Add a Java-type timestamp matcher to the java log filter

[2.3.2] - 2015-10-15

Changed

  • Set flag x_input_processor on all inputs

[2.3.1] - 2015-10-15

Changed

  • Add service restart notifications to logstash_config resource calls

[2.3.0] - 2015-10-15

Added

  • Add log4j input for logstash

[2.2.0] - 2015-10-08

Added

  • A rails_basic filter handler for handling basic rails logs without fancy multiline parsing

[2.1.1] - 2015-10-05

Added

  • Attributes describing the plugins required for the logstash configs in et_elk

[2.1.0] - 2015-09-18

Changed

  • Version pins for elk_elasticsearch and elk_logstash

Added

  • Added sidekiq log proccessing support for events with the tag sidekiq

[2.0.0] - 2015-08-10

Changed

  • Changed the name elk -> et_elk
  • Match multiline log entries correctly
  • Clean up a bunch of recipe code
  • Spin off a new recipe from default called server (and its sub-recipe server_config)
  • Add a bunch of new ServerSpec tests
  • Avoid using storage cookbook w/ Vagrant
  • Update kitchen-ec2 config as per deprecations
  • Avoid the use of custom timestamp parsers
  • Drop support for 12.04
  • Change test environment name from _default to dev

1.3.5 - 2015-08-07

Added

  • Java App Patterns and Filters

1.3.4 - 2015-07-01

Changed

  • Specify that the multiline filter should send lines to the previous event

1.3.3 - 2015-07-01

Changed

  • Added extra patterns to attempt to match mesos timestamps
  • Parse and Remove the timestamp field for syslog events
  • Add Multiline support for singularity

1.3.2 - 2015-06-29

Changed

  • Removed .conf from template names

1.3.1 - 2015-06-29

Changed

  • The namespace for logstash input, filter, and output configuration attributes from elk_logstash to just elk

1.3.0 - 2015-06-29

Added

  • The inputs, outputs, and syslog filter moved from the elk_logstash cookbook

1.2.5 - 2015-06-26

Added

  • A common filter to perform a geoip lookup if a client_ip field is present
  • Instructions and guidlines for adding new patterns to evertrue_patterns

Changed

  • The raw timestamp field from mesos_timestamp to just timestamp as it is the primary time for that log
  • Removed year, month, day, hour, minute, second, and timezone fields as they aren't useful when the timestamp is parsed into a searchable date
  • Major cleanup of the schema. Lots of renaming of fields.

1.2.4 - 2015-06-26

Changed

  • Quoted ISO8601 to make configuration valid

1.2.3 - 2015-06-26

Changed

  • Used or instead of || in mesos slave filter

1.2.2 - 2015-06-25

Added

  • Filters to parse and remove the timestamp fied for haproxy and rails apps into @timestamp

1.2.1 - 2015-06-25

Changed

  • Add logstash_config resource for mesos filters

1.2.0 - 2015-06-24

Added

  • Test Kitchen role and attributes to converge a working All in One box
  • Initial Patterns and Filters for Mesos and Singlularity
  • Fauxhai to mock ohai data
  • date filter to NGINX logs

1.1.1 - 2015-06-23

Changed

  • Updated the ET_NGINXERROR pattern to properly match error logs

1.1.0 - 2015-06-22

Added

  • NGINX log parsing configuration

1.0.7 - 2015-06-18

Added

  • Fixes templates cookbook for evertrue patterns

1.0.6 - 2015-06-18

Added

  • Add HAPROXY config

1.0.5 - 2015-06-17

Changed

  • Loosen format of view and activerecord timing.

1.0.4 - 2015-06-17

Changed

  • The multiline filter's match pattern to accomodate logs with timestamp prefixed lines

1.0.3 - 2015-06-17

Added

  • SSL Certificate discovery attributes to find the CA cert

Changed

  • Fixed the rails app filter

1.0.2 - 2015-06-15

Added

  • A Fixup to wipe the elk_forwarder namespace if ['elk_forwarder']['config']['files'] is an array not the new format which is a hash

1.0.1 - 2015-06-15

Changed

  • Updated to elk_forwarder v2.0

1.0.0 - 2015-06-15

Added

  • Rails App log filter and pattern

Changed

Removed

0.0.1 - YYYY-MM-DD

Added

  • Initial Release

Foodcritic Metric
            

4.0.0 failed this metric

FC002: Avoid string interpolation where not required: /tmp/cook/b55fdf86463ba3169be9067c/et_elk/recipes/logstash.rb:9
FC003: Check whether you are running with chef server before using server-specific features: /tmp/cook/b55fdf86463ba3169be9067c/et_elk/recipes/elasticsearch.rb:11
FC007: Ensure recipe dependencies are reflected in cookbook metadata: /tmp/cook/b55fdf86463ba3169be9067c/et_elk/recipes/elasticsearch.rb:13