Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

RSS

et_elk (4) Versions 2.3.3

Configures an Elk Cluster

Berkshelf/Librarian
Policyfile
Knife
cookbook 'et_elk', '= 2.3.3'
cookbook 'et_elk', '= 2.3.3', :supermarket
knife cookbook site install et_elk
knife cookbook site download et_elk
README
Dependencies
Changelog
Quality

ELK Build Status

ELK

This cookbook is the top level wrapper for the EverTrue ELK cookbook ecosystem

Requirements

  • et_elk Ties all the peices of an ELK cluster together
    • elk_forwarder Installs and Configures Logstash forwarder
    • elk_logstash Installs and Configures a central Logstash server
    • Leverages the logstash cookbook
    • elk_elasticsearch Installs and Configures an optimized Elasticsearch for use with Logstash
    • Leverages the elasticsearch cookbook
    • elk_kibana Installs and Configures Kibana for visualizing logs
    • Leverages the kibana cookbook

How do I actually use this system

1. Forwarders

You need to apply the client recipe to all your nodes. This will install and configure the Logstash Forwarder using elk_forwarder

2. Configure Forwarders

You then need to determine what logs you want to forward and you need to come up with a unique "type" to identify them. I.E. logs from rails app log files will have their type field set to rails_app.

From your Attributes

set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log']
set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp'
set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'

Or from your recipe. This one loops through a list of apps

apps.each do |app|
  node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"]
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = app
  node.set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'
end

3. Write your pattern

We need to build a pattern(s) to add to evertrue_patterns.erb or another patterns template that you have setup. Grab a bunch of sample logs and use the Grok Constructor utility to construct a pattern that will match your logs. I usually find an existing pattern online and then tweak it to properly match the logs I am parsing.

Give your pattern a name like ET_PUPPIES_APP and add it to your patterns template.

4. Write your filter

This is pretty open ended, but here are the key requirements.

  • Checks the log's type field with an if condition
  • Groks the log with the pattern you just created
  • Parses a field containing some gross timestamp into the standard @timestamp field using the date filter. Note that the timestamp field is very different from @timestamp. timestamp is just a string extracted from the log which needs to be converted into a nice date/time.

Example:

filter {
  if [type] == "nginx_access" {
    grok {
      match => ["message", "%{ET_NGINXACCESS}"]
    }

    date {
      match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
      remove_field => [ "timestamp" ]
    }
  }
}

Once you have written your filter, save it to a new template and call the logstash_config resource with something like this in your cookbook's recipe

logstash_config 'nginx filter' do
  templates_cookbook 'yourcookbook'
  templates 'filter_nginx' => 'filter_nginx.erb'
  instance 'server'
end

And now you are shipping, recieving, groking, parsing, inspecting, mutating, indexing, and storing your logs!

Recipes

default

Installs everything on the node and configures it to be a central log server that recieves logs from the clients

  1. Install Elasticsearch, Logstash, and kibana
  2. Configure logstash with inputs and outputs using elk::server

client

Configures the node to be a client that ships logs to the central logstash server

  1. Install logstash-forwarder via elk_forwarder
  2. Configure logstash-forwarder to forward logs to central log servers
    • Central logstash servers are discovered via the node['et_elk']['logstash_discovery'] attribute
    • To Disable auto-discovery simply set node['et_elk']['logstash_discovery'] to ''

Usage

depends 'et_elk', '~> 1.0'
# To install and configure the forwarder

include_recipe 'et_elk::client'

# To install everything on the node

include_recipe 'et_elk::default'

Contributing

  1. Fork the repository on Github
  2. Create a named feature branch (like add_component_x)
  3. Write your change
  4. Write tests for your change (if applicable)
  5. Run the tests with kitchen test, ensuring they all pass
  6. Submit a Pull Request using Github

License and Authors

Author:: EverTrue, inc. (eddie.hurtig@evertrue.com)

Dependent cookbooks

elk_forwarder ~> 2.0
elk_kibana ~> 1.1
elk_logstash ~> 1.0
elk_elasticsearch ~> 1.0

Contingent cookbooks

logserver Applicable Versions

Change Log for et_elk

All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.

Unreleased

Changed

[2.3.3] - 2015-10-15

Changed

  • Add a Java-type timestamp matcher to the java log filter

[2.3.2] - 2015-10-15

Changed

  • Set flag x_input_processor on all inputs

[2.3.1] - 2015-10-15

Changed

  • Add service restart notifications to logstash_config resource calls

[2.3.0] - 2015-10-15

Added

  • Add log4j input for logstash

[2.2.0] - 2015-10-08

Added

  • A rails_basic filter handler for handling basic rails logs without fancy multiline parsing

[2.1.1] - 2015-10-05

Added

  • Attributes describing the plugins required for the logstash configs in et_elk

[2.1.0] - 2015-09-18

Changed

  • Version pins for elk_elasticsearch and elk_logstash

Added

  • Added sidekiq log proccessing support for events with the tag sidekiq

[2.0.0] - 2015-08-10

Changed

  • Changed the name elk -> et_elk
  • Match multiline log entries correctly
  • Clean up a bunch of recipe code
  • Spin off a new recipe from default called server (and its sub-recipe server_config)
  • Add a bunch of new ServerSpec tests
  • Avoid using storage cookbook w/ Vagrant
  • Update kitchen-ec2 config as per deprecations
  • Avoid the use of custom timestamp parsers
  • Drop support for 12.04
  • Change test environment name from _default to dev

1.3.5 - 2015-08-07

Added

  • Java App Patterns and Filters

1.3.4 - 2015-07-01

Changed

  • Specify that the multiline filter should send lines to the previous event

1.3.3 - 2015-07-01

Changed

  • Added extra patterns to attempt to match mesos timestamps
  • Parse and Remove the timestamp field for syslog events
  • Add Multiline support for singularity

1.3.2 - 2015-06-29

Changed

  • Removed .conf from template names

1.3.1 - 2015-06-29

Changed

  • The namespace for logstash input, filter, and output configuration attributes from elk_logstash to just elk

1.3.0 - 2015-06-29

Added

  • The inputs, outputs, and syslog filter moved from the elk_logstash cookbook

1.2.5 - 2015-06-26

Added

  • A common filter to perform a geoip lookup if a client_ip field is present
  • Instructions and guidlines for adding new patterns to evertrue_patterns

Changed

  • The raw timestamp field from mesos_timestamp to just timestamp as it is the primary time for that log
  • Removed year, month, day, hour, minute, second, and timezone fields as they aren't useful when the timestamp is parsed into a searchable date
  • Major cleanup of the schema. Lots of renaming of fields.

1.2.4 - 2015-06-26

Changed

  • Quoted ISO8601 to make configuration valid

1.2.3 - 2015-06-26

Changed

  • Used or instead of || in mesos slave filter

1.2.2 - 2015-06-25

Added

  • Filters to parse and remove the timestamp fied for haproxy and rails apps into @timestamp

1.2.1 - 2015-06-25

Changed

  • Add logstash_config resource for mesos filters

1.2.0 - 2015-06-24

Added

  • Test Kitchen role and attributes to converge a working All in One box
  • Initial Patterns and Filters for Mesos and Singlularity
  • Fauxhai to mock ohai data
  • date filter to NGINX logs

1.1.1 - 2015-06-23

Changed

  • Updated the ET_NGINXERROR pattern to properly match error logs

1.1.0 - 2015-06-22

Added

  • NGINX log parsing configuration

1.0.7 - 2015-06-18

Added

  • Fixes templates cookbook for evertrue patterns

1.0.6 - 2015-06-18

Added

  • Add HAPROXY config

1.0.5 - 2015-06-17

Changed

  • Loosen format of view and activerecord timing.

1.0.4 - 2015-06-17

Changed

  • The multiline filter's match pattern to accomodate logs with timestamp prefixed lines

1.0.3 - 2015-06-17

Added

  • SSL Certificate discovery attributes to find the CA cert

Changed

  • Fixed the rails app filter

1.0.2 - 2015-06-15

Added

  • A Fixup to wipe the elk_forwarder namespace if ['elk_forwarder']['config']['files'] is an array not the new format which is a hash

1.0.1 - 2015-06-15

Changed

  • Updated to elk_forwarder v2.0

1.0.0 - 2015-06-15

Added

  • Rails App log filter and pattern

Changed

Removed

0.0.1 - YYYY-MM-DD

Added

  • Initial Release

Foodcritic Metric
            

2.3.3 failed this metric

FC003: Check whether you are running with chef server before using server-specific features: /tmp/cook/9eb9fe39e4efe65d68f2f995/et_elk/recipes/client.rb:19