Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

RSS

elkstack (29) Versions 1.1.0

Installs/Configures elkstack

Berkshelf/Librarian
Policyfile
Knife
cookbook 'elkstack', '= 1.1.0'
cookbook 'elkstack', '= 1.1.0', :supermarket
knife cookbook site install elkstack
knife cookbook site download elkstack
README
Dependencies
Changelog
Quality

elkstack

Elasticsearch, Logstash, and Kibana stack. Due to the recommendations of the community, we are not using the embedded elasticsearch functionality of logstash at this point. This cookbook provides recipes for all three components, along with wrapper recipes such as single or cluster to facilitate different use cases.

This stack's design is intended for one or many standalone nodes, with a full stack of elasticsearch, logstash, and kibana. The only difference between one and many nodes is that elasticsearch is clustered together. Data dispatched to
Logstash on a particular node will use the local elasticsearch transport interface to index those logs to the node (and thus, the cluster). HTTP traffic dispatched to Kibana on port 80 on any node will also use the local elasticsearch HTTP interface to fetch and manipulate data.

Please read the individual recipe summaries to understand what each recipe does, as well as what each wrapper recipe is actually wrapping. As much as possible, upstream attributes have been exposed/overriden for our needs.

Pre-requistes before converging this stack

  • Please note that this cookbook does not restart elasticsearch automatically, in order to avoid causing an outage of the cluster. It does restart nginx and logstash, however. You will have to restart elasticsearch after the initial bootstrap.

  • You may want to consider adjusting node['elasticsearch']['discovery']['search_query'] if you are sharing one cluster among multiple environments. Just put a chef search in that attribute and this will use that search instead of one scoped to chef environments.

  • The agent recipe requires a pre-generated SSL key and certificate with something like openssl req -x509 -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -nodes -days 1000. This key and certificate data should be placed in data bag with name node['elkstack']['config']['lumberjack_data_bag'] under key and certificate keys, and base64 encoded into a single line string. You may also supply these with some other method and populate the appropriate node.run_state values (see _secrets.rb for more details). Note that this is not a PKI trust model, but an explicit trust model.

Changelog

See CHANGELOG.md for additional information about changes to this stack over time.

Supported Platforms

Ubuntu 12.04

Ubuntu 14.04

CentOS 6.5

Attributes

<table> <tr> <th>Key</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> <tr> <td><tt>['elkstack']['config']['logstash']['instance_name']</tt></td> <td>String</td> <td>Default logstash instance name</td> <td><tt>server</tt></td> </tr> <tr> <td><tt>['elkstack']['config']['cluster']</tt></td> <td>Boolean</td> <td>Whether to search for and connect Elasticsearch to cluster nodes</td> <td><tt>false</tt></td> </tr> <tr> <td><tt>['elkstack']['config']['data_disk']['disk_config_type']</tt></td> <td>Boolean or String</td> <td>See customizing the stack section below.</td> <td><tt>false</tt></td> </tr> <tr> <td><tt>['elkstack']['iptables']['enabled']</tt></td> <td>Boolean</td> <td>Enable/Disable iptables functionality</td> <td><tt>true</tt></td> </tr> <tr> <td><tt>['elkstack']['config']['site_name']</tt></td> <td>String</td> <td>Control the name of the self-signed SSL key and cert in /etc/nginx/ssl</td> <td><tt>kibana</tt></td> </tr> <tr> <td><tt>['elkstack']['config']['kibana']['redirect']</tt></td> <td>Boolean</td> <td>Enable/Disable nginx redirect for kibana from port 80 to port 443</td> <td><tt>true</tt></td> </tr> <tr> <td><tt>node.run_state['elkstack_kibana_username']</tt> and <tt>['elkstack']['config']['kibana']['username']</tt></td> <td>String</td> <td>Default username for basic auth for kibana, run_state used first</td> <td><tt>kibana</tt></td> </tr> <tr> <td><tt>node.run_state['elkstack_kibana_password']</tt></td> <td>String</td> <td>Password for basic auth for kibana</td> <td>random from <tt>Opscode::OpenSSL::Password</tt></td> </tr> <tr> <td><tt>['elkstack']['config']['lumberjack_data_bag']</tt></td> <td>String</td> <td>Data bag name for lumberjack key and certificate</td> <td><tt>lumberjack</tt></td> </tr> </table>

Customizing the stack

To override local storage for elasticsearch nodes (the stack will format and mount, as well as configure elasticsearch), set ['elkstack']['config']['data_disk']['disk_config_type'] to custom and provide each storage device and mount point in the following way: ```ruby disk_config = { 'file_system' => 'ext4', 'mount_options' => 'rw,user', 'mount_path' => '/usr/local/var/data/elasticsearch/disk1', 'format_command' => 'mkfs -t ext4 ', 'fs_check_command' => 'dumpe2fs' }

node.override['elasticsearch']['data']['devices']['/dev/xvde1'] = disk_config node.override['elasticsearch']['path']['data'] = disk_config['mount_path'] ```

To add additional logstash configuration to this stack, simply add additional templates in your wrapper cookbook. They should be placed in "#{@basedir}/#{@instance}/etc/conf.d" (see the config provider in the logstash cookbook).

To override the nginx configuration, simply supply a new template and specify your cookbook using ['kibana']['nginx']['template_cookbook'] and ['kibana']['nginx']['template']. You can also override just the password for the reverse proxy using node.run_state['elkstack_password'].

To override anything else, set the appropriate node hash (logstash, kibana, or elasticsearch).

Usage

elkstack::default

Default recipe, does not do anything.

elkstack::single

A simple wrapper recipe that sets up Elasticsearch, Logstash, and Kibana. Also configures an rsyslog sink into logstash on the local box. Everything except Logstash and Kibana is locked down to listen only on localhost.

elkstack::cluster

A simple wrapper recipe that sets up Elasticsearch, Logstash, and Kibana. Also configures an rsyslog sink into logstash on the local box. Sets the cluster flag so that the elasticsearch recipe builds it in a cluster-friendly way.

elkstack::agent

A simple wrapper recipe that sets up a logstash agent on the local box. Also configures an rsyslog sink into logstash on the local box.

elkstack::elasticsearch

Leans on the upstream elasticsearch/cookbook-elasticsearch cookbook for much of its work. We do override the default set of plugins to be installed, as well as the amount of JVM heap. See attributes/default.rb for those settings.

This recipe also tags the node so that other nodes that run this recipe can discover it, and configure Elasticsearch appropriately to join their cluster. It uses a tag, the current chef environment, and the cluster name as the default search criteria.

Most of this is configurable using the upstream Elasticsearch cookbook's attributes, including the chef search itself. There is not an easy toggle to turn off the search, however. Enables iptables rules if default['elkstack']['iptables']['enabled'] not nil

elkstack::logstash

Leans on the upstream lusis/chef-logstash cookbook for much of its work. We do override the default set of plugins to be installed, as well as the amount of JVM heap. See attributes/default.rb for those settings.

elkstack::kibana

Leans on the upstream lusis/chef-kibana cookbook for most of its work. Sets up an nginx site for kibana by default. By default, it also does not pass through most of the http paths directly to elasticsearch (whitelist).

elkstack::java

Wrapper for a java recipe. This is not included on the run list normally, so if you don't already, you must include this recipe or get another JVM installed before including anything else in this cookbook.

elkstack::newrelic

Validates if there is a newrelic license set and based on that, see if the node is tagged as 'elkstack' or 'elkstack_cluster' and creates a file with elasticsearch details. Installs python, pip and setuptools packages in order to support newrelic_meetme_plugin

elkstack::acl

Adds cluster node basic iptables rules and cluster iptables rules if appropriate attributes are set.

elkstack::agent_acl

Adds agent node basic iptables rules if appropriate attributes are set.

elkstack::disk_setup

Look for node['elkstack']['config']['data_disk']['disk_config_type'] to be truthy, and configure the upstream elasticsearch cookbook to format, mount, and use devices appropriately.

elkstack::*_monitoring

These correspond with the recipes above, and just provide a way to pull out the monitoring work to make the original recipes cleaner.

Miscellaneous

The wrapper recipes are single and cluster. These change attributes and then invoke elasticsearch, logstash, kibana, and rsyslog. Finally, there are utility recipes like java and newrelic (not invoked otherwise), as well as acl which is called by _base if node['elkstack']['iptables']['enabled'].

Contributing

See CONTRIBUTING.

Authors

Author:: Rackspace (devops-chef@rackspace.com)

License

# Copyright 2014, Rackspace Hosting
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

1.1.0

  • Add chef_environment fields to be shipped to central cluster
  • Add ACL recipe for agents to open up to :9300 on the ELK cluster nodes
  • Fix protocol from node to transport on logstash agents (to avoid needing more ACLs)

1.0.0

  • Bump logstash to version 1.4.2.
  • Added top-level agent recipe and test suite, intended for running logstash-agent on a regular server. Uses elasticsearch::search_discovery to find and list elk cluster nodes (can override this as well, see elasticsearch cookbook for how). Agent uses lumberjack protocol by default and requires a keypair before converging.
  • Switch nodes to communicate using node protocol on eslocal:9300 between logstash and elasticsearch on the same box.
  • Write chefspec tests for default (server nodes) and agent to 100% coverage. Add chef-sugar to Gemfile to be able to converge with chefspec and fauxhai.
  • Drop dependency on logstash_stack (didn't need to be there any longer, wasn't used).
  • Nodes no longer forward directly to logstash on the remote side, they forward to a local logstash listening for syslog.
  • Bugfix: /etc/hosts is now correctly populated for all classes of elkstack, not just multi-node.
  • Bugfix: The newrelic plugin now monitors against eslocal, not localhost.

0.3.0

  • Bumped default Elasticsearch version to 1.3.2. Java 7 is now required, and some tests had to be adjusted.
  • Fix bug in username not being used in tests because it was wrong in /etc/nginx/htpassword.curl.
  • Fix bug where port 443 was not open in iptables.

0.2.0

  • Added the ability to disable redirects on kibana

0.1.3

  • Sheppy Reno - Convert process monitors to platformstack

0.1.2

  • Add more options for kibana username and password fields under basic auth over SSL on nginx.

0.1.1

  • Seperate recipes per service, add searching and tests. Major workarounds for logstash cookbook.

0.1.0

  • Initial release of elkstack

Foodcritic Metric
            

1.1.0 passed this metric