cookbook 'filebeat', '= 0.5.0'
filebeat
(36) Versions
0.5.0
-
Follow19
Installs/Configures Elastic Filebeat
cookbook 'filebeat', '= 0.5.0', :supermarket
knife supermarket install filebeat
knife supermarket download filebeat
filebeat Cookbook
This is a Chef cookbook to manage Filebeat.
For Production environment, always prefer the most recent release.
Most Recent Release
cookbook 'filebeat', '~> 0.5.0'
From Git
cookbook 'filebeat', github: 'vkhatri/chef-filebeat', tag: 'v0.5.0'
Repository
https://github.com/vkhatri/chef-filebeat
Supported OS
This cookbook was tested on Windows, Amazon & Ubuntu Linux and expected to work on other RHEL platforms.
This also works on Solaris zones given a physical Solaris 11.2 server. For that, use the .kitchen.zone.yml file. Check usage at (https://github.com/criticalmass/kitchen-zone). You will need an url to a filebeat package that works on Solaris 11.2. Checkout Building-Filebeat-On-Solaris11.md for instructions to build a filebeat package.
Major Changes
v0.2.5
- Removed default output configuration attributes for
elasticsearch
,logstash
andfile
- Removed attributed
default['filebeat']['enable_localhost_output']
as defaultoutput
attributes are disabled
Cookbook Dependency
- windows
- apt
- yum
Recipes
filebeat::attributes
- cookbook derived default attributesfilebeat::config
- configure filebeatfilebeat::default
- default recipe (use it for run_list)filebeat::install_package
- install filebeat package for linux platformfilebeat::install_solaris
- install filebeat package for solaris platformfilebeat::install_windows
- install filebeat for windows platform
LWRP filebeat_prospector
LWRP filebeat_prospector
creates filebeat prospector configuration yaml file under directory node['filebeat']['prospectors_dir']
with file name prospector-#{resource_name}.yml
.
LWRP example
filebeat_prospector 'messages' do paths ['/var/log/messages'] document_type 'apache' ignore_older '24h' scan_frequency '15s' harvester_buffer_size 16384 fields 'type' => 'apacheLogs' end
LWRP Options
- action (optional) - default :create, options: :create, :delete
- paths (optional, String) - filebeat prospector configuration attribute
- type (optional, String) - filebeat prospector configuration attribute
- encoding (optional, String) - filebeat prospector configuration attribute
- fields (optional, Hash) - filebeat prospector configuration attribute
- fields_under_root (optional, TrueClass/FalseClass) - filebeat prospector configuration attribute
- ignore_older (optional, String) - filebeat prospector configuration attribute
- document_type (optional, String) - filebeat prospector configuration attribute
- input_type (optional, String) - filebeat prospector configuration attribute
- close_older (optional, String) - filebeat prospector configuration attribute
- scan_frequency (optional, String) - filebeat prospector configuration attribute
- harvester_buffer_size (optional, Integer) - filebeat prospector configuration attribute
- tail_files (optional, TrueClass/FalseClass) - filebeat prospector configuration attribute
- backoff (optional, String) - filebeat prospector configuration attribute
- max_backoff (optional, String) - filebeat prospector configuration attribute
- backoff_factor (optional, Integer) - filebeat prospector configuration attribute
- force_close_files (optional, TrueClass/FalseClass) - filebeat prospector configuration attribute
- include_lines (optional, Array) - A list of regular expressions to match the lines that you want Filebeat to include. Filebeat exports only the lines that match a regular expression in the list. By default, all lines are exported.
- exclude_lines (optional, Array) - A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped.
- max_bytes (optional, Integer) - filebeat prospector configuration attribute
-
multiline (optional, Hash) - Multiline configuration hash. Options:
pattern
: <regex pattern to match>,negate
: [true/false],match
: [before/after] - exclude_files (optional, Array) - A list of regular expressions to match the files that you want Filebeat prospector instance to exclude.
- spool_size (optional, Integer) - ilebeat prospector configuration attribute
- publish_async (optional, TrueClass/FalseClass) - filebeat prospector configuration attribute
- idle_timeout (optional, String) - filebeat prospector configuration attribute
- registry_file (optional, String) - filebeat prospector configuration attribute
-
json_message_key (optional, String) - filebeat prospector configuration attribute
json.message_key
-
json_keys_under_root (optional, String) - filebeat prospector configuration attribute
json.keys_under_root
-
json_overwrite_keys (optional, String) - filebeat prospector configuration attribute
json.overwrite_keys
-
json_add_error_key (optional, String) - filebeat prospector configuration attribute
json.add_error_key
- tags (optional, Array) - filebeat prospector configuration attribute
How to Add Filebeat Output via Node Attribute
Redis Output
"default_attributes": { "filebeat": { "config": { "output": { "redis": { "enable": true, "host": "127.0.0.1", "port": 6379, "save_topology": false, "index": "filebeat", "db": 0, "db_topology": 1, "password": "", "timeout": 5, "reconnect_interval": 1 } } } } }
ElasticSearch Output
"default_attributes": { "filebeat": { "config": { "output": { "elasticsearch": { "enable": true, "hosts": ["127.0.0.1:9200"], "save_topology": false, "max_retries": 3, "bulk_max_size": 1000, "flush_interval": null, "protocol": "http", "username": null, "password": null, "index": "filebeat", "path": "/elasticsearch", "tls": { "certificate_authorities": ["/etc/ca.crt"], "certificate": "/etc/client.crt", "certificate_key": "/etc/client.key", "insecure": false } } } } } }
Logstash Output
"default_attributes": { "filebeat": { "config": { "output": { "logstash": { "enable": true, "hosts": ["127.0.0.1:5000"], "loadbalance": true, "save_topology": false, "index": "filebeat", "tls": { "certificate_authorities": ["/etc/ca.crt"], "certificate": "/etc/client.crt", "certificate_key": "/etc/client.key", "insecure": false } } } } } }
File Output
"default_attributes": { "filebeat": { "config": { "output": { "file": { "path": "/tmp/filebeat", "filename": "filebeat", "rotate_every_kb": 1000, "number_of_files": 7 } } } } }
How to Add Filebeat Prospectors via Node Attribute
Individual prospectors configuration file can be added using attribute default['filebeat']['prospectors']
. Each prospector configuration will
be created as a different yaml file under default['filebeat']['prospector_dir']
with prefix prospector-
"default_attributes": { "filebeat": { "prospectors": { "system_logs": { "filebeat": { "prospectors": [ { "paths": [ "/var/log/messages", "/var/log/syslog" ], "type": "log", "fields": { "type": "system_logs" } } ] } }, "secure_logs": { "filebeat": { "prospectors": [ { "paths": [ "/var/log/secure", "/var/log/auth.log" ], "type": "log", "fields": { "type": "secure_logs" } } ] } }, "apache_logs": { "filebeat": { "prospectors": [ { "paths": [ "/var/log/apache/*.log" ], "type": "log", "ignore_older": "24h", "scan_frequency": "15s", "harvester_buffer_size": 16384, "fields": { "type": "apache_logs" } } ] } } } } }
Above configuration will create three different prospector files - prospector-system_logs.yml, prospector-secure_logs.yml and prospector-apache_logs.yml
Core Attributes
default['filebeat']['version']
(default:5.4.2
): filebeat versiondefault['filebeat']['ignore_version']
(default:false
): ignore filebeat version forpackage
installdefault['filebeat']['release']
(default:1
): filebeat release for yum packagedefault['filebeat']['service']['init_style']
(default:init
): filebeat service init system, options: init, runitdefault['filebeat']['package_url']
(default:auto
): package url for windows installationdefault['filebeat']['conf_dir']
(default:/etc/filebeat
): filebeat yaml configuration file directorydefault['filebeat']['conf_file']
(default:/etc/filebeat/filebeat.yml
): filebeat configuration filedefault['filebeat']['notify_restart']
(default:true
): whether to restart filebeat service on configuration file changedefault['filebeat']['disable_service']
(default:false
): whether to stop and disable filebeat servicedefault['filebeat']['prospectors_dir']
(default:/etc/filebeat/conf.d
): prospectors configuration file directorydefault['filebeat']['prospectors']
(default:{}
): prospectors configuration file
Configuration File filebeat.yml Attributes
default['filebeat']['config']['filebeat']['prospectors']
(default:[]
): filebeat interface device namedefault['filebeat']['config']['filebeat']['registry_file']
(default:/var/lib/filebeat/registry
): filebeat services to capture packetsdefault['filebeat']['config']['filebeat']['config_dir']
(default:node['filebeat']['prospectors_dir']
): filebeat prospectors configuration files folderdefault['filebeat']['config']['output']
(default:{}
): configure elasticsearch. logstash, file etc. output
For more attribute info, visit below links:
https://github.com/elastic/filebeat/blob/master/etc/filebeat.yml
Filebeat YUM/APT Repository Attributes
default['filebeat']['yum']['description']
(default: ``): beats yum reporitory attributedefault['filebeat']['yum']['gpgcheck']
(default:true
): beats yum reporitory attributedefault['filebeat']['yum']['enabled']
(default:true
): beats yum reporitory attributedefault['filebeat']['yum']['baseurl']
(default:https://packages.elastic.co/beats/yum/el/$basearch
): beatsyum reporitory attributedefault['filebeat']['yum']['gpgkey']
(default:https://packages.elasticsearch.org/GPG-KEY-elasticsearch
): beats yum reporitory attributedefault['filebeat']['yum']['metadata_expire']
(default:3h
): beats yum reporitory attributedefault['filebeat']['yum']['action']
(default::create
): beats yum reporitory attributedefault['filebeat']['apt']['description']
(default:calculated
): beats apt reporitory attributedefault['filebeat']['apt']['components']
(default:['stable', 'main']
): beats apt reporitory attributedefault['filebeat']['apt']['uri']
(default:https://packages.elastic.co/beats/apt
): beats apt reporitory attributedefault['filebeat']['apt']['key']
(default:http://packages.elasticsearch.org/GPG-KEY-elasticsearch
): beats apt reporitory attributedefault['filebeat']['apt']['action']
(default::add
): filebeat apt reporitory attribute
Other Attributes
default['filebeat']['service']['name']
(default:filebeat
): filebeat service namedefault['filebeat']['service']['retries']
(default::0
): filebeat service resource attributedefault['filebeat']['service']['retry_delay']
(default::2
): filebeat service resource attribute
Contributing
- Fork the repository on Github
- Create a named feature branch (like
add_component_x
) - Write your change
- Write tests for your change (if applicable)
- Run the tests (
rake & rake knife
), ensuring they all pass - Write new resource/attribute description to
README.md
- Write description about changes to PR
- Submit a Pull Request using Github
Copyright & License
Authors:: Virender Khatri and Contributors
<pre>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
</pre>
Dependent cookbooks
windows >= 0.0.0 |
apt >= 0.0.0 |
yum >= 0.0.0 |
yum-plugin-versionlock >= 0.1.2 |
runit >= 0.0.0 |
Contingent cookbooks
There are no cookbooks that are contingent upon this one.
filebeat CHANGELOG
This file is used to list changes made in each version of the filebeat cookbook.
0.5.0
Virender Khatri - Updated Beats Version to v5.4.2
Virender Khatri - Updated config file permissions to 0600
0.4.9
- Virender Khatri - Updated Beats Version to v5.2.2
0.4.8
Kyle Gochenour - PR #89, correct spool_size to use the correct parameter
Virender Khatri - Issue #91, allow to ignore package verson in favor of pre installed filebeat packages
Virender Khatri - PR #94, Travis CI Fix
Len Smith - PR #93, Added check to avoid restart if service is disabled or notify restart is set to false
0.4.7
Tom Michaud - PR #86, Supports prospector 'tags' attribute
Virender Khatri - Fixed Issue #88, Filebeat 5.2.0 Released, Unable to Install on Ubuntu
Virender Khatri - Updated default version to v5.2.0
0.4.6
- William Soula - PR #83, upgrade to filebeat 5.1.2, fix for #84
0.4.5
Virender Khatri - issue #78, fixed windows filebeat directory issue for v5.x
Virender Khatri - use chefdk
Virender Khatri - added kitchen dokken
0.4.4
- Virender Khatri - fixed runit service
0.4.3
Michael Mosher - added json attributes to filebeat_prospector
Virender Khatri - added v5.x support
Virender Khatri - updated default filebeat version to v5.1.1
0.4.2
- Andrei Scopenco - include yum-plugin-versionlock recipe for platform_family instead
0.4.1
- Jesse Cotton - Add support for using runit instead of the default init system
0.4.0
- Virender Khatri - fix for #60, HWRP does not restart filebeat service
0.3.9
Virender Khatri - fix for #65, added yum apt version lock
Virender Khatri - updated beats version to v1.3.1
0.3.8
- Virender Khatri - #68, config_dir attribute not getting set
0.3.7
Virender Khatri - Move derived attributes to recipe attributes
Virender Khatri - Update filebeat version to v1.3.0
0.3.6
Arif Khan - Added Solaris Support
Tom Noonan - Handle when new_resource.action is an array
Virender Khatri - Updated filebeat config deprecated url reference
Virender Khatri - Added service resource configurable attributes with default values
Scott Nelson Windels - Add enable attribute back (used to be enabled, not it has been brought back as enable)
Eric Herot - Only call powershell resource on windows machines
Virender Khatri - Fix Travis
Virender Khatri - Use powershell_script instead
Scott Nelson Windels - Remove enabled attribute for now, as it isn't a feature in filebeat beyond the rc versions
0.3.4
- Samuel Sampaio - added new filebeat prospector attributes
0.3.3
- Virender Khatri - updated beats version to v1.2.3
0.3.2
Prerak Shah - Fixed Travis Errors
Prerak Shah - Fixed Unit Tests
Al Lefebvre - Added missing attribute for exclude_files
Luke Lowery - Fixing issue with patterns and older versions of Ruby
Updated YAML engine to adhere to ruby style guidePrerak Shah - Fixed default Install paths for windows
Martin Smith - Respect service flags on package installation
Azat Khadiev - Support spaces in file path for Windows
0.3.1
Virender Khatri - fix for #41
Virender Khatri - added apt-get options, fix for #39
Virender Khatri - bump filebeat version to v1.2.1
0.2.8
Spencer Owen - Adds a tag to berksfile
Chris Barber - cleaning up mixmatch of tabs and spaces. fixed spelling error on prospector
Eric Herot - Fix the formatting on the LWRP example in the README
Roberto Rivera - Add include_lines and exclude lines.
Sean Nolen - #33 added multiline support for LWRP
Sean Nolen - #30 fixed rubocop error
Seva Orlov - #34, restart filebeat on upgrade
Virender Khatri - update to beat v1.1.2
0.2.7
Virender Khatri - #21, add yum_repository resource attribute metadata_expire
Virender Khatri - #20, update to beat v1.0.1
0.2.6
Virender Khatri - #18, added LWRP resource for prospectors
Virender Khatri - #15, fix kitchen test
0.2.5
Virender Khatri - disabled default output configuration and enable_localhost_output attributes
Virender Khatri - #10, handle missing attribute node['filebeat']['windows']['version_string']
Virender Khatri - #6, added specs
Virender Khatri - #13, major changes to support repository package install
0.2.1
Virender Khatri - Added platforms metadata info
Virender Khatri - #8, add missing dependency on powershell for windows platform
Virender Khatri - #9, use resource powershell instead of powershell_script
0.2.0
Brandon Wilson - Include dpkg options to keep old config files when upgrading filebeat to a new release. Without specifying the dpkg options, dpkg will attempt to interactively ask if it should keep the old conf file, or replace it with the vendor supplied conf file which comes with the new version of the package. Since chef is running dpkg non-interactively, it causes dpkg to exit with code 1, and the chef run fails.
Virender Khatri - Fix for #4, handle derived attribute for package_url
Patrick Christopher - Added support for Windows OS
0.1.0
- Virender Khatri - Initial release of filebeat
Check the Markdown Syntax Guide for help with Markdown.
The Github Flavored Markdown page describes the differences between markdown on github and standard markdown.
Foodcritic Metric
0.5.0 failed this metric
FC066: Ensure chef_version is set in metadata: filebeat/metadata.rb:1
FC069: Ensure standardized license defined in metadata: filebeat/metadata.rb:1
Run with Foodcritic Version 13.0.0 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
License Metric
0.5.0 passed this metric
No Binaries Metric
0.5.0 passed this metric
Version Tag Metric
0.5.0 passed this metric
0.5.0 failed this metric
FC069: Ensure standardized license defined in metadata: filebeat/metadata.rb:1
Run with Foodcritic Version 13.0.0 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
0.5.0 passed this metric
No Binaries Metric
0.5.0 passed this metric
Version Tag Metric
0.5.0 passed this metric
0.5.0 passed this metric
0.5.0 passed this metric