cookbook 'storm-platform', '= 2.0.0'
storm-platform (9) Versions 2.0.0 Follow1
Install/Configure a Storm cluster
cookbook 'storm-platform', '= 2.0.0', :supermarket
knife supermarket install storm-platform
knife supermarket download storm-platform
Storm Cluster
Description
Apache Storm is a free and open source distributed
realtime computation system. Storm makes it easy to reliably process unbounded
streams of data, doing for realtime processing what Hadoop did for batch
processing.
This cookbooks installs and configures Storm (> 1.0.0).
Requirements
Cookbooks and gems
Declared in [metadata.rb](metadata.rb) and in [Gemfile](Gemfile).
Platforms
A systemd managed distribution:
- RHEL Family 7, tested on Centos
Note: it should work quite fine on Debian 8 (with some attributes tuning) but
the official docker image does not allow Systemd to work easily, so it could
not be tested.
Usage
First Setup
The recommended way to use this cookbook is through the creation of a role
per Storm cluster. This enables the search by role feature, allowing a
simple service discovery.
In fact, there are two ways to configure the search:
1. with a static configuration through a list of hostnames (attributes hosts
that is ['storm-platform']['hosts']
) (empty by default so deactivated)
2. with a real search, performed on a role (attributes role
and size
like
in ['storm-platform']['role']
). The role should be in the run-list of all
nodes of the cluster. The size is a safety and should be the number of nodes
in the cluster.
If hosts is configured, role
and size
are ignored and search is
deactivated.
See [roles](test/integration/roles) for some examples and
Cluster Search documentation for more information.
By default, the first node in hosts array or in search result (sorted on fqdn)
will be setup with a Nimbus, a LogViewer and a UI. All other nodes will host a
Supervisor and a LogViewer. To get this cookbook to setup the nth node to be
the Nimbus instead of the first one, change attribute
node['storm-platform']['nimbus_id']
. You can also activate Nimbus High
Availability. You will find more information in the following sections.
Zookeeper Cluster
To install properly a Storm cluster, you need a Zookeeper cluster.
This is not in the scope of this cookbook but if you need one, you should
consider using Zookeeper Platform.
The configuration of Zookeeper hosts use search and is done similarly as for
Storm hosts, ie with a static list of hostnames or by using a search on
a role. The attribute to configure are located in
node['storm-platform']['zookeeper']
.
Java
By default, this cookbook installs openjdk from the official repositories
(openjdk-headless 8 on centos 7) just before starting the service. You can
change this behavior by setting node['storm-platform']['java']
to ""
(empty
string), or choose your package by setting the package name in
node['storm-platform']['java'][node[:platform]]
.
Nimbus High Availability
A new feature of Storm 1.0.0 is Nimbus High Availability. This allows a backup
nimbus to be elected as master nimbus when the later crashes. To do so, you
have to declare all possible nimbus. This can be done by setting
node['storm-platform']['nimbus_id']
(default: 1) to the first nimbus and
node['storm-platform']['n_of_nimbus']
(default: 1) to the number of nimbus.
Test
This cookbook is fully tested through the installation of the full platform
in docker hosts. This uses kitchen, docker and some monkey-patching.
If you run kitchen list
, you will see 4 suites:
- zookeeper-centos-7
- storm-01-centos-7
- storm-02-centos-7
- storm-03-centos-7
Each corresponds to a different node in the cluster. They are connected through
a bridge network named kitchen, which is created if necessary.
For more information, see [.kitchen.yml](.kitchen.yml) and [test](test)
directory.
Local cluster
The cluster installed with kitchen converge
is fully working and can thus be
used as a local cluster for developments tests.
You can access it by using internal DNS of the docker network named kitchen
or by declaring each node in your hosts file. You can get each IP by
running:
docker inspect --format \
'{{.NetworkSettings.Networks.kitchen.IPAddress}}' container_name
Attributes
Configuration is done by overriding default attributes. All configuration keys
have a default defined in [attributes/default.rb](attributes/default.rb).
Please read it to have a comprehensive view of what and how you can configure
this cookbook behavior.
Recipes
default
Install and fully configure a given Storm by including the recipes:
install, create_user, config and systemd_service, in that order.
install
Install Storm from tar archive with ark cookbook.
create_user
Create a system user for Storm.
config
Configure Storm.
systemd_service
Create Systemd service files and set them up. Different Storm services will
be set up following the kind of node (both may apply).
For a nimbus:
- nimbus
- logviewer
- ui
For a supervisor:
- supervisor
- logviewer
If multiple nimbus exist (to activate nimbus High Availability) then we
consider each nimbus to be both nimbus and supervisor. In this case, a
nimbus will run all the services.
Note: install java package (OpenJDK 8 on centos 7) by default, can be
disabled by setting node['confluent-platform']['java']
to "" (empty string).
A platform specific configuration for the package to install is also possible.
Resources/Providers
None.
Changelog
Available in [CHANGELOG](CHANGELOG).
Contributing
Please read carefully [CONTRIBUTING.md](CONTRIBUTING.md) before making a merge
request.
License and Author
- Author:: Samuel Bernard (samuel.bernard@s4m.io)
- Contributor:: Guillaume Alaux (guillaume.alaux@s4m.io)
Copyright (c) 2015-2016 Sam4Mobile Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Dependent cookbooks
ark >= 0.0.0 |
cluster-search >= 0.0.0 |
Contingent cookbooks
There are no cookbooks that are contingent upon this one.
Changelog
2.0.0
Main:
- Require Storm > 1.0.0 (because of HA, nimbus.host is no longer valid)
- Configure nimbus HA by allowing multiple nimbus (with n_of_nimbus key)
- Fix default config for Storm > 1.0.0
- Fetch log4j2 default config directly from github
- Do not wait anymore after (re)starting a service
Test:
- Strengthen tests: test all nodes and check deactivated services
- Smart sleeps to wait services to be up
Misc:
- Improve documentation (better explanation of search, add HA, etc.)
1.5.0
Main:
- Use Storm 1.0.1 by default
- Fix invalid check on cluster size
- Use Continuous Integration with gitlab-ci
Misc:
- Fix Gemfile, need k-docker_cli instead of k-docker
- Fix the too long line in kitchen_sandbox
- Rename CHANGELOG -> CHANGELOG.md and fix markdown
1.4.0
- Fix: config could contain ruby/chef -specific yaml
- Make Systemd unit path configurable, default is now "/etc/systemd/system"
1.3.0
Main:
- Rename recipes containing '-' in their name (small breaking change):
- create-user -> create_user
- systemd-service -> systemd_service
Test:
- Switch to docker_cli, use prepared docker image
- Switch kitchen driver from docker to docker_cli
- Use sbernard/centos-systemd-kitchen image instead of bare centos
- Remove privileged mode :)
- Remove some useless monkey patching
- Use loops for suites generation
- Remove dnsdock, use docker DNS (docker >= 1.10)
- Use "kitchen" network, create it if needed
Misc:
- Fix rubocop offenses
- Use specific names for resources to avoid cloning
- Improve documentation, search config, etc.
1.2.0
Main:
- Use storm 0.10.0 by default
- Generate config in 2 steps: can use chef variables in erb style
Fixes:
- Fix dirty yaml of nested structure for storm.yaml
- Fix topology launch: create a link from storm.home/logs to log_dir
- Fix auto-restart and also monitor log4j2 files
- Fix: apply STORM-945 in default log4j configuration
Test:
- Rationalize docker provision to limit images
Packaging:
- Reorganize README:
- Move changelog from README to CHANGELOG
- Move contribution guide to CONTRIBUTING.md
- Reorder README, fix Gemfile missing
- Add Apache 2 license file
- Add missing chefignore
1.1.0
- Strengthen tests reliability by sleeping after a Storm service is launch
- Dependencies use supermarket, switch to zookeeper-platform
1.0.1
- Cleaning, use java-1.8.0-openjdk-headless as default java package
1.0.0
- Initial version with Centos 7 support
Collaborator Number Metric
2.0.0 failed this metric
Failure: Cookbook has 1 collaborators. A cookbook must have at least 2 collaborators to pass this metric.
Contributing File Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file
Foodcritic Metric
2.0.0 failed this metric
FC066: Ensure chef_version is set in metadata: storm-platform/metadata.rb:1
FC069: Ensure standardized license defined in metadata: storm-platform/metadata.rb:1
Run with Foodcritic Version 13.1.1 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
No Binaries Metric
2.0.0 passed this metric
Testing File Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.0.0 failed this metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file
Foodcritic Metric
2.0.0 failed this metric
FC066: Ensure chef_version is set in metadata: storm-platform/metadata.rb:1
FC069: Ensure standardized license defined in metadata: storm-platform/metadata.rb:1
Run with Foodcritic Version 13.1.1 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
No Binaries Metric
2.0.0 passed this metric
Testing File Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.0.0 failed this metric
FC069: Ensure standardized license defined in metadata: storm-platform/metadata.rb:1
Run with Foodcritic Version 13.1.1 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
2.0.0 passed this metric
Testing File Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.0.0 failed this metric
2.0.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must include a tag that matches this cookbook version number