cookbook 'confluent-platform', '= 1.2.0', :supermarket
confluent-platform (16) Versions 1.2.0 Follow1
Install and configure Confluent platform (Kafka)
cookbook 'confluent-platform', '= 1.2.0'
knife supermarket install confluent-platform
knife supermarket download confluent-platform
Confluent Platform
Description
Apache Kafka, an open source technology created and maintained by the founders of Confluent, acts as a real-time, fault tolerant, highly scalable messaging system. It is widely adopted for use cases ranging from collecting user activity data, logs, application metrics, stock ticker data, and device instrumentation. Its key strength is its ability to make high volume data available as a real-time stream for consumption in systems with very different requirements—from batch systems like Hadoop, to real-time systems that require low-latency access, to stream processing engines that transform the data streams as they arrive.
This infrastructure lets you build around a single central nervous system transmitting messages to all the different systems and applications within your company. Learn more on http://confluent.io.
This cookbook focuses on deploying Confluent Platform elements on your clusters via Chef on systemd managed distributions. At the moment, this includes Kafka, Schema Registry and Kafka Rest.
Requirements
Cookbooks and gems
Declared in [metadata.rb](metadata.rb) and in [Gemfile](Gemfile).
Platforms
A systemd managed distribution: - RHEL Family 7, tested on Centos
Note: it should work fine on Debian 8 but the official docker image does not allow systemd to work easily, so it could not be tested.
Usage
Easy Setup
Default recipe does nothing. Each service Kafka, Schema Registry or Kafka Rest will be installed by calling respectively recipe [install-kafka](recipes/install-kafka.rb), [install-registry](recipes/install-registry.rb) and [install-rest](recipes/install-rest.rb).
By default, this cookbook installs openjdk from the official repositories
(openjdk 8 on centos 7) in services recipe, just before
launching the service. You can deactivate this behavior by setting
node['confluent-platform']['java']
to ""
, or choose your package by setting
the package name in node['confluent-platform']['java'][node[:platform]]
.
Search
The recommended way to use this cookbook is through the creation of a different role per cluster, that is a role for Kafka, Schema Registry and Kafka Rest. This enables the search by role feature, allowing a simple service discovery.
See [roles](test/integration/roles) for some examples and Cluster Search documentation for more information.
Test
This cookbook is fully tested through the installation of the full platform in docker hosts. This uses kitchen, docker and some monkey-patching.
If you run kitchen list
, you will see 7 suites:
- dnsdock-centos-7
- zookeeper-centos-7
- kafka-01-centos-7
- kafka-02-centos-7
- kafka-03-centos-7
- registry-01-centos-7
- rest-01-centos-7
Each corresponds to a different node in the cluster.
For more information, see [.kitchen.yml](.kitchen.yml) and [test](test) directory.
Local cluster
Of course, the cluster you install by running kitchen converge
is fully
working so you can use it as a local cluster to test your development (like a
new Kafka client). Moreover, compared to a single node cluster usually
installed on workstations, you can detected partition/timing/fault-tolerance
issues you could not because of the simplicity of a single-node system.
You can access it by adding the dnsdock used in the cluster as your main DNS
resolver: add
docker inspect --format '{{.NetworkSettings.IPAddress}}' dnsdock-kafka
in /etc/resolv.conf
.
Then to produce some messages:
kafka-console-producer.sh \
--broker-list kafka-kitchen-01.kitchen:9092 \
--topic my_topic
And to read them:
kafka-console-consumer.sh \
--zookeeper zookeeper-kafka.kitchen/kafka-kitchen \
--topic my_topic \
--from-beginning
Or you can use Rest API with http://rest-kitchen-01.kitchen:8082 and full Schema Registry support, located at http://registry-kitchen-01.kitchen:8081.
Attributes
Configuration is done by overriding default attributes. All configuration keys have a default defined in [attributes/default.rb](attributes/default.rb). Please read it to have a comprehensive view of what and how you can configure this cookbook behavior.
Recipes
default
Does nothing.
repository
Configure confluent repository.
install-service
Install and fully configure a given service by running repository and its 4 dedicated recipes: package, user, config and service, in that order.
service-package
Install given service from confluent repository.
service-user
Create given service system user and group.
service-config
Generate service configuration. May search for dependencies (like Zookeeper or other nodes of the same cluster) with the help of cluster-search cookbook.
service-service
Install systemd unit for the given service, then enable and start it.
Note: install java package (OpenJDK 8 on centos 7) by default, can be
disabled by setting node['confluent-platform']['java']
to "". A platform
specific configuration for the package to install is also possible.
Resources/Providers
None.
Changelog
Available in [CHANGELOG](CHANGELOG).
Contributing
Please read carefully [CONTRIBUTING.md](CONTRIBUTING.md) before making a merge request.
License and Author
- Author:: Samuel Bernard (samuel.bernard@s4m.io)
Copyright:: 2015, Sam4Mobile
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Dependent cookbooks
cluster-search >= 0.0.0 |
yum >= 0.0.0 |
Contingent cookbooks
There are no cookbooks that are contingent upon this one.
Foodcritic Metric
1.2.0 passed this metric
1.2.0 passed this metric