cookbook 'confluent', '~> 1.3.0'
confluent (8) Versions 1.3.0 Follow2
Installs/Configures confluent
cookbook 'confluent', '~> 1.3.0', :supermarket
knife supermarket install confluent
knife supermarket download confluent
Confluent Cookbook
Installs the Confluent package and can run its services,
- Kafka
- Kafka REST
- Schema Registry
- Kafka Connect
View the [Change Log](CHANGELOG.md) to see what has changed.
Getting Started
Install Confluent Package
If you include the recipe[confluent]
this will install the Confluent package and nothing else.
You can find the package installed under /opt/confluent
(by default) with the name confluent-VERSION
.
You can also find the configuration under,
- /etc/kafka
- /etc/kafka-rest
- /etc/schema-registry
- /etc/kafka-connect
Kafka Service
If you include the recipe[confluent::kafka]
this will install the Confluent package, configure and start the Kafka service.
You can configure the service using the attribtues node["confluent"]["kafka"]["server.properties"][...] = ...
.
Use Confluent's Kafka doc
to figure out the appropriate configuration for yourself.
You can find the SysV script at /etc/init.d/kafka-rest
or service kafka-rest [start|stop|restart|status]
.
You can find the logs at /var/log/confluent/kafka.log
.
Kafka REST Service
If you include the recipe[confluent::kafka-rest]
this will install the Confluent package, configure and start the
Kafka REST service.
You can configure the service using the attribtues node["confluent"]["kafka-rest"]["kafka-rest.properties"][...] = ...
.
Use Confluent's Kafka REST doc to figure out the
appropriate configuration for yourself.
You can find the SysV script at /etc/init.d/kafka-rest
or service kafka-rest [start|stop|restart|status]
.
You can find the logs at /var/log/confluent/kafka-rest.log
.
Schema Registry Service
If you include the recipe[confluent::schema-registry]
this will install the Confluent package, configure and start the
Schema Registry service.
You can configure the service using the attribtues node["confluent"]["schema-registry"]["schema-registry.properties"][...] = ...
.
Use Confluent's Schema Registry doc to figure out the
appropriate configuration for yourself.
You can find the SysV script at /etc/init.d/schema-registry
or service schema-registry [start|stop|restart|status]
.
You can find the logs at /var/log/confluent/schema-registry.log
.
Kafka Connect
If you include the recipe[confluent::kafka-connect]
this will install the Confluent package, configure and start the
Kafka connector. This will listen on port 8083 exposing its rest api to control spinning up new connectors.
You can configure the service using the attribtues node["confluent"]["kafka-connect"]["worker.properties"][...] = ...
.
Use Confluent's Kafka Connect doc to figure out the
appropriate configuration for yourself.
You can find the SysV script at /etc/init.d/kafka-connect
or service kafka-connect [start|stop|restart|status]
.
You can find the logs at /var/log/confluent/kafka-connect.log
.
Zookeeper (For development purposes only)
If you include the recipe[confluent::zookeeper]
this will install the Confluent package and start a zookeeper process. It will listen on port 2181. This is a single zookeeper worker and is not recomended for production use. The primary purpose of this recipe is to get everything running inside vagrant as a self contained system without having to run process on the host machine.
Kerberos
The cookbook helps to setup the services with kerberos enabled. To do so,
- Set
node["confluent"]["kerberos"]["enable"]
totrue
- Set a value for
node["confluent"]["kerberos"]["keytab"]
- Set a value for
node["confluent"]["kerberos"]["realm"]
ornode["confluent"]["kerberos"]["principal"]
The cookbook will create a JAAS configuration file to be used for the services and
automatically include the -Djava.security.auth.login.config
java option.
The cookbook however will not handle creating and kerberos keytab files. That is
outside the scope of this cookbook.
ZooKeeper client authentication can additionally be enabled by setting
node["kafka"]["kerberos"]["enable_zk"]
to true
.
Custom Krb5LoginModule options can be set using the node["kafka"]["kerberos"]["krb5_properties"]
attribute hash for Kafka, or node["kafka"]["kerberos"]["zk_krb5_properties"]
for ZooKeeper (see attributes file for defaults).
Note that enabling Kerberos does not automatically set any configuration into
any of the service's property/config files. Those should be evaluated as well.
Attributes
Generic
-
node["confluent"]["version"]
: The version of the Confluent package to install (default=2.0.1
) -
node["confluent"]["scala_version"]
: The scala version of the Confluent package to install (default=2.11.7
) -
node["confluent"]["artifact_url"]
: The URL to the Confluent package to install. This is generated using theversion
andscala_version
attributes. It downloads frompackages.confluent.io
. -
node["confluent"]["install_dir"]
: The directory to install the Confluent package (default=/opt/confluent
) -
node["confluent"]["user"]
: The user that owns the Confluent package files and runs the services (default=confluent
) -
node["confluent"]["uid"]
: optional staticly assign a uid for above user (default=unset
picks form system config) -
node["confluent"]["group"]
: The group that owns the Confluent package files and runs the services (default=confluent
) -
node["confluent"]["gid"]
: optional staticly assign a gid for above group (default=unset
picks form system config) -
node["confluent"]["backup_templates"]
: If template backups are desired set this to the number of backups to keep. (default=false
) -
node["confluent"]["kerberos"]["enable"]
: if kerberos configuration should be applied (default=false
) -
node["confluent"]["kerberos"]["keytab"]
: the path to the kerberos keytab file (default=nil
) -
node["confluent"]["kerberos"]["realm"]
: the name of the kerberos realm to use (default=nil
) -
node["confluent"]["kerberos"]["principal"]
: the kerberos principal to use (default is dynamically generated from other attributes. See attributes file) -
node["confluent"]["kerberos"]["enable_zk"]
: if zookeeper kerberos configuration should be applied (default=false
) -
node["confluent"]["kerberos"]["krb5_properties"]
: a hash of krb5 key/values used for kerberos kafka server/client configuration (See attributes file for default) -
node["confluent"]["kerberos"]["zk_krb5_properties"]
: a hash of krb5 key/values used for kerberos zookeeper client configuration (See attributes file for default)
Kafka
-
node["confluent"]["kafka"]["server.properties"]
: A Hash of properties that configure the Kafka service (default={}
) -
node["confluent"]["kafka"]["env_vars"]
: A Hash of environment variables applied when running the service -
node["confluent"]["kafka"]["log4j.properties"]
: A Hash of properties that configure log4j for the Kafka service (see attributes for defaults) -
node['confluent']['kafka']['brokers']
: A single broker String or List of brokers by hostname, fqdn, or ipaddress -
node['confluent']['kafka']['zookeepers']
: A list of zookeeper hostname:port's to add to kafka config (default=nil
) -
node['confluent']['kafka']['zookeeper_chroot']
: An optional chroot path for zookeeper hostname:port/chroot's to add to kafka config (default=nil
)
Kafka REST
-
node["confluent"]["kafka-rest"]["kafka-rest.properties"]
: A Hash of properties that configure the Kafka REST service (default={}
) -
node["confluent"]["kafka-rest"]["env_vars"]
: A Hash of environment variables applied when running the service -
node["confluent"]["kafka-rest"]["log4j.properties"]
: A Hash of properties that configure log4j for the Kafka REST service (see attributes for defaults)
Schema Registry
-
node["confluent"]["schema-registry"]["schema-registry.properties"]
: A Hash of properties that configure the Schema Registry service (default={}
) -
node["confluent"]["schema-registry"]["env_vars"]
: A Hash of environment variables applied when running the service -
node["confluent"]["schema-registry"]["log4j.properties"]
: A Hash of properties that configure log4j for the Schema Registry service (see attributes for defaults)
Kafka Connect
-
node["confluent"]["kafka-connect"]["jar_urls"]
: an array of urls to remote files to download and install in the directoryshare/java/kafka-connect-all
located in the extracted confluent directory which is where connect looks by default. -
node["confluent"]["kafka-connect"]["properties_files"]
: a hash where the key is a property file name, and the value is a hash of keys/values for the property file. This is can be used to drop in connector configuration for standalone mode via chef config as opposed to the rest api. -
node["confluent"]["kafka-connect"]["distributed_mode"]
: Boolean used to decide if it should launch in standalone or distributed mode. Defaults to true -
node["confluent"]["kafka-connect"]["worker_properties_file_name"]
: The name of the properties file to use when starting the connect service. -
node["confluent"]["kafka-connect"]["worker.properties"]
: hash of properties to configure the connect properties with. -
node["confluent"]["kafka-connect"]["env_vars"]
: A Hash of environment variables applied when running the service -
node["confluent"]["kafka-connect"]["log4j.properties"]
: A Hash of properties that configure log4j for the Schema Registry service (see attributes for defaults)
Zookeeper
-
node["confluent"]["zookeeper"]["zookeeper.properties"]
: a hash of properties to configure the zookeeper server with
Testing
Style
-
rake style
: runs foodcritic and rubocop- todo in .rubocop.yml
-
rake unit
: runs chefspec tests form ./spec- todo from untouched resources
-
rake kitchen
: runs kitchen tests- problem with gem version conflict in Rakefile. Run kitchen from command line:
kitchen test
- problem with gem version conflict in Rakefile. Run kitchen from command line:
Dependent cookbooks
java >= 0.0.0 |
Contingent cookbooks
There are no cookbooks that are contingent upon this one.
Change Log
1.3.0 - 07-25-2017
- [Enhancement] Issue-28 : make template backups optional
1.2.0 - 07-24-2017
- [Enhancement] Issue-26 : PID File generation for Kafka Monitoring
- [Bug] Issue-25 : Fixed usage of extra kafka connect attribute standalone_connectors to…
- [Enhancement] Issue-23 : Connect should clean up jars not referenced in jar_urls
- [Bug] Issue-20 : Bug with replacing kafka start script
- [Enhancement] Issue-18 : Added support for kerberos
1.1.0 - 12-14-2016
- [Feature] Issue-15 : Kafka connect
1.0.1 - 05-19-2016
- [Bug] Issue-11 : Schema-registry service does not work
1.0.0 - 03-10-2016
- [Enhancement] Issue-10 : Update to confluent 2.0
0.3.0 - 11-06-2015
- [Enhancement] Issue-9 : Add tests and discoverable brokers
0.2.0 - 08-05-2015
- [Enhancement] Issue-7 : Restart services if config files change
- [Enhancement] Issue-6 : Add unit tests
- [Enhancement] Issue-4 : Setup proper logging for services by default
- [Feature] Issue-3 : Allow configuring java settings
- [Feature] Issue-2 : Support schema registry
0.1.0 - 03-30-2015
- [Feature] Issue-1 : Initial working cookbook
Collaborator Number Metric
1.3.0 failed this metric
Failure: Cookbook has 0 collaborators. A cookbook must have at least 2 collaborators to pass this metric.
Contributing File Metric
1.3.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file
Foodcritic Metric
1.3.0 passed this metric
No Binaries Metric
1.3.0 passed this metric
Testing File Metric
1.3.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
1.3.0 passed this metric
1.3.0 failed this metric
1.3.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file
Foodcritic Metric
1.3.0 passed this metric
No Binaries Metric
1.3.0 passed this metric
Testing File Metric
1.3.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
1.3.0 passed this metric
1.3.0 passed this metric
1.3.0 passed this metric
Testing File Metric
1.3.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
1.3.0 passed this metric
1.3.0 failed this metric
1.3.0 passed this metric