Apache Kafka® is a message queueing technology that is used for distributed event streaming.
Kafka captures data in real-time from event sources like microservices in the form of streams of events. It stores these event streams in ‘topics’ for later consumption. Kafka is typically installed alongside ZooKeeper. ZooKeeper has five primary functions. Specifically, ZooKeeper is used for controller election, cluster membership, topic configuration, access control lists, and quotas.
We will use the Confluent platform to deploy Kafka, zookeeper and a graphic console.
For a step-by-step guide, visit our guide on how to create a Kubernetes cluster.
Once you are successful, continue here:
Now that our cluster is ready and we have downloaded the kubeconfig file, let’s verify that our cluster is up and running. To verify that you have a running master and worker node, list the nodes by executing:
If everything works, you should something similar to this:
Now we are ready to deploy Confluent®.
We will be using Helm to install an operator for Confluent. Confluence adds custom resource definitions (CRDs) and you use the resources to define what you need the operator to take care of. First, add the Confluent helm repo and update it.
Next, add the CRDs to kubectl. Download the plugin. Unpack the kubectl plugin that matches your client environment (Linux, Windows or Mac OS (Darwin)) into your client environment local executables directory. This will allow the kubectl to find the plugin. For example, on Linux:
Alternatively, you can use krew to install the plugin.
Create a namespace for Confluent:
Deploy the operator on your cluster :
The output will look like this:
Make sure the operator was created by listing all the resources in the namespace:
Finally, apply the following file that will create Kafka, Zookeeper and ControlCenter:
Apply the configuration:
The creation will take a while. List all the resources in the namespace to track the progress.
Enter the following URL in your browser:
Your browser will present this page:
Kafka is ready for use. Note that there are already many topics defined. These are all internal topics.