An Interest In:
Web News this Week
- April 26, 2024
- April 25, 2024
- April 24, 2024
- April 23, 2024
- April 22, 2024
- April 21, 2024
- April 20, 2024
Running Kafka on Kubernetes for local development with Helm
Now that I have posted about running Kafka on Kubernetes using Persistent Volumes(PV) and Persistent Volume Claims(PVC), and another post about running it using Storage Classes it's time to grab that knowledge and have a version of it running with Helm Charts.
Helm is a package manager for Kubernetes, Helm helps you manage, define, install, and upgrade Kubernetes applications. It promotes re-usability and standardization.
In this post we will use Helm to convert the Local Kafka setup using Storage Classes I explained in a previous post. So if you haven't seen that post yet I suggest that you read and play with that setup first as it will help you to understand and extend the approach explained here.
Helm has many more features than the ones covered in this simple post.
As usual you can find the source code with instructions in my Github Repository. To clone it: git clone [email protected]:mmaia/kafka-local-kubernetes.git
, have fun.
Pre-reqs, install:
The setup using Helm
In this next section I will just cover the relevant information covering the helm setup and the actual Kubernetes
files used are basically the same as the ones covered within the Running Kafka for local development with Storage Class post, so please refer to that article for further details on the Kubernetes setup.
After you clone the repo as described above if you expand the folder helm
you will find the following structure:
As seen in my previous posts the kind-config.yaml
file is used to setup the local Kubernetes cluster using Kind.
kind.config.yaml
file:
apiVersion: kind.x-k8s.io/v1alpha4kind: Clusternodes: - role: control-plane - role: worker extraPortMappings: - containerPort: 30092 # internal kafka nodeport hostPort: 9092 # port exposed on "host" machine for kafka - containerPort: 30081 # internal schema-registry nodeport hostPort: 8081 # port exposed on "host" machine for schema-registry extraMounts: - hostPath: ./tmp containerPath: /var/local-path-provisioner readOnly: false selinuxRelabel: false propagation: Bidirectional
Then inside the local-kafka-dev
folder you'll find the main Helm Chart file: Chart.yaml
and a file to set variables: values.yaml
.
Chart.yaml
file:
apiVersion: v2 #helm 3name: local-kafka-devappVersion: "7.0.1"description: A Helm chart for a local, single node Confluent Kafka, Schema Registry and Zookeeper for development.version: 1.0.0type: application
values.yaml
file:
confluent: version: 7.0.1 kafka: confluentinc/cp-kafka schemaRegistry: confluentinc/cp-schema-registry zookeeper: confluentinc/cp-zookeeper
Then inside the templates
folder you will find all the same Kubernetes files used in the previous post as mentioned before with same minimal changes to use a few of the Helm variables defined in the values.yaml
file. The only new file in there is the NOTES.txt
which is used to display a nice message after you run the Helm chart as you will be able to see next while running it.
Running it
- Open a terminal and cd to helm folder
- Start Kind running:
kind create cluster --config=kind-config.yml
- Run
helm install ${pick_a_name} local-kafka-dev
. i.e -helm install local-kafka local-kafka-dev
- When done stop kafka setup using
helm uninstall ${name_picked_step_3}
, you may also stop Kind if you want:kind delete cluster
That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes and managed with Helm Charts which you can reach from your application running on your developer machine or IDE.
Have fun!
Photo by ThisisEngineering RAEng on Unsplash
Original Link: https://dev.to/thegroo/running-kafka-on-kubernetes-for-local-development-with-helm-2ne8
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To