Your Web News in One Place

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

Sign up for GreenGeeks web hosting
January 17, 2022 02:13 pm GMT

Running Kafka on Kubernetes for local development with Storage class

In a recent post I showed a setup to run Kafka on Kubernetes locally using Persistent Volumes(PV) and Persistent Volume Claims(PVC), this post covers the setup using Kubernetes Storage Class(SC) and leveraging the default one provisioned automatically by Kind which is the Rancher local-path-provisioner.

This setup is simpler and uses less code than the previous one with the trade off of having a bit less control over the path of data externalized to the host machine while requiring some internal knowledge of Kind to set it up. In general I favor this approach for local development over the one from my previous post.

Strimzi is an awesome simpler alternative to achieve the same, check it out. My goal with this setup here is for learning and to have a more "realistic" Kubernetes setup on local development machine so I opted to not use Strimzi or Helm charts.

I have created and tested these approaches on a Linux Development machine. It should work for Mac and Windows with some minimal adjustments also but I have never tried it.

You can get the full source from Github repo where you will find the files and Quick Start for both aforementioned approaches. To clone the repo git clone [email protected]:mmaia/kafka-local-kubernetes.git.

The setup using Storage class

If you checked out the repo described above the setup presented here is under storage-class-setup folder. You will find multiple Kubernetes declarative files in this folder, please notice that you could also combine all files in a single one separating them with a line containing triple dashes ---, if combining them is your preference you can open a terminal and from the storage-class-setup folder run for each in ./kafka-k8s/*; do cat $each; echo "---"; done > local-kafka-combined.yaml this will concatenate all files in a single one called local-kafka-combined.yaml.

The main thing to notice in this setup below compared to the previous one is that you don't have any PV or PVC configuration files this time because we're leveraging the default Rancher local-path-provisioner provided by Kind automatically through it's default Storage class.

I keep them separate to explicitly separate each type in this case and because it's convenient as you can just run kubectl pointing to the directory as described below in the "Running it" section.

kind-config.yaml - This file configures Kind to expose the kafka and schema-registry ports to the local machine host so you can connect your application while developing from your IDE or command line and connect with Kafka running on Kubernetes.

kind-config.yaml - This file configures Kind to expose the kafka and schema-registry ports to the local machine host so you can connect your application while developing from your IDE or command line and connect with Kafka running on Kubernetes it also maps the default path of the Rancher storage provisioner from Kind container to your local host machine.

apiVersion: kind.x-k8s.io/v1alpha4kind: Clusternodes:  - role: control-plane  - role: worker    extraPortMappings:      - containerPort: 30092 # internal kafka nodeport        hostPort: 9092 # port exposed on "host" machine for kafka      - containerPort: 30081 # internal schema-registry nodeport        hostPort: 8081 # port exposed on "host" machine for schema-registry    extraMounts:      - hostPath: ./tmp        containerPath: /var/local-path-provisioner        readOnly: false        selinuxRelabel: false        propagation: Bidirectional

kafka-network-np.yaml - Sets up the internal Kubernetes network used by the setup.

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: kafka-networkspec:  ingress:    - from:        - podSelector:            matchLabels:              network/kafka-network: "true"  podSelector:    matchLabels:      network/kafka-network: "true"

kafka-service.yaml - This file defines the mappings between the internal containers and ports that are exposed, called NodePorts by defaul in Kubernetes nodeports can be used in the range 30000 to 32767.

apiVersion: v1kind: Servicemetadata:  labels:    service: kafka  name: kafkaspec:  selector:    service: kafka  ports:    - name: internal      port: 29092      targetPort: 29092    - name: external      port: 30092      targetPort: 9092      nodePort: 30092  type: NodePort

kafka-ss.yaml - This is the definition of Kafka in this setup, this time we use a Stateful Set.

apiVersion: apps/v1kind: StatefulSetmetadata:  labels:    service: kafka  name: kafkaspec:  serviceName: kafka  replicas: 1  selector:    matchLabels:      service: kafka  template:    metadata:      labels:        network/kafka-network: "true"        service: kafka    spec:      enableServiceLinks: false      containers:      - name: kafka        imagePullPolicy: IfNotPresent        image: confluentinc/cp-kafka:7.0.1        ports:          - containerPort: 29092          - containerPort: 9092        env:          - name: CONFLUENT_SUPPORT_CUSTOMER_ID            value: "anonymous"          - name: KAFKA_ADVERTISED_LISTENERS            value: "INTERNAL://kafka:29092,LISTENER_EXTERNAL://kafka:9092"          - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE            value: "true"          - name: KAFKA_BROKER_ID            value: "1"          - name: KAFKA_DEFAULT_REPLICATION_FACTOR            value: "1"          - name: KAFKA_INTER_BROKER_LISTENER_NAME            value: "INTERNAL"          - name: KAFKA_LISTENERS            value: "INTERNAL://:29092,LISTENER_EXTERNAL://:9092"          - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP            value: "INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT"          - name: KAFKA_NUM_PARTITIONS            value: "1"          - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR            value: "1"          - name: KAFKA_LOG_CLEANUP_POLICY            value: "compact"          - name: KAFKA_ZOOKEEPER_CONNECT            value: "zookeeper:2181"        resources: {}        volumeMounts:          - mountPath: /var/lib/kafka/data            name: kafka-data      hostname: kafka      restartPolicy: Always  volumeClaimTemplates:    - metadata:        name: kafka-data      spec:        accessModes:          - ReadWriteOnce        resources:          requests:            storage: 1Gi

The remaining files are declarative Kubernetes configurations files to schema-registry and zookeeper.

schema-registry-deployment.yaml

apiVersion: apps/v1kind: Deploymentmetadata:  labels:    service: schema-registry  name: schema-registryspec:  replicas: 1  selector:    matchLabels:      service: schema-registry  strategy: {}  template:    metadata:      labels:        network/kafka-network: "true"        service: schema-registry    spec:      enableServiceLinks: false      containers:        - env:            - name: SCHEMA_REGISTRY_HOST_NAME              value: "schema-registry"            - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS              value: "kafka:29092"            - name: SCHEMA_REGISTRY_LISTENERS              value: "http://0.0.0.0:30081"          image: confluentinc/cp-schema-registry:7.0.1          name: schema-registry          ports:            - containerPort: 30081          resources: {}      hostname: schema-registry      restartPolicy: Always

schema-registry-service.yaml

apiVersion: v1kind: Servicemetadata:  labels:    service: schema-registry  name: schema-registryspec:  ports:    - port: 30081      name: outport      targetPort: 30081      nodePort: 30081  type: NodePort  selector:    service: schema-registry

zookeeper-service.yaml

apiVersion: v1kind: Servicemetadata:  labels:    service: zookeeper  name: zookeeperspec:  ports:    - name: "2181"      port: 2181      targetPort: 2181  selector:    service: zookeeper

zookeeper-ss.yaml - Again the main difference this time is the usage of Stateful Set.

apiVersion: apps/v1kind: StatefulSetmetadata:  labels:    service: zookeeper  name: zookeeperspec:  serviceName: zookeeper  replicas: 1  selector:    matchLabels:      service: zookeeper  template:    metadata:      labels:        network/kafka-network: "true"        service: zookeeper    spec:      enableServiceLinks: false      containers:        - name: zookeeper          imagePullPolicy: IfNotPresent          image: confluentinc/cp-zookeeper:7.0.1          ports:            - containerPort: 2181          env:            - name: ZOOKEEPER_CLIENT_PORT              value: "2181"            - name: ZOOKEEPER_DATA_DIR              value: "/var/lib/zookeeper/data"            - name: ZOOKEEPER_LOG_DIR              value: "/var/lib/zookeeper/log"            - name: ZOOKEEPER_SERVER_ID              value: "1"          resources: {}          volumeMounts:            - mountPath: /var/lib/zookeeper/data              name: zookeeper-data            - mountPath: /var/lib/zookeeper/log              name: zookeeper-log      hostname: zookeeper      restartPolicy: Always  volumeClaimTemplates:    - metadata:        name: zookeeper-data      spec:        accessModes:          - ReadWriteOnce        resources:          requests:            storage: 250Mi    - metadata:        name: zookeeper-log      spec:        accessModes:          - ReadWriteOnce        resources:          requests:            storage: 250Mi

Running it

  1. Open a terminal and cd to the storage-class-setup folder.
  2. Create a folder called "tmp", this is where the storage will be automatically provisioned by the default Kind storageclass.
  3. Run kind specifying configuration: kind create cluster --config=kind-config.yml. This will start a Kind kubernetescontrol plane + worker.
  4. Run kubernetes configuration for kafka kubectl apply -f kafka-k8s
  5. When done stop kubernetes objects: kubectl delete -f kafka-k8s and then if you want also stop the kind clusterwhich will also delete the storage on the host machine: kind delete cluster.

After running the kubectl apply command(step 4 above) check your local tmp folder where you will find the automated storage mapped to your local host disk, notice that those folders will be deleted when you shutdown the Kind cluster but they will persist over pod restarts of Kafka and zookeeper.

That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes that you can reach from your application running on your developer machine or IDE.

Photo by Fotis Fotopoulos on Unsplash


Original Link: https://dev.to/thegroo/running-kafka-on-kubernetes-for-local-development-with-storage-class-4oa9

Share this article:    Share on Facebook
View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To