This is my installation slides for Kafka Confluent Platform services on Kubernetes. This installation includes the following Confluent Platform services: Kafka brokers ZooKeeper Kafka Connect Confluent Schema Registry Confluent REST Proxy

1. Cluster Info:
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ kubectl get pods
No resources found.
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ kubectl get ns
NAME STATUS AGE
default Active 7m
kube-public Active 7m
kube-system Active 7m
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-kafka-standard-clust-default-pool-028d284a-43rh Ready <none> 7m v1.10.9-gke.5
gke-kafka-standard-clust-default-pool-028d284a-6s6h Ready <none> 7m v1.10.9-gke.5
gke-kafka-standard-clust-default-pool-028d284a-n8l1 Ready <none> 7m v1.10.9-gke.5
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ kubectl get pods --all-
namespacesNAMESPACE NAME READY STATUS RESTARTS AGEkube-system event-exporter-v0.2.3-
54f94754f4-xch4l 2/2 Running 0 16mkube-system fluentd-gcp-scaler-6d7bbc67c5-h7m5c 1/1 Running 0
16mkube-system fluentd-gcp-v3.1.0-c2g4r 2/2 Running 0 15mkube-system fluentd-gcp-v3.1.0-kskkc 2/2
Running 0 15mkube-system fluentd-gcp-v3.1.0-wn2lh 2/2 Running 0 15mkube-system heapster-v1.5.3-
789c467788-skx7d 3/3 Running 0 15mkube-system kube-dns-788979dc8f-kp5rx 4/4 Running 0
16mkube-system kube-dns-788979dc8f-zkl22 4/4 Running 0 15mkube-system kube-dns-autoscaler-
79b4b844b9-ffq6p 1/1 Running 0 16mkube-system kube-proxy-gke-kafka-standard-clust-default-pool-
028d284a-43rh 1/1 Running 0 16mkube-system kube-proxy-gke-kafka-standard-clust-default-pool-
028d284a-6s6h 1/1 Running 0 16mkube-system kube-proxy-gke-kafka-standard-clust-default-pool-
028d284a-n8l1 1/1 Running 0 16mkube-system kubernetes-dashboard-598d75cb96-v25wp 1/1 Running
0 16mkube-system l7-default-backend-5d5b9874d5-pjdhh 1/1 Running 0 16mkube-system metrics-
server-v0.2.1-7486f5bd67-7qwsp 2/2 Running 0 15mkube-system tiller-deploy-7995854f67-gl2cj 1/1
Running 0 10m
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$

2. Installing helm:
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ curl
https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left
Speed
100 7236 100 7236 0 0 46247 0 --:--:-- --:--:-- --:--:-- 46384
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ chmod 700 get_helm.sh
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ ./get_helm.sh
Downloading https://kubernetes-helm.storage.googleapis.com/helm-v2.12.3-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/local/bin
helm installed into /usr/local/bin/helm
tiller installed into /usr/local/bin/tiller
Run 'helm init' to configure helm.
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ helm init
Creating /home/ahmedmekawy/.helm
Creating /home/ahmedmekawy/.helm/repositoryCreating
/home/ahmedmekawy/.helm/repository/cacheCreating
/home/ahmedmekawy/.helm/repository/localCreating
/home/ahmedmekawy/.helm/pluginsCreating
/home/ahmedmekawy/.helm/startersCreating
/home/ahmedmekawy/.helm/cache/archiveCreating
/home/ahmedmekawy/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/ahmedmekawy/.helm.Tiller (the Helm server-side
component) has been installed into your Kubernetes Cluster.

3. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.To
prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-
your-helm-installation
Happy Helming!
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e",
GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3",GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e",
GitTreeState:"clean"}
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$
Creating and Configuring StorageClass:
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ cat storage.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: kafka-sc
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
type: pd-standard
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ kubectl create –f storage.yaml
storageclass.storage.k8s.io "kafka-sc" created
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ kubectl get sc
NAME PROVISIONER AGE
kafka-sc kubernetes.io/gce-pd 8m
standard (default) kubernetes.io/gce-pd 20m
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$

4. The StorageClass that was created can be specified
in dataLogDirStorageClass and dataDirStorageClass under cp-zookeeper and in storageClass under cp-
kafka in values.yaml.
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ grep -i storage values.yaml
dataDirStorageClass: kafka-sc
dataLogDirStorageClass: kafka-sc
storageClass: kafka-sc
Installing Kaka confluent chart:
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ helm repo add confluent
https://confluentinc.github.io/cp-helm-charts/
"confluent" has been added to your repositories
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "confluent" chart repository
...Successfully got an update from the "stable" chart repositoryUpdate
Complete. ⎈ Happy Helming!⎈
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$ helm search confl
NAME CHART VERSION APP VERSION DESCRIPTION
confluent/cp-helm-charts 0.1.0 1.0 A Helm chart for Confluent Open Source
stable/schema-registry-ui 0.2.1 v0.9.4 This is a web tool for the confluentinc/schema-registry i...
ahmedmekawy@cloudshell:~/.kube (essential-rider-208218)$

5. Installation Logs:

6. Presitence Disk Claims:

7. Installation Verification:
ahmedmekawy@cloudshell:~ (essential-rider-208218)$ helm test my-confluent-oss
RUNNING: my-confluent-oss-canary
PASSED: my-confluent-oss-canary
Staring Producer:
ahmedmekawy@cloudshell:~ (essential-rider-208218)$ kubectl exec -c cp-kafka-broker -it my-
confluent-oss-cp-kafka-0 -- /bin/bash /usr/bin/kafka-console-producer --broker-list localhost:9092 --
topic test
>hello from producer
[2019-01-23 08:52:45,848] WARN [Producer clientId=console-producer] Error while fetching metadata
with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>this is 2n producer message>

8. Staring Consumer:
>ahmedmekawy@cloudshell:~ (essential-rider-208218)$ kubectl exec -c cp-kafka-broker -it my-
confluent-oss-cp-kafka-0 -- /bin/bash /usr/bin/kafka-console-consumer -bootstrap-server localhost:9092
--topic test --from-beginning
hello from producer
this is 2n producer message

9. Congratulation Kafka Confluent is installed successfully 
Reference:
https://docs.confluent.io/current/installation/installing_cp/cp-helm-charts/docs/index.html