4K - Kubernetes with Knative, Kafka and Kamel

1. bit.ly/kubemaster1 1 4K - Kubernetes dn.dev/4k-kubernetes with Knative, Kafka and Kamel

2. dn.dev/4K-kubernetes Download dn.dev/knative-cookbook

3. dn.dev/4K-kubernetes Kamesh Sampath(devadvocate.yaml) apiVersion: bio/v1 kind: DeveloperAdvocate spec: title: “Director of Developer Experience @ Red Hat” gde: expert: cloud year: 2019 javaDeveloper: true opensourceContributions: - Eclipse Che - Quarkus - Supersonic Subatomic Java - kubernetes/minikube: Run Kubernetes locally - Camel K social-handles: linkedin: https://linkedin.com/in/kameshsampath github: kameshsampath (Kamesh Sampath) twitter: Kamesh Sampath (@kamesh_sampath) blogs: developers.redhat.com(Kamesh Sampath) kubectl email --address kameshs@redhat.com

4. dn.dev/4k-kubernetes Agenda Understand four pillars of Cloud Data Streaming ○ Kubernetes ○ Knative ○ Apache Kafka ○ Camel-K(Kamel)

5. dn.dev/4k-kubernetes Why Kubernetes?

6. dn.dev/4k-kubernetes Love Thy Mono Old School New School

7. dn.dev/4k-kubernetes A Challenge Server Hardware Operating System Application Server .war or .ear Java Virtual Machine Custom Configuration Linux Kernel Version & Distribution Java 1.6.6_45 or Java 1.7.0_67 Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y JDBC driver, datasource, JMS queue, users Have you ever had “/” vs “” break your app? Or perhaps needed a unique version of a JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a patch for the JVM or app server that broke your code? Containerize Your App

8. dn.dev/4k-kubernetes Email MyApp.war has been tested with the following On my Windows 7 desktop JDK 1.8.43 Wildfly 9 Configuration: Datasource: MySQLDS Tested with: mysql-connector-java-5.1.31-bin.jar Production Environment Red Hat Enterprise Linux 6.2 JRE 1.7.3 WebSphere 8.5.5 Oracle 9

9. dn.dev/4k-kubernetes Dockerfile Container Guest OS Java App Server custom configuration FROM centos/wildfly COPY xyz.xml /opt/myserver/configuration/ COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/ COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/ COPY myapp/target/your.war /opt/wildfly/standalone/deployments/ your.war dependencies

10. dn.dev/4k-kubernetes DevOps Challenges for Multiple Containers ▪ How to scale? ▪ How to avoid port conflicts? ▪ How to manage them on multiple hosts? ▪ What happens if a host has trouble? ▪ How to keep them running? ▪ How to update them? ▪ Rebuild Container Images? Node Node Node Node Node Logger Node

11. dn.dev/4k-kubernetes https://www.openshift.com/https://kubernetes.io @kubernetesio @openshift

12. dn.dev/4k-kubernetes ▪ Greek for “Helmsman,” also the root of the word “Governor” (from latin: gubernator) ▪ Container orchestrator ▪ Supports multiple cloud and bare-metal environments ▪ Inspired by Google’s experience with containers ▪ Open source, written in Go ▪ Manage applications, not machines Meet Kubernetes

13. dn.dev/4k-kubernetes History of Microservices Continuous Integration via XP 1999 AWS EC2 2006 DropWizard May 2011 Agile Manifesto Feb 2001 NETFLIX to AWS 2010 Ribbon March 2012 Hystrix March 2012 Eureka July 2012 Microservices Assess Thoughtworks Radar March 2012 Spring Boot Sept 2013 Microservices Defined Thoughtworks Fowler, Lewis March 2014 Kubernetes June 2014 Java EE6 2009 DevOps 2009 Docker March 2013 Vert.x June 2011 2 0 1 5

14. dn.dev/4k-kubernetes https://www.youtube.com/watch?v=GCtpncA0Ea0&feature=youtu.be&t=1031 2015 Launch 1000+ Containers Audience Claims a Container

15. dn.dev/4k-kubernetes What is Kubernetes?

16. dn.dev/4k-kubernetes Pods A group of whales is commonly referred to as a pod and a pod usually consists a group of whales that have bonded together either because of biological reasons or through friendships developed between two or more whales. In many cases a typical whale pod consists of anywhere from 2 to 30 whales or more.* *http://www.whalefacts.org/what-is-a-group-of-whales-called/

17. dn.dev/4k-kubernetes Kubernetes JargonsKubernetes Jargons

18. dn.dev/4k-kubernetes Pod Replicaset/ Deployment Service Label ✓ 1+ containers ✓ Shared IP ✓ Shared storage (ephemeral) ✓ Shared resources ✓ Shared lifecycle ✓ The Desired State - replicas, pod template: health checks, resources, image ✓ Grouping of pods (acting as one) has stable virtual IP and DNS name ✓ Key/Value pairs associated with Kubernetes objects (env=production) Kubernetes Terms Persistent Volume ✓ Network available storage ✓ PVs and PVCs

19. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Nodes kubelet kubelet kubelet kubeletkubeletkubelet

20. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Declarative kubelet kubelet kubelet kubeletkubeletkubelet image: repo/mytomcat:v1 replicas: 4

21. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - 4 Tomcats kubelet kubelet kubelet kubeletkubeletkubelet

22. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Pod Fail kubelet kubelet kubelet kubeletkubeletkubelet X

23. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Correcting kubelet kubelet kubelet kubeletkubeletkubelet

24. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Node Fail kubelet kubelet kubelet kubeletkubeletkubelet X

25. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Dev Ops api etcd scheduler controllers kubelet kubelet kubeletkubeletkubelet Kubernetes Cluster - Pods Replaced

26. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - New Node kubelet kubelet kubeletkubeletkubelet Node kubelet

27. dn.dev/4k-kubernetes Serverless https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf

28. dn.dev/4k-kubernetes

29. dn.dev/4k-kubernetes Serverless Computing “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” The CNCF takes steps toward serverless computing

30. dn.dev/4k-kubernetes Why we need Serverless ?? Agility in any cloud environment Event driven cloud native applications Focus on business differentiation Consistent and scalable operations Resource and Cost Optimization

31. dn.dev/4k-kubernetes Serverless Computing “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” The CNCF takes steps toward serverless computing

32. dn.dev/4k-kubernetes Short History of Serverless AWS S3 March 2006 AWS Lambda November 2014 AWS EC2 August 2006 Serverless Described Thoughtworks August 2016 Serverless Coined October 2012 Serverless Assess Thoughtworks Radar April 2016 AWS Dynamo DB January 2012 Firebase September 2011 Iron.io July 2011 Google Cloud Function Feb 2016 Azure Functions Mar 2016 Pivotal Riff Dec 2017 Oracle Functions Sept 2016 IBM Bluemix OpenWhisk Feb 2016 Oracle Fn Project Oct 2017

33. dn.dev/4k-kubernetes Architectural Styles Services Microservices Serverless • Autonomous • Loosely Coupled • YourControl • Known Programming Model • Very Matured • Single Purpose • Stateless • Independently Scalable • Automated • Your/Cloud Control • Known Programming Model • Matured • Ephemeral • Cloud Control • Short-Lived Processes • New Programming Model • Event-Driven Async • Immature

34. dn.dev/4k-kubernetes When to choose what ? Services Microservices Serverless • Autonomous • Loosely Coupled • Single Purpose • Stateless • Independently Scalable • Automated • Single Action • Ephemeral Your Control Cloud Control Portability

35. dn.dev/4k-kubernetes Microservices Serverless Functions Your Control Long-Lived Processes Known Programming Model Often Sync Request-Response Mature: IDE Integration Debuggers Tracers Monitoring CI/CD Cloud Control Short-Lived Processes New Programming Model Event-Driven Async Immature: ?

36. dn.dev/4k-kubernetes Microservices Serverless Functions Your Control Long-Lived Processes Known Programming Model Often Sync Request-Response Mature: IDE Integration Debuggers Tracers Monitoring CI/CD Cloud Control Short-Lived Processes New Programming Model Event-Driven Async Immature: ? Knative

37. dn.dev/4k-kubernetes Serving https://knative.dev/docs/serving @KnativeProject

38. dn.dev/4k-kubernetes Capabilities ● Rapid application deployment ● Scale-to-zero ○ No pod == no memory ● Scale-from-zero ○ Traffic spike starts N pods ● Configurations and Revisions ○ Deployment snapshots ○ Built-in Blue/Green ● Routing and Network configurations

39. dn.dev/4k-kubernetes Demo Serving

40. dn.dev/4k-kubernetes Eventing https://knative.dev/docs/eventing @KnativeProject

41. dn.dev/4k-kubernetes Exploring Knative Eventing ● Eventing Sources ○ Producers of events e.g. Github, Camel, GCP Pub/Sub,CronJob ● Eventing Channels ○ Event forward or persistence layer e.g. Apache Kafka Channel, GCP Pub/Sub , InMemory ● Brokers ○ Event mesh ● Triggers ○ Connecting Broker and Subscriber ○ Filtering Events

42. dn.dev/4k-kubernetes Exploring Knative Eventing (contd.,) ● Event Registries ○ Repository of available event types that can be used with Broker ● Event Sink ○ Kubernetes Service ○ Knative Service ○ Channel ○ Broker

43. dn.dev/4k-kubernetes CNCF, CloudEvents and Serverless https://cloudevents.io/ https://www.cncf.io/

44. dn.dev/4k-kubernetes Demo Eventing

45. dn.dev/4k-kubernetes Data Streaming

46. dn.dev/4k-kubernetes Apache Kafka https://kafka.apache.org @apachekafka

47. dn.dev/4k-kubernetes Apache Kafka ? ● Developed at Linkedin back in 2010, open sourced in 2011 ● Designed to be fast, scalable, durable and available ● Ability to handle huge number of consumers ● Distributed by nature ● Messages are sent to and received from a topic

48. dn.dev/4k-kubernetes DevOps challenges - Apache Kafka ● Installation and Configuration is not simple ● Configuring a new cluster: ○ Configuring single brokers ○ Pay attention to conflicting configuration ● Updating a cluster: ○ Updating every single broker ○ Rolling restart of Brokers ○ Rollout and synchronization of Apache Zookeeper

49. dn.dev/4k-kubernetes http://strimzi.io/ @strimziio Strimzi

50. dn.dev/4k-kubernetes Topic & User operators Zookeeper Cluster Kafka Cluster Cluster Operator Kafka Custom Resource Deployment Apache Kafka on Kubernetes

51. dn.dev/4k-kubernetes When to use Apache Kafka ? ● Kafka is a great tool for event-driven architectures ○ Backbone for event driven systems ○ Streaming platform ● Events can have different forms ○ Absolute: Temperature is 21 degrees celsius ○ Deltas: Temperature increased by 1 degree celsius ● Event streams ○ Event sourcing ○ Audit trails ○ Log and metrics analytics

52. dn.dev/4k-kubernetes Demo

53. dn.dev/4k-kubernetes Enterprise Integration

54. dn.dev/4k-kubernetes https://camel.apache.org/ @ApacheCamel

55. dn.dev/4k-kubernetes What is Apache Camel •The swiss knife of integration •>10 years of development - still one of the most active Apache projects •Java-based integration framework that can be installed on multiple runtimes: Spring-Boot, JEE, JBoss, Karaf, Standalone, ... •Based on Enterprise Integration Patterns •Supports 250+ components •Uses a powerful DSL •Can integrate anything

56. dn.dev/4k-kubernetes K https://camel.apache.org/ @ApacheCamel Camel K

57. dn.dev/4k-kubernetes K https://camel.apache.org/ @ApacheCamel

58. dn.dev/4k-kubernetes Apache Camel-K ? ● Lightweight integration platform to run Apache Camel integrations as Kubernetes applications ● Run integrations as serverless applications ● Ability to integrate with Knative Channels

59. dn.dev/4k-kubernetes Development Environment kamel CLI Remote Cloud Integration Definition Camel-K Operator Live Update Developer Workﬂow

60. dn.dev/4k-kubernetes Demo K

61. dn.dev/4k-kubernetes The End

62. dn.dev/4k-kubernetes Resources Tutorials •Kubernetes Tutorial - dn.dev/kube-tutorial •Knative Tutorial – dn.dev/knative-tutorial •Istio Tutorial - dn.dev/istio-tutorial •Quarkus Tutorial - dn.dev/quarkus-tutorial •Tekton Tutorial - dn.dev/tekton-tutorial Master Courses - dn.dev/master DevNation Tech Talks - dn.dev/upcoming

4K - Kubernetes with Knative, Kafka and Kamel

CodeOps Technologies LLP

4K - Kubernetes with Knative, Kafka and Kamel