Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

4K - Kubernetes with Knative, Kafka and Kamel

896 views

Published on

Staring 2019, we started to see “Serverless” taking center stage in all application architectures with enterprise integration and data streaming being no exception. With Kubernetes becoming the standard to run your application workloads in cloud, there isn’t’ one simple way to build highly scalable platform to integrate streaming data and enterprise applications without compromising enterprise compliance. In this session we will explore the four pillars Kubernetes,Knative,Kafka and Kamel to know how they help you build the much needed platform for enterprise integration and data streaming which can scale to infinity without comprising on your cloud resource costs.

Presented as part of Cloud Community Days on 19th June - ccdays.konfhub.com

Published in: Software
  • Be the first to comment

  • Be the first to like this

4K - Kubernetes with Knative, Kafka and Kamel

  1. 1. bit.ly/kubemaster1 1 4K - Kubernetes dn.dev/4k-kubernetes with Knative, Kafka and Kamel
  2. 2. dn.dev/4K-kubernetes Download dn.dev/knative-cookbook
  3. 3. dn.dev/4K-kubernetes Kamesh Sampath(devadvocate.yaml) apiVersion: bio/v1 kind: DeveloperAdvocate spec: title: “Director of Developer Experience @ Red Hat” gde: expert: cloud year: 2019 javaDeveloper: true opensourceContributions: - Eclipse Che - Quarkus - Supersonic Subatomic Java - kubernetes/minikube: Run Kubernetes locally - Camel K social-handles: linkedin: https://linkedin.com/in/kameshsampath github: kameshsampath (Kamesh Sampath) twitter: Kamesh Sampath (@kamesh_sampath) blogs: developers.redhat.com(Kamesh Sampath) kubectl email --address kameshs@redhat.com
  4. 4. dn.dev/4k-kubernetes Agenda Understand four pillars of Cloud Data Streaming ○ Kubernetes ○ Knative ○ Apache Kafka ○ Camel-K(Kamel)
  5. 5. dn.dev/4k-kubernetes Why Kubernetes?
  6. 6. dn.dev/4k-kubernetes Love Thy Mono Old School New School
  7. 7. dn.dev/4k-kubernetes A Challenge Server Hardware Operating System Application Server .war or .ear Java Virtual Machine Custom Configuration Linux Kernel Version & Distribution Java 1.6.6_45 or Java 1.7.0_67 Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y JDBC driver, datasource, JMS queue, users Have you ever had “/” vs “” break your app? Or perhaps needed a unique version of a JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a patch for the JVM or app server that broke your code? Containerize Your App
  8. 8. dn.dev/4k-kubernetes Email MyApp.war has been tested with the following On my Windows 7 desktop JDK 1.8.43 Wildfly 9 Configuration: Datasource: MySQLDS Tested with: mysql-connector-java-5.1.31-bin.jar Production Environment Red Hat Enterprise Linux 6.2 JRE 1.7.3 WebSphere 8.5.5 Oracle 9
  9. 9. dn.dev/4k-kubernetes Dockerfile Container Guest OS Java App Server custom configuration FROM centos/wildfly COPY xyz.xml /opt/myserver/configuration/ COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/ COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/ COPY myapp/target/your.war /opt/wildfly/standalone/deployments/ your.war dependencies
  10. 10. dn.dev/4k-kubernetes DevOps Challenges for Multiple Containers ▪ How to scale? ▪ How to avoid port conflicts? ▪ How to manage them on multiple hosts? ▪ What happens if a host has trouble? ▪ How to keep them running? ▪ How to update them? ▪ Rebuild Container Images? Node Node Node Node Node Logger Node
  11. 11. dn.dev/4k-kubernetes https://www.openshift.com/https://kubernetes.io @kubernetesio @openshift
  12. 12. dn.dev/4k-kubernetes ▪ Greek for “Helmsman,” also the root of the word “Governor” (from latin: gubernator) ▪ Container orchestrator ▪ Supports multiple cloud and bare-metal environments ▪ Inspired by Google’s experience with containers ▪ Open source, written in Go ▪ Manage applications, not machines Meet Kubernetes
  13. 13. dn.dev/4k-kubernetes History of Microservices Continuous Integration via XP 1999 AWS EC2 2006 DropWizard May 2011 Agile Manifesto Feb 2001 NETFLIX to AWS 2010 Ribbon March 2012 Hystrix March 2012 Eureka July 2012 Microservices Assess Thoughtworks Radar March 2012 Spring Boot Sept 2013 Microservices Defined Thoughtworks Fowler, Lewis March 2014 Kubernetes June 2014 Java EE6 2009 DevOps 2009 Docker March 2013 Vert.x June 2011 2 0 1 5
  14. 14. dn.dev/4k-kubernetes https://www.youtube.com/watch?v=GCtpncA0Ea0&feature=youtu.be&t=1031 2015 Launch 1000+ Containers Audience Claims a Container
  15. 15. dn.dev/4k-kubernetes What is Kubernetes?
  16. 16. dn.dev/4k-kubernetes Pods A group of whales is commonly referred to as a pod and a pod usually consists a group of whales that have bonded together either because of biological reasons or through friendships developed between two or more whales. In many cases a typical whale pod consists of anywhere from 2 to 30 whales or more.* *http://www.whalefacts.org/what-is-a-group-of-whales-called/
  17. 17. dn.dev/4k-kubernetes Kubernetes JargonsKubernetes Jargons
  18. 18. dn.dev/4k-kubernetes Pod Replicaset/ Deployment Service Label ✓ 1+ containers ✓ Shared IP ✓ Shared storage (ephemeral) ✓ Shared resources ✓ Shared lifecycle ✓ The Desired State - replicas, pod template: health checks, resources, image ✓ Grouping of pods (acting as one) has stable virtual IP and DNS name ✓ Key/Value pairs associated with Kubernetes objects (env=production) Kubernetes Terms Persistent Volume ✓ Network available storage ✓ PVs and PVCs
  19. 19. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Nodes kubelet kubelet kubelet kubeletkubeletkubelet
  20. 20. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Declarative kubelet kubelet kubelet kubeletkubeletkubelet image: repo/mytomcat:v1 replicas: 4
  21. 21. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - 4 Tomcats kubelet kubelet kubelet kubeletkubeletkubelet
  22. 22. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Pod Fail kubelet kubelet kubelet kubeletkubeletkubelet X
  23. 23. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Correcting kubelet kubelet kubelet kubeletkubeletkubelet
  24. 24. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Node Fail kubelet kubelet kubelet kubeletkubeletkubelet X
  25. 25. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Dev Ops api etcd scheduler controllers kubelet kubelet kubeletkubeletkubelet Kubernetes Cluster - Pods Replaced
  26. 26. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - New Node kubelet kubelet kubeletkubeletkubelet Node kubelet
  27. 27. dn.dev/4k-kubernetes Serverless https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf
  28. 28. dn.dev/4k-kubernetes
  29. 29. dn.dev/4k-kubernetes Serverless Computing “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” The CNCF takes steps toward serverless computing
  30. 30. dn.dev/4k-kubernetes Why we need Serverless ?? Agility in any cloud environment Event driven cloud native applications Focus on business differentiation Consistent and scalable operations Resource and Cost Optimization
  31. 31. dn.dev/4k-kubernetes Serverless Computing “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” The CNCF takes steps toward serverless computing
  32. 32. dn.dev/4k-kubernetes Short History of Serverless AWS S3 March 2006 AWS Lambda November 2014 AWS EC2 August 2006 Serverless Described Thoughtworks August 2016 Serverless Coined October 2012 Serverless Assess Thoughtworks Radar April 2016 AWS Dynamo DB January 2012 Firebase September 2011 Iron.io July 2011 Google Cloud Function Feb 2016 Azure Functions Mar 2016 Pivotal Riff Dec 2017 Oracle Functions Sept 2016 IBM Bluemix OpenWhisk Feb 2016 Oracle Fn Project Oct 2017
  33. 33. dn.dev/4k-kubernetes Architectural Styles Services Microservices Serverless • Autonomous • Loosely Coupled • YourControl • Known Programming Model • Very Matured • Single Purpose • Stateless • Independently Scalable • Automated • Your/Cloud Control • Known Programming Model • Matured • Ephemeral • Cloud Control • Short-Lived Processes • New Programming Model • Event-Driven Async • Immature
  34. 34. dn.dev/4k-kubernetes When to choose what ? Services Microservices Serverless • Autonomous • Loosely Coupled • Single Purpose • Stateless • Independently Scalable • Automated • Single Action • Ephemeral Your Control Cloud Control Portability
  35. 35. dn.dev/4k-kubernetes Microservices Serverless Functions Your Control Long-Lived Processes Known Programming Model Often Sync Request-Response Mature: IDE Integration Debuggers Tracers Monitoring CI/CD Cloud Control Short-Lived Processes New Programming Model Event-Driven Async Immature: ?
  36. 36. dn.dev/4k-kubernetes Microservices Serverless Functions Your Control Long-Lived Processes Known Programming Model Often Sync Request-Response Mature: IDE Integration Debuggers Tracers Monitoring CI/CD Cloud Control Short-Lived Processes New Programming Model Event-Driven Async Immature: ? Knative
  37. 37. dn.dev/4k-kubernetes Serving https://knative.dev/docs/serving @KnativeProject
  38. 38. dn.dev/4k-kubernetes Capabilities ● Rapid application deployment ● Scale-to-zero ○ No pod == no memory ● Scale-from-zero ○ Traffic spike starts N pods ● Configurations and Revisions ○ Deployment snapshots ○ Built-in Blue/Green ● Routing and Network configurations
  39. 39. dn.dev/4k-kubernetes Demo Serving
  40. 40. dn.dev/4k-kubernetes Eventing https://knative.dev/docs/eventing @KnativeProject
  41. 41. dn.dev/4k-kubernetes Exploring Knative Eventing ● Eventing Sources ○ Producers of events e.g. Github, Camel, GCP Pub/Sub,CronJob ● Eventing Channels ○ Event forward or persistence layer e.g. Apache Kafka Channel, GCP Pub/Sub , InMemory ● Brokers ○ Event mesh ● Triggers ○ Connecting Broker and Subscriber ○ Filtering Events
  42. 42. dn.dev/4k-kubernetes Exploring Knative Eventing (contd.,) ● Event Registries ○ Repository of available event types that can be used with Broker ● Event Sink ○ Kubernetes Service ○ Knative Service ○ Channel ○ Broker
  43. 43. dn.dev/4k-kubernetes CNCF, CloudEvents and Serverless https://cloudevents.io/ https://www.cncf.io/
  44. 44. dn.dev/4k-kubernetes Demo Eventing
  45. 45. dn.dev/4k-kubernetes Data Streaming
  46. 46. dn.dev/4k-kubernetes Apache Kafka https://kafka.apache.org @apachekafka
  47. 47. dn.dev/4k-kubernetes Apache Kafka ? ● Developed at Linkedin back in 2010, open sourced in 2011 ● Designed to be fast, scalable, durable and available ● Ability to handle huge number of consumers ● Distributed by nature ● Messages are sent to and received from a topic
  48. 48. dn.dev/4k-kubernetes DevOps challenges - Apache Kafka ● Installation and Configuration is not simple ● Configuring a new cluster: ○ Configuring single brokers ○ Pay attention to conflicting configuration ● Updating a cluster: ○ Updating every single broker ○ Rolling restart of Brokers ○ Rollout and synchronization of Apache Zookeeper
  49. 49. dn.dev/4k-kubernetes http://strimzi.io/ @strimziio Strimzi
  50. 50. dn.dev/4k-kubernetes Topic & User operators Zookeeper Cluster Kafka Cluster Cluster Operator Kafka Custom Resource Deployment Apache Kafka on Kubernetes
  51. 51. dn.dev/4k-kubernetes When to use Apache Kafka ? ● Kafka is a great tool for event-driven architectures ○ Backbone for event driven systems ○ Streaming platform ● Events can have different forms ○ Absolute: Temperature is 21 degrees celsius ○ Deltas: Temperature increased by 1 degree celsius ● Event streams ○ Event sourcing ○ Audit trails ○ Log and metrics analytics
  52. 52. dn.dev/4k-kubernetes Demo
  53. 53. dn.dev/4k-kubernetes Enterprise Integration
  54. 54. dn.dev/4k-kubernetes https://camel.apache.org/ @ApacheCamel
  55. 55. dn.dev/4k-kubernetes What is Apache Camel •The swiss knife of integration •>10 years of development - still one of the most active Apache projects •Java-based integration framework that can be installed on multiple runtimes: Spring-Boot, JEE, JBoss, Karaf, Standalone, ... •Based on Enterprise Integration Patterns •Supports 250+ components •Uses a powerful DSL •Can integrate anything
  56. 56. dn.dev/4k-kubernetes K https://camel.apache.org/ @ApacheCamel Camel K
  57. 57. dn.dev/4k-kubernetes K https://camel.apache.org/ @ApacheCamel
  58. 58. dn.dev/4k-kubernetes Apache Camel-K ? ● Lightweight integration platform to run Apache Camel integrations as Kubernetes applications ● Run integrations as serverless applications ● Ability to integrate with Knative Channels
  59. 59. dn.dev/4k-kubernetes Development Environment kamel CLI Remote Cloud Integration Definition Camel-K Operator Live Update Developer Workflow
  60. 60. dn.dev/4k-kubernetes Demo K
  61. 61. dn.dev/4k-kubernetes The End
  62. 62. dn.dev/4k-kubernetes Resources Tutorials •Kubernetes Tutorial - dn.dev/kube-tutorial •Knative Tutorial – dn.dev/knative-tutorial •Istio Tutorial - dn.dev/istio-tutorial •Quarkus Tutorial - dn.dev/quarkus-tutorial •Tekton Tutorial - dn.dev/tekton-tutorial Master Courses - dn.dev/master DevNation Tech Talks - dn.dev/upcoming
  63. 63. dn.dev/4K-kubernetes Download dn.dev/knative-cookbook
  64. 64. dn.dev/4K-kubernetes Download dn.dev/k8spatterns1
  65. 65. dn.dev/4k-kubernetes Download bit.ly/mono2microdb
  66. 66. dn.dev/4k-kubernetes bit.ly/istiobook Download
  67. 67. dn.dev/4k-kubernetes bit.ly/javamicroservicesbook Download
  68. 68. dn.dev/4k-kubernetes The End

×