Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB
•Download as PPTX, PDF•
6 likes•3,355 views
This session introduced technologies such as Docker, Kubernetes, and Kafka, which are driving the microservices revolution. Learn about containers and orchestration – and most importantly, how to exploit them for stateful services such as MongoDB. What You Will Learn: * Why organizations are choosing to use microservice architectures, what benefits they deliver, and when they should – or shouldn't – be used. * Technologies that are used to build microservices – including containers, orchestration, and messaging systems. * Why MongoDB is a good fit for microservices and what special steps need to be taken to make them work well together.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB
Similar to Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB
- 1. Andrew Morgan – MongoDB Product Marketing POWERING MICROSERVICES WITH DOCKER, KUBERNETES, KAFKA, & MONGODB andrewmorgan
- 2. AGENDA Microservice s What, Why, How? Containers Docker, Kafka Orchestratio n Kubernetes, Mesos, … MongoDB Why, How? When to use Use cases Who, Why? 1 2 3 4 5 6
- 4. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
- 5. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
- 6. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
- 7. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
- 8. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
- 9. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
- 15. MICROSERVICES EXAMPLE Twitter Ingest Snapcha t Ingest Feed merge Faceboo k Ingest Whatsap p Ingest
- 16. MICROSERVICES EXAMPLE Twitter Ingest Snapcha t Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapcha t Ingest Snapcha t Ingest
- 18. CONTAINERS
- 19. CONTAINERS – POWERING MICROSERVICES Real world shipping containers • Road, rail & sea • Contents untouched • Ubiquitous & standardized • Simple • Contents protected • Constraints
- 20. CONTAINERS – POWERING MICROSERVICES Software containers • 1 image -> Many containers ‒ Laptop, DC, cloud ‒ Dev, QA, production, support • Simple, efficient • Isolation • Constraints
- 21. VMS VS. CONTAINERS VM VMVM Bare Metal Host Operating System Hypervisor Guest OS Libraries Apps Service Guest OS Libraries Apps Service Guest OS Libraries Apps Service Container ContainerContainer Bare Metal Host Operating System Docker Engine Libraries Libraries Apps Libraries Apps Service ServiceService
- 22. VMS VS. CONTAINERS VM VMVM Bare Metal Host Operating System Hypervisor Guest OS Libraries Apps Service Guest OS Libraries Apps Service Guest OS Libraries Apps Service Container ContainerContainer Bare Metal Host Operating System Docker Engine Libraries Libraries Apps Libraries Apps Service ServiceService
- 23. VMS VS. CONTAINERS VM VMVM Bare Metal Host Operating System Hypervisor Guest OS Libraries Apps Service Guest OS Libraries Apps Service Guest OS Libraries Apps Service Container ContainerContainer Bare Metal Host Operating System Docker Engine Libraries Libraries Apps Libraries Apps Service ServiceService
- 24. DOCKER • Simple to use • 100K+ images on Docker Hub • Build images from images • Platforms ‒ Linux, OS X, Windows ‒ Laptop, VM, Cloud,… ‒ Cloud services
- 25. RUN MONGODB docker run -d mongo
- 26. TITLE ONLY
- 27. TITLE ONLY
- 28. MICROSERVICE ARCHITECTURES BUILT ON CONTAINERS Many small, focused containers -> sophisticated services • Well defined APIs • Independent languages & libraries • Modular: easy maintenance + reuse • Fault tolerant • Scalable
- 29. CONNECTING THE MICROSERVICES – KAFKA Producer 9 8 7 123... Topic A Consumer
- 30. CONNECTING THE MICROSERVICES – KAFKA Producer 9 8 7 123... Topic A Consumer Producer Consumer
- 31. CONNECTING THE MICROSERVICES – KAFKA Producer 9 8 7 123... Partition 0 Topic A Consumer Producer Consumer 4 3 5 123... Partition 1
- 32. CONNECTING THE MICROSERVICES – KAFKA Producer LEADER Topic A / Partition 0 Broker 1 FOLLOWER Topic A / Partition 1 FOLLOWER Topic A / Partition 0 Broker 2 LEADER Topic A / Partition 1
- 33. CONNECTING THE MICROSERVICES – KAFKA Producer Producer Producer 9 8 7 123 ... Partition 0 4 3 5 123 ... Partition 1 7 3 2 123 ... Partition N Topic A Topic B 7 6 5 123 ... Partition 0 New Old Consumer Consumer
- 34. ORCHESTRATION
- 35. ORCHESTRATION Automated deployment, connecting, and maintenance of multiple containers • Provision hosts • Containers ‒ Instantiate ‒ Reschedule ‒ Link ‒ Scale Out/In • Expose services
- 36. KUBERNETES Created by Google, feature-rich and widely adopted • Deployment and ‘replication’ • On-line scale out/in • Rolling upgrades • High Availability • Persistence • Ports • Load balancing • Google Compute Engine
- 37. APACHE MESOS 10,000s of physical servers; used by Twitter, Airbnb & Apple • Code (“frameworks) vs. declarative • Less feature rich than Kubernetes • Kubernetes as a Mesos framework • Foundation for distributed systems ‒ Apache Aurora, Chronos, Marathon
- 38. CHOOSING AN ORCHESTRATION FRAMEWORK • What you have: ‒ Skills? ‒ DevOps frameworks? ‒ Number of hosts? ‒ Bare metal, VMs, or cloud? • Lifecycle • Features ‒ Automated High Availability? ‒ Grouping and load balancing? ‒ As a service?
- 39. MONGODB
- 40. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
- 41. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
- 42. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
- 43. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
- 44. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
- 45. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
- 46. HOW TO USE MONGODB WITH CONTAINERS Twitter Ingest Snapch at Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapch at Ingest Snapch at Ingest MongoDB Atlas
- 47. HOW TO USE MONGODB WITH CONTAINERS Twitter Ingest Snapch at Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapch at Ingest Snapch at Ingest Kubernetes Ops Mgr agent Ops Mgr agent Ops Mgr agent
- 48. HOW TO USE MONGODB WITH CONTAINERS Twitter Ingest Snapch at Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapch at Ingest Snapch at Ingest Kubernetes mongod mongod mongod
- 49. ORCHESTRATING MONGODB USING KUBERNETES Distributed, stateful application • Persistent volumes • External IP addresses for internal comms • Init MongoDB replica set • Monitor • Backup
- 50. Volume name: mongo-persistent-storage1 pdName: mongodb-disk1 mongodb -disk1 Container name: mongo-node1 image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage1 mountPath: /data/db Docker Hub Registry Pod labels: [name: mongo-node1; instance: rod] ReplicationController name: mongo-rc1 labels: [name: mongo-rc] replicas: 1 selector: [name: mongo -node1] Service: LoadBalancer name: mongo-svc-a labels: [name: mongo-svc-a] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node, instance: rod] ExternalIP Address 104.1.1.1 App 104.1.1.1:27017
- 51. ReplicationControler name: mongo-rc-europe labels: [name: mongo-europe] replicas: 1 selector: [name: mongo -node] Volume name: mongo-persistent-storage pdName: mongodb-disk-europe mongodb- disk-europe Container name: mongo-node image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage mountPath: /data/db Pod labels: [name: mongo-node] Service: LoadBalancer name: mongo-svc-europe labels: [name: mongo-svc-europe] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node] ReplicationControler name: mongo-rc-asia labels: [name: mongo-asia] replicas: 1 selector: [name: mongo -node] Volume name: mongo-persistent-storage pdName: mongodb-disk-asia mongodb- disk-asia Container name: mongo-node image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage mountPath: /data/db Pod labels: [name: mongo-node] Service: LoadBalancer name: mongo-svc-asia labels: [name: mongo-svc-asia] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node] ReplicationControler name: mongo-rc-us labels: [name: mongo-us] replicas: 1 selector: [name: mongo -node] Volume name: mongo-persistent-storage pdName: mongodb-disk-us mongodb- disk-us Container name: mongo-node image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage mountPath: /data/db Pod labels: [name: mongo-node] Service: LoadBalancer name: mongo-svc-us labels: [name: mongo-svc-us] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node]
- 52. STATEFULSETS Beta in Kubernetes 1.5/6 • Stable, predictable, unique network identifiers. ‒ IP addresses may change • Stable, persistent storage • Ordered, graceful deployment and scaling (0 N-1) • Ordered, graceful deletion and termination (N-1 0)
- 53. PUTTING IT INTO PRACTICE
- 55. WHEN TO USE MICROSERVICES
- 56. USE CASES
- 57. MONGODB & MICROSERVICES IN THE WILD
- 58. REFERENCES • Enabling Microservices – Containers & Orchestration Explained https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained • Microservices: The Evolution of Building Modern Applications https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern- applications • Data Streaming with Apache Kafka & MongoDB https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb
- 59. PUTTING IT INTO PRACTICE
Editor's Notes
- 1 + 8 Microservice architectures implement applications as a series of small, self-contained, loosely coupled software components. Each has a specific and well understood role.
- Web & Mobile Apps => Enterprise Micro Web Services Microservices were pioneered in the web and then mobile App worlds; at one time called micro-web-services. Now other enterprises are looking for the same benefits. Microservice architectures implement applications as a series of small, self-contained, loosely coupled software components. Each has a specific and well understood role. Benefits of microservices: - Development Speed - Rapid Iteration Evolve quickly, continuous deployment Isolate impact of changes to existing functions or just add a new one Reactive development Maintainable Independent, empowered work teams
- Spaghetti Code Reverse engineer? No one understands entire code base Change one line of code, impacts scores of other, unexpected places Monolith is like Spaghetti Changing anything impacts everything else. <= 1990s Pre-SOA (monolithic) Tight coupling For a monolith to change, all must agree on each change. Each change has unanticipated effects requiring careful testing beforehand
- Add a new flavor independently. Chef making chocolate knows that well but need not know the others Blueberry goes out of fashion, remove them Need more green cakes, add them Improved pink frosting, throw out the old ones and the new ones. Microservices are like Cupcakes Can add new ones with different flavors, remove ones that you no longer need, add more pink ones if there’s greater demand Developers can create and activate new microservices without prior coordination with others. Their adherence to MSA principles makes continuous delivery of new or modified services possible Greater modularity, looser coupling. Started in the web and mobile app world, moving to Enterprise. Big in media and startups Plan for flexibility rather than reuse
- Each of the ovals represents a microservice. Each source of social media feeds has its own microservice which is specialised in interfacing with the relevant API. Each of those microservices passes messages to the ‘feed merge’ microservice which can then make them available for further microservices to work with. Communication between the microservices is over the network – they can be local to the same machine or distributed. Best practice is for each microservice to be stateless and to have its own database or schema
- Individual microservices can be updated in isolation or even removed if their role is no longer needed
- When a new role (or even a change to an existing one) appears, best practice is to implement a new microservice rather than extending an existing one.
- When a new role (or even a change to an existing one) appears, best practice is to implement a new microservice rather than extending an existing one.
- Microservices allow scale-out. Each type of microservice can be scaled independently – add extra instances just for the functions that are being overworked. Multiple instances of each service can provide High Availability
- Size. Ownership Netflix Cloud Native approach => grow org Best practice is for each microservice to be small enough that a single developer should be able to understand its entire codebase (think in hundreds rather than 10,000s of lines of code). The code for a microservice should be owned by the organization responsible for that function; for example the marketing development team should own the microservice responsible for sending nurture track emails. Netflix was one of the pioneers of Microservices with their ”Cloud Native” approach – it was really all about being able to scale their development organisation.
- 9+9
- Real world shipping containers Same container efficiently transports goods by road, rail, and sea Contents remain untouched through all those environments; no repacking needed Ubiquitous and standardized Simple to use – open, fill, close Contents of each container safe from the others Know how much space the container consumes
- Software containers Build an image containing the full application stack only once Spin up many containers from the same image in multiple environments Laptop, data center, cloud Development, QA, production, support Simple to use & efficient Contents of each container isolated from the others Storage, memory, namespace Constrain resources available to each container Storage, memory, CPU, IO
- The most popular container technology Simple to use and has a rich ecosystem 100,000+ images available from Docker Hub Including mongo hub.docker.com/_/mongo/ Syncs with GitHub projects Define new images built upon base images Define interfaces between containers LINUX, (and now) Windows, and OS X Runs on bare metal, VMs, and cloud. Cloud providers supply the Docker infrastructure (e.g. Google Container Engine)
- A single container can be interesting and useful. O2 Arena
- Microservices built by combining multiple containers Build sophisticated services from many small, focused processes (containers) Well defined APIs between components Each component can use different libraries, middleware & programming languages Modular, decoupled architecture simplifies maintenance and enables reuse Fault tolerant Scalable
- To do useful work, microservices need a way of communicating – Apache Kafka Kafka provides a flexible, scalable, and reliable method to distribute streams of event data from one or more **producers** to one or more **consumers**. Examples of **events** (or **messages**) include: A periodic sensor reading such as the current temperature A user adding an item to the shopping cart in an online store A Tweet being sent with a specific hashtag A log entry generated for each click in a web application Streams of Kafka events are organized into **topics**. A producer chooses a topic to send a given event to and consumers select which topics they pull events from. For example, a financial application could pull NYSE stock trades from one topic, and company financial announcements from another in order to look for trading opportunities. Kafka actually stores all of the messages that it passes around – this makes it ideal for production microservice deployments A microservice can be upgraded and then catch up on everything it missed Or even apply its updated business logic to the full history of events A new microservice can be added and it can be brought up to speed with everything that’s gone before If one service is generating more work than another can keep up with then Kafka operates as a buffer
- To do useful work, microservices need a way of communicating – Apache Kafka Kafka provides a flexible, scalable, and reliable method to distribute streams of event data from one or more **producers** to one or more **consumers**. Examples of **events** (or **messages**) include: A periodic sensor reading such as the current temperature A user adding an item to the shopping cart in an online store A Tweet being sent with a specific hashtag A log entry generated for each click in a web application Streams of Kafka events are organized into **topics**. A producer chooses a topic to send a given event to and consumers select which topics they pull events from. For example, a financial application could pull NYSE stock trades from one topic, and company financial announcements from another in order to look for trading opportunities. Kafka actually stores all of the messages that it passes around – this makes it ideal for production microservice deployments A microservice can be upgraded and then catch up on everything it missed Or even apply its updated business logic to the full history of events A new microservice can be added and it can be brought up to speed with everything that’s gone before If one service is generating more work than another can keep up with then Kafka operates as a buffer
- To do useful work, microservices need a way of communicating – Apache Kafka Kafka provides a flexible, scalable, and reliable method to distribute streams of event data from one or more **producers** to one or more **consumers**. Examples of **events** (or **messages**) include: A periodic sensor reading such as the current temperature A user adding an item to the shopping cart in an online store A Tweet being sent with a specific hashtag A log entry generated for each click in a web application Streams of Kafka events are organized into **topics**. A producer chooses a topic to send a given event to and consumers select which topics they pull events from. For example, a financial application could pull NYSE stock trades from one topic, and company financial announcements from another in order to look for trading opportunities. Kafka actually stores all of the messages that it passes around – this makes it ideal for production microservice deployments A microservice can be upgraded and then catch up on everything it missed Or even apply its updated business logic to the full history of events A new microservice can be added and it can be brought up to speed with everything that’s gone before If one service is generating more work than another can keep up with then Kafka operates as a buffer
- To do useful work, microservices need a way of communicating – Apache Kafka Kafka provides a flexible, scalable, and reliable method to distribute streams of event data from one or more **producers** to one or more **consumers**. Examples of **events** (or **messages**) include: A periodic sensor reading such as the current temperature A user adding an item to the shopping cart in an online store A Tweet being sent with a specific hashtag A log entry generated for each click in a web application Streams of Kafka events are organized into **topics**. A producer chooses a topic to send a given event to and consumers select which topics they pull events from. For example, a financial application could pull NYSE stock trades from one topic, and company financial announcements from another in order to look for trading opportunities. Kafka actually stores all of the messages that it passes around – this makes it ideal for production microservice deployments A microservice can be upgraded and then catch up on everything it missed Or even apply its updated business logic to the full history of events A new microservice can be added and it can be brought up to speed with everything that’s gone before If one service is generating more work than another can keep up with then Kafka operates as a buffer
- To do useful work, microservices need a way of communicating – Apache Kafka Kafka provides a flexible, scalable, and reliable method to distribute streams of event data from one or more **producers** to one or more **consumers**. Examples of **events** (or **messages**) include: A periodic sensor reading such as the current temperature A user adding an item to the shopping cart in an online store A Tweet being sent with a specific hashtag A log entry generated for each click in a web application Streams of Kafka events are organized into **topics**. A producer chooses a topic to send a given event to and consumers select which topics they pull events from. For example, a financial application could pull NYSE stock trades from one topic, and company financial announcements from another in order to look for trading opportunities. Kafka actually stores all of the messages that it passes around – this makes it ideal for production microservice deployments A microservice can be upgraded and then catch up on everything it missed Or even apply its updated business logic to the full history of events A new microservice can be added and it can be brought up to speed with everything that’s gone before If one service is generating more work than another can keep up with then Kafka operates as a buffer
- 18+5
- Automated deployment, connecting, and maintenance of multiple containers Provision hosts Instantiate containers Reschedule failed containers Link containers through defined interfaces Expose services to the outside world Scale out and back in
- Created by Google, feature-rich and widely adopted Automated container deployment and ‘replication’ On-line scale out/in Rolling upgrades HA – automatic rescheduling of failed containers Exposure of network ports to external apps Load balancing over groups of containers providing a service Provided as a service by Google Compute Engine
- Designed to scale to 10,000s of physical servers; used by Twitter, Airbnb & Apple Developer writes code to turn application into a framework to run on Mesos Less feature rich than Kubernetes; considers many functions such as load balancing, rescheduling, and scaling to be a higher level function Project exists to run Kubernetes as a Mesos framework Foundation for distributed systems Apache Aurora, Chronos, Marathon
- Don’t forget Docker Compose Factors to consider… Integration with existing DevOps frameworks? Number of hosts? Bare metal, VMs, or cloud deployment? Automated High Availability? Grouping and load balancing? Existing skills? Install your own orchestration framework or use as a service?
- 23+7
- Kubernetes not designed for stateful services 3 choices: MongoDB outside of container (Atlas?) Kubernetes manages MongoDB Kubernetes manages Ops Manager agent; Ops/Cloud Manager manage MongoDB Orchestrating MongoDB containers requires special treatment as it’s a distributed, stateful application… State should survive rescheduling; use Kubernetes’ persistent volumes abstraction Replica Set members must communicate with each other; expose external IP addresses/ports which survive rescheduling Replica Set must be initialized from exactly one member MongoDB must still be monitored and backed up – MongoDB Cloud Manager
- Kubernetes. Single Pod/comtainer/mongod in a ReplicationController. Use external IP addresses (other IP addresses & hostnames are local to Kubernetes cluster and they change) Refer to white paper for details
- 30+3
- Fast > Elegant Sagrada Familia –1882-2026 (144 years). Frequent, localised changes Localised scaling Upgrades Only if: Scaling team Designing for change Fast is more important than elegant. Change in the application’s functionality and usage is frequent. Change occurs at different rates within the application, so functional isolation and simple integration are more important than module cohesiveness. Functionality is easily separated into simple, isolatable components. When you have the developer/DevOps skillsets. Where development org boundaries match service boundaries. Don’t forget that you’re building a distributed system -> complexity but there are precedents to read up on. One argument is that you shouldn’t bother with microservices unless you need either: - Scale your team - Design for change Sagrada Familia – designed by Gaudi; construction started on March 19, 1882. Expected to be finished in 2026.
- 35+2
- Gap (flexibility): Monolith -> microservice (75 days). New types of PO took just days FuboTV (scalability). Single cluster for dev, QA + production. Cope with 100x bursts. Run MongoDB on Kubernetes Otto (arch == org). Fast, iterative delivery Backcountry (> distributed dev team): Schema changes were taking 20% of dev time. Flexible schema Compare The Market (Use Docker, Kafka, MongoDB & Ops Manager). GAP moved their purchase order system from a monolith architecture to microservices. Due to MongoDB’s flexible schema, it took just 75 days to build the new system. When requirements changed and they had to add new types of purchase orders, it took days instead of months. FuboTV is a North American soccer streaming service. Using Microservices with Kubernetes, Docker & MongoDB. Isolation means that they can use a single cluster of machines (in Google Cloud) for dev, QA & production. Very birsty application – scalability lets them handle 100x increases in traffic. Otto – the key was to have an architecture that fits with their organization. Microservices empower loosely couple development teams (business, project management, IT). This is all enabling Fast test & deployment + Iterative, Continuous Delivery Backcountry.com is an online specialty retailer that sells outdoor clothing and gear. The driver to Microservices for them was a growing, distributed development team. As more and more developers joined and made contributions to the code, the schemas became convoluted and harder to maintain; contributing to 20% of the Scrum backlog. Taking advantage of MongoDB’s flexible data model, Backcountry was able to iterate faster, reduce development time, and mitigate technical debt. Compare The Market: In the cloud, each microservice, or logical grouping of related microservices, is provisioned with its own MongoDB replica set running in Encrypted storage engine to further reduce our security-related surface area. Use Docker, Kafka & MongoDB.