Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Andrew Morgan – MongoDB Product Marketing
POWERING MICROSERVICES
WITH DOCKER,
KUBERNETES, KAFKA, &
MONGODB
andrewmorgan
AGENDA
Microservice
s
What, Why,
How?
Containers
Docker, Kafka
Orchestratio
n
Kubernetes,
Mesos, …
MongoDB
Why, How?
When ...
MICROSERVICES
WHY USE MICROSERVICES? (TL;DR
WEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple comp...
WHY USE MICROSERVICES? (TL;DR
WEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple comp...
WHY USE MICROSERVICES? (TL;DR
WEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple comp...
WHY USE MICROSERVICES? (TL;DR
WEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple comp...
WHY USE MICROSERVICES? (TL;DR
WEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple comp...
WHY USE MICROSERVICES? (TL;DR
WEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple comp...
Monolithic
Cross-Team
Tight Coupling
Small
Change
Huge
Impact
System
Retests
Microservices
Decoupled
Independent
Dev
Isolated
Impact
MICROSERVICES EXAMPLE
Twitter
IngestGoogle+
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
MICROSERVICES EXAMPLE
Twitter
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
MICROSERVICES EXAMPLE
Twitter
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
MICROSERVICES EXAMPLE
Twitter
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
MICROSERVICES EXAMPLE
Twitter
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapcha
t Ingest
Snapch...
DEVELOPMENT TEAMS
CONTAINERS
CONTAINERS – POWERING
MICROSERVICES
Real world shipping containers
• Road, rail & sea
• Contents untouched
• Ubiquitous & ...
CONTAINERS – POWERING
MICROSERVICES
Software containers
• 1 image -> Many containers
‒ Laptop, DC, cloud
‒ Dev, QA, produc...
VMS VS. CONTAINERS
VM VMVM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
...
VMS VS. CONTAINERS
VM VMVM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
...
VMS VS. CONTAINERS
VM VMVM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
...
DOCKER
• Simple to use
• 100K+ images on Docker Hub
• Build images from images
• Platforms
‒ Linux, OS X, Windows
‒ Laptop...
RUN MONGODB
docker run -d mongo
TITLE ONLY
TITLE ONLY
MICROSERVICE ARCHITECTURES BUILT ON
CONTAINERS
Many small, focused containers ->
sophisticated services
• Well defined API...
CONNECTING THE MICROSERVICES –
KAFKA
Producer
9
8
7
123...
Topic A
Consumer
CONNECTING THE MICROSERVICES –
KAFKA
Producer
9
8
7
123...
Topic A
Consumer
Producer Consumer
CONNECTING THE MICROSERVICES –
KAFKA
Producer
9
8
7
123...
Partition 0
Topic A
Consumer
Producer Consumer
4
3
5
123...
Par...
CONNECTING THE MICROSERVICES –
KAFKA
Producer
LEADER
Topic A / Partition 0
Broker 1
FOLLOWER
Topic A / Partition 1
FOLLOWE...
CONNECTING THE MICROSERVICES –
KAFKA
Producer
Producer
Producer
9
8
7
123
...
Partition 0
4
3
5
123
...
Partition 1
7
3
2
...
ORCHESTRATION
ORCHESTRATION
Automated deployment, connecting,
and maintenance of multiple
containers
• Provision hosts
• Containers
‒ In...
KUBERNETES
Created by Google, feature-rich and
widely adopted
• Deployment and ‘replication’
• On-line scale out/in
• Roll...
APACHE MESOS
10,000s of physical servers; used by
Twitter, Airbnb & Apple
• Code (“frameworks) vs.
declarative
• Less feat...
CHOOSING AN ORCHESTRATION
FRAMEWORK
• What you have:
‒ Skills?
‒ DevOps frameworks?
‒ Number of hosts?
‒ Bare metal, VMs, ...
MONGODB
WHY MONGODB IS A GOOD FIT FOR
MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR
MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR
MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR
MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR
MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR
MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
HOW TO USE MONGODB WITH CONTAINERS
Twitter
Ingest
Snapch
at Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapch
at ...
HOW TO USE MONGODB WITH CONTAINERS
Twitter
Ingest
Snapch
at Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapch
at ...
HOW TO USE MONGODB WITH CONTAINERS
Twitter
Ingest
Snapch
at Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapch
at ...
ORCHESTRATING MONGODB USING
KUBERNETES
Distributed, stateful application
• Persistent volumes
• External IP addresses for ...
Volume
name:
mongo-persistent-storage1
pdName: mongodb-disk1
mongodb
-disk1
Container
name: mongo-node1
image: mongo
comma...
ReplicationControler
name: mongo-rc-europe
labels: [name: mongo-europe]
replicas: 1
selector: [name: mongo -node]
Volume
n...
STATEFULSETS
Beta in Kubernetes 1.5/6
• Stable, predictable, unique
network identifiers.
‒ IP addresses may change
• Stabl...
PUTTING IT INTO PRACTICE
WHEN TO USE
MICROSERVICES
WHEN TO USE MICROSERVICES
USE CASES
MONGODB & MICROSERVICES IN THE WILD
REFERENCES
• Enabling Microservices – Containers & Orchestration Explained
https://www.mongodb.com/collateral/microservice...
PUTTING IT INTO PRACTICE
Upcoming SlideShare
Loading in …5
×

Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB

1,562 views

Published on

This session introduced technologies such as Docker, Kubernetes, and Kafka, which are driving the microservices revolution. Learn about containers and orchestration – and most importantly, how to exploit them for stateful services such as MongoDB.

What You Will Learn:

* Why organizations are choosing to use microservice architectures, what benefits they deliver, and when they should – or shouldn't – be used.
* Technologies that are used to build microservices – including containers, orchestration, and messaging systems.
* Why MongoDB is a good fit for microservices and what special steps need to be taken to make them work well together.

Published in: Software
  • Be the first to comment

Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB

  1. 1. Andrew Morgan – MongoDB Product Marketing POWERING MICROSERVICES WITH DOCKER, KUBERNETES, KAFKA, & MONGODB andrewmorgan
  2. 2. AGENDA Microservice s What, Why, How? Containers Docker, Kafka Orchestratio n Kubernetes, Mesos, … MongoDB Why, How? When to use Use cases Who, Why? 1 2 3 4 5 6
  3. 3. MICROSERVICES
  4. 4. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
  5. 5. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
  6. 6. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
  7. 7. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
  8. 8. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
  9. 9. WHY USE MICROSERVICES? (TL;DR WEBSCALE) Speed Change Maintain Scale Empower Build MVP quickly Rapid iterations Simple components Product Team == Component React to market Isolated impact Team Committees
  10. 10. Monolithic Cross-Team Tight Coupling Small Change Huge Impact System Retests
  11. 11. Microservices Decoupled Independent Dev Isolated Impact
  12. 12. MICROSERVICES EXAMPLE Twitter IngestGoogle+ Ingest Snapcha t Ingest Feed merge Faceboo k Ingest
  13. 13. MICROSERVICES EXAMPLE Twitter Ingest Snapcha t Ingest Feed merge Faceboo k Ingest
  14. 14. MICROSERVICES EXAMPLE Twitter Ingest Snapcha t Ingest Feed merge Faceboo k Ingest
  15. 15. MICROSERVICES EXAMPLE Twitter Ingest Snapcha t Ingest Feed merge Faceboo k Ingest Whatsap p Ingest
  16. 16. MICROSERVICES EXAMPLE Twitter Ingest Snapcha t Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapcha t Ingest Snapcha t Ingest
  17. 17. DEVELOPMENT TEAMS
  18. 18. CONTAINERS
  19. 19. CONTAINERS – POWERING MICROSERVICES Real world shipping containers • Road, rail & sea • Contents untouched • Ubiquitous & standardized • Simple • Contents protected • Constraints
  20. 20. CONTAINERS – POWERING MICROSERVICES Software containers • 1 image -> Many containers ‒ Laptop, DC, cloud ‒ Dev, QA, production, support • Simple, efficient • Isolation • Constraints
  21. 21. VMS VS. CONTAINERS VM VMVM Bare Metal Host Operating System Hypervisor Guest OS Libraries Apps Service Guest OS Libraries Apps Service Guest OS Libraries Apps Service Container ContainerContainer Bare Metal Host Operating System Docker Engine Libraries Libraries Apps Libraries Apps Service ServiceService
  22. 22. VMS VS. CONTAINERS VM VMVM Bare Metal Host Operating System Hypervisor Guest OS Libraries Apps Service Guest OS Libraries Apps Service Guest OS Libraries Apps Service Container ContainerContainer Bare Metal Host Operating System Docker Engine Libraries Libraries Apps Libraries Apps Service ServiceService
  23. 23. VMS VS. CONTAINERS VM VMVM Bare Metal Host Operating System Hypervisor Guest OS Libraries Apps Service Guest OS Libraries Apps Service Guest OS Libraries Apps Service Container ContainerContainer Bare Metal Host Operating System Docker Engine Libraries Libraries Apps Libraries Apps Service ServiceService
  24. 24. DOCKER • Simple to use • 100K+ images on Docker Hub • Build images from images • Platforms ‒ Linux, OS X, Windows ‒ Laptop, VM, Cloud,… ‒ Cloud services
  25. 25. RUN MONGODB docker run -d mongo
  26. 26. TITLE ONLY
  27. 27. TITLE ONLY
  28. 28. MICROSERVICE ARCHITECTURES BUILT ON CONTAINERS Many small, focused containers -> sophisticated services • Well defined APIs • Independent languages & libraries • Modular: easy maintenance + reuse • Fault tolerant • Scalable
  29. 29. CONNECTING THE MICROSERVICES – KAFKA Producer 9 8 7 123... Topic A Consumer
  30. 30. CONNECTING THE MICROSERVICES – KAFKA Producer 9 8 7 123... Topic A Consumer Producer Consumer
  31. 31. CONNECTING THE MICROSERVICES – KAFKA Producer 9 8 7 123... Partition 0 Topic A Consumer Producer Consumer 4 3 5 123... Partition 1
  32. 32. CONNECTING THE MICROSERVICES – KAFKA Producer LEADER Topic A / Partition 0 Broker 1 FOLLOWER Topic A / Partition 1 FOLLOWER Topic A / Partition 0 Broker 2 LEADER Topic A / Partition 1
  33. 33. CONNECTING THE MICROSERVICES – KAFKA Producer Producer Producer 9 8 7 123 ... Partition 0 4 3 5 123 ... Partition 1 7 3 2 123 ... Partition N Topic A Topic B 7 6 5 123 ... Partition 0 New  Old Consumer Consumer
  34. 34. ORCHESTRATION
  35. 35. ORCHESTRATION Automated deployment, connecting, and maintenance of multiple containers • Provision hosts • Containers ‒ Instantiate ‒ Reschedule ‒ Link ‒ Scale Out/In • Expose services
  36. 36. KUBERNETES Created by Google, feature-rich and widely adopted • Deployment and ‘replication’ • On-line scale out/in • Rolling upgrades • High Availability • Persistence • Ports • Load balancing • Google Compute Engine
  37. 37. APACHE MESOS 10,000s of physical servers; used by Twitter, Airbnb & Apple • Code (“frameworks) vs. declarative • Less feature rich than Kubernetes • Kubernetes as a Mesos framework • Foundation for distributed systems ‒ Apache Aurora, Chronos, Marathon
  38. 38. CHOOSING AN ORCHESTRATION FRAMEWORK • What you have: ‒ Skills? ‒ DevOps frameworks? ‒ Number of hosts? ‒ Bare metal, VMs, or cloud? • Lifecycle • Features ‒ Automated High Availability? ‒ Grouping and load balancing? ‒ As a service?
  39. 39. MONGODB
  40. 40. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
  41. 41. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
  42. 42. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
  43. 43. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
  44. 44. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
  45. 45. WHY MONGODB IS A GOOD FIT FOR MICROSERVICES Monitoring & Automation Flexible Data Model Redundancy Scalability Simplicity
  46. 46. HOW TO USE MONGODB WITH CONTAINERS Twitter Ingest Snapch at Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapch at Ingest Snapch at Ingest MongoDB Atlas
  47. 47. HOW TO USE MONGODB WITH CONTAINERS Twitter Ingest Snapch at Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapch at Ingest Snapch at Ingest Kubernetes Ops Mgr agent Ops Mgr agent Ops Mgr agent
  48. 48. HOW TO USE MONGODB WITH CONTAINERS Twitter Ingest Snapch at Ingest Feed merge Faceboo k Ingest Whatsap p Ingest Snapch at Ingest Snapch at Ingest Kubernetes mongod mongod mongod
  49. 49. ORCHESTRATING MONGODB USING KUBERNETES Distributed, stateful application • Persistent volumes • External IP addresses for internal comms • Init MongoDB replica set • Monitor • Backup
  50. 50. Volume name: mongo-persistent-storage1 pdName: mongodb-disk1 mongodb -disk1 Container name: mongo-node1 image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage1 mountPath: /data/db Docker Hub Registry Pod labels: [name: mongo-node1; instance: rod] ReplicationController name: mongo-rc1 labels: [name: mongo-rc] replicas: 1 selector: [name: mongo -node1] Service: LoadBalancer name: mongo-svc-a labels: [name: mongo-svc-a] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node, instance: rod] ExternalIP Address 104.1.1.1 App 104.1.1.1:27017
  51. 51. ReplicationControler name: mongo-rc-europe labels: [name: mongo-europe] replicas: 1 selector: [name: mongo -node] Volume name: mongo-persistent-storage pdName: mongodb-disk-europe mongodb- disk-europe Container name: mongo-node image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage mountPath: /data/db Pod labels: [name: mongo-node] Service: LoadBalancer name: mongo-svc-europe labels: [name: mongo-svc-europe] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node] ReplicationControler name: mongo-rc-asia labels: [name: mongo-asia] replicas: 1 selector: [name: mongo -node] Volume name: mongo-persistent-storage pdName: mongodb-disk-asia mongodb- disk-asia Container name: mongo-node image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage mountPath: /data/db Pod labels: [name: mongo-node] Service: LoadBalancer name: mongo-svc-asia labels: [name: mongo-svc-asia] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node] ReplicationControler name: mongo-rc-us labels: [name: mongo-us] replicas: 1 selector: [name: mongo -node] Volume name: mongo-persistent-storage pdName: mongodb-disk-us mongodb- disk-us Container name: mongo-node image: mongo command: mongod –replSet my_replica_set containerPort: 27017 volumeMounts: name: mongo-persistent-storage mountPath: /data/db Pod labels: [name: mongo-node] Service: LoadBalancer name: mongo-svc-us labels: [name: mongo-svc-us] ports: [port: 27017, targetPort: 27017]] selector: [name: mongo-node]
  52. 52. STATEFULSETS Beta in Kubernetes 1.5/6 • Stable, predictable, unique network identifiers. ‒ IP addresses may change • Stable, persistent storage • Ordered, graceful deployment and scaling (0 N-1) • Ordered, graceful deletion and termination (N-1  0)
  53. 53. PUTTING IT INTO PRACTICE
  54. 54. WHEN TO USE MICROSERVICES
  55. 55. WHEN TO USE MICROSERVICES
  56. 56. USE CASES
  57. 57. MONGODB & MICROSERVICES IN THE WILD
  58. 58. REFERENCES • Enabling Microservices – Containers & Orchestration Explained https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained • Microservices: The Evolution of Building Modern Applications https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern- applications • Data Streaming with Apache Kafka & MongoDB https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb
  59. 59. PUTTING IT INTO PRACTICE

×