Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Docker Multitenancy Problem: A Journey through Infrastructure Hell
Similar to The Docker Multitenancy Problem: A Journey through Infrastructure Hell (20)
Recently uploaded
Recently uploaded (20)
The Docker Multitenancy Problem: A Journey through Infrastructure Hell
- 1. A JOURNEY THROUGH MULTITENANCY WITH DOCKER Peter Klipfel
- 2. STOP ME IF YOU WANT TO LOOK THERE’S A LOT TO SEE
- 3. I WILL WHEN I SEE A TURTLE
- 4. SOME CONTEXT
- 5. WHAT WE’RE TRYING TO DO EACH USER GETS ▸Private data storage ▸Notebook (executable code on our servers) ▸Deployed microservices
- 6. WHAT WE’RE TRYING TO DO WE NEED ▸Scalability ▸Fault Tolerance ▸Security
- 7. HOW HARD IS IT TO CREATE A MULTI- TENANT LET’S START WITH A QUESTION
- 8. VERY HARD
- 9. MULTITENANT ELASTICSEARCH POSSIBLE SOLUTIONS ▸Built in multi tenancy ▸Shield ▸Search-guard
- 10. MULTITENANT ELASTICSEARCH NONE OF THEM WORK ▸Built in multi tenancy: update yml file every user -> restart ▸Shield: Not Free ▸Search-guard: SSL was painful
- 12. HOW CAN WE DO THAT? EACH USER GETS THEIR OWN DATABASE
- 13. ELASTICSEARCH INSTANCE PER USER POSSIBLE SOLUTIONS ▸Use hosted ES: Really expensive ▸Use a cloud provider: expensive ▸Use Docker: not as expensive
- 14. DOCKER TO THE
- 15. HOW DO WE CREATE DOCKER BUT WAIT
- 16. DOCKER CONTAINERS ON DEMAND POSSIBLE SOLUTIONS ▸Mesos (+ marathon) ▸Docker Swarm ▸Kubernetes
- 17. DOCKER CONTAINERS ON DEMAND WHAT ARE THOSE TOOLS? ▸Container schedulers ▸APIs to run a docker container somewhere in the cluster ▸Uniform cluster nodes
- 18. DOCKER CONTAINERS ON DEMAND WHAT ARE THOSE TOOLS? MAST ER AGEN T AGEN T AGEN T AGEN T AGENT CONTA INER CONTA INER CONTA INER
- 20. DOCKER CONTAINERS ON DEMAND THE PROBLEMS ▸How do users get to their services (databases)? ▸What if a node goes down? ▸How do I separate users?
- 21. HOW DO USERS GET TO THEIR
- 22. SERVICE ACCESS WHAT ARE THOSE TOOLS? MAST ER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER REVE RSE PROX Y
- 23. SERVICE ACCESS REVERSE PROXY ▸Nginx (reloads good) ▸HAProxy (reloads bad) ▸And we will need Consul
- 24. SERVICE ACCESS CONSUL: THE EASIEST WAY ▸We need Registrator on every node ▸consul-dns creates routing ▸consul-template builds nginx config
- 25. SERVICE ACCESS NOW OUR REVERSE PROXY WORKS! MAST ER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER REVE RSE PROX Y …
- 28. WHAT IF A NODE GOES DOWN? GREAT! USERS CAN ACCESS THINGS!
- 29. STATEFUL SERVICES PROBLEMS ▸Containers have different fs mounts on each instance ▸Node spin-up is non-deterministic (which disk will it use) ▸Network file systems require implementation changes
- 30. STATEFUL SERVICES SOME SOLUTIONS ▸We can mount docker container filesystems with volumes ▸Can specify certain nodes for services ▸Force stateful services to same node
- 32. STATEFUL SERVICES CLUSTERING ▸Failure is ok, as long as it’s not the whole cluster ▸Storage can be ephemeral ▸Most databases cluster
- 34. HOW DO WE KEEP OUR USERS GREAT! LET’S CLUSTER
- 35. NETWORK ISOLATION THEY’RE ALL ON THE SAME SYSTEM MAST ER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER AGEN T CONTAI NER CONTAI NER CONTAI NER REVE RSE PROX Y …
- 36. NETWORK ISOLATION PROBLEMS WITH CLUSTERING ▸Reverse proxy works only for HTTP ▸Don’t want to DOS the internal network ▸Need isolation between users
- 37. NETWORK ISOLATION SOLUTION: DOCKER OVERLAY NETWORK ▸Weave ▸Callico ▸Flannel
- 41. NETWORK ISOLATION PROBLEMS WITH OPENSTACK ▸Maintaining it sucks ▸Upgrading it sucks ▸Paying for it sucks
- 42. SO I USED OPENSTACK FOR A WHILE
- 43. NETWORK ISOLATION HOW IT WORKED ▸User gets their own account ▸Every user gets their own network ▸Every user gets their own persistent storage
- 45. KUBERNET AND AFTER IT STOPPED SCALING I TRIED
- 46. GOOGLE CONTAINER AND AFTER IT STOPPED SCALING I TRIED
- 47. GOOGLE CONTAINER ENGINE (GKE) THE BEST SOLUTION I HAVE FOUND ▸Persistent volumes ▸Decent library support ▸Hopeful networking promised land
- 48. GOOGLE CONTAINER ENGINE (GKE) PERSISTENT VOLUMES ▸I don’t need automated clustering if disks are persistent ▸Manual deploy for customers that require larger clusters ▸Can separate disk utilization by service
- 49. GOOGLE CONTAINER ENGINE (GKE) HOPEFUL NETWORKING PROMISED LAND ▸Configuration defines subnetwork id ▸Subnets can exist across data centers ▸Lots of opportunities for more clever reverse proxying
- 50. CONCLUSI WHAT HAVE WE LEARNED?
- 51. CONCLUSION WHAT HAVE WE LEARNED? ▸Docker is a glorified package manager ▸Complex microservice architectures are still hard ▸The promised land is close