Data focused docker clustering
•Download as PPTX, PDF•
1 like•1,102 views
See also the more detailed blog post at https://clusterhq.com/blog/data-focused-docker-clustering/ In this talk from the July Docker Edinburgh Meetup Luke Marsden, from ClusterHQ, explains how a data driven approach to clustering can improve robustness and and scalability, while reducing complexity.
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Data focused docker clustering
Similar to Data focused docker clustering (20)
More from Luke Marsden
More from Luke Marsden (9)
Recently uploaded
Recently uploaded (20)
Data focused docker clustering
- 1. Data focused Docker clustering Luke Marsden Founder, ClusterHQ clusterhq.com github.com/clusterhq/flocker
- 2. Background: Container clusters where volumes are a first class citizen
- 4. Modern applications are complex distributed systems
- 5. Containers and PaaS today capture the easy, data-free bits
- 6. We should aim to capture the entire app enabling larger class of apps to move to containers
- 8. Ease of operational manageability Homogeneous servers No single point of failure Multi-region Data services the cluster depends on must be protected by/within the system itself
- 9. Containers should be portable, even when they have state (corollary: the system should expose portable, resilient volumes)
- 10. Dependent services or applications should not need to be reconfigured when a service moves around
- 11. Stateful things scale vertically, stateless things scale horizontally
- 13. Management layer Strongly consistent group messaging semantics Distributed consensus protocols No single master
- 14. Storage ZFS: storage analogue to containers for compute Independently snapshotted and replicated Sharding container volumes across a cluster Redundancy invariant Automatic slave allocation
- 15. Networking AwesomeProxy any request to any node Routes vs. Links
- 16. Live migration Pausing in flight requests Quiesce filesystem Replicate last snapshot Unleash requests
- 17. Thank you - questions? Check it out! github.com/clusterhq/flocker clusterhq.com
Editor's Notes
- So let’s look at what that means today… take a look at a sample modern application. There’s a huge macro trend towards modern applications being complicated distributed systems, as shown in this example. (And this example doesn’t even included monitoring, logging, caching or lock servers.) These sophisticated apps consists of lots of moving parts - typically, a stateless application tier, here shown on the right, and a data tier, shown on the left, indicated with a database icon. Some parts are horizontally scalable, as indicated by double-boxes. Current platforms do not make it easy for you to move these data services, and certainly not automatically. There’s this hugely important class of data services which (of course) depend upon having a resilient writable file system.
- Containers - and PaaS in general - are great at capturing the stateless (data-free) parts of the applications. When well-designed, according to Heroku’s guidelines known as the 12factor manifesto, the application tier is stateless - which btw was a great business move by Heroku, to avoid having to do the hard work. :) Docker has done a great job of helping people to agree on how to encapsulate their stateless app tiers so that they can run in exactly the same way on your laptop, your staging server, or in production. This is the whole shipping container metaphor that everyone’s super excited about. However, these complex applications depend on stateful services too. In a public IaaS or PaaS, there exist external/outsourced data services, but these don’t exist in a standardised or consistent way in a private PaaS let alone on your developer’s laptop, and that’s a huge problem for devops because it means they can’t guarantee consistency between dev and production. We’re here to change that. You can actually containerise these data services today, but as soon as you do, the data gets locked to the server it’s on, and from our years of experience doing ops we know that’s no good because every server fails eventually. Container platforms today have no answer to the problems of data portability, let alone resilience, failover and versioning, all of which are critically important problems to solve for critical data in challenging production environments. In other words, containers are great for dev environment consistency but it’s not ready yet for ops people who want to use it in production. Docker and the Deis, Flynn crowd are super early, and even the more established players like CloudFoundry and OpenShift don’t have answers to these problems. As Adrian says, “watch the hand-wave”. They only tackle the stateless app tier which is a small component of just about every real application, because real applications have databases, queues and key value stores in them!
- The technology that we bring to the table, backed by 5 years of hard computer science, R&D and 7 patents is how you make that data tier portable too, using resilient distributed storage, as well as how to encapsulate the whole application including the data parts in a self-organising cluster with no single point of failure. This is THE big problem that needs to be solved before Docker can be used to put data and applications together into production.