Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to Data Focused Docker Clustering. Docker Hamburg
Similar to Data Focused Docker Clustering. Docker Hamburg (20)
Recently uploaded
Recently uploaded (20)
Data Focused Docker Clustering. Docker Hamburg
- 1. Docker Hamburg The upcoming talk will be presented in English
- 2. Data focused Docker clustering Marcus Hughes
- 3. What was HybridCluster? • Full stack LAMP platform • FreeBSD Chroot containers • ZFS • Auto scaling • Failover • “AwesomeProxy”
- 4. Docker • Containers! • Application level containerisation • Data in “volumes” • Lack of data portability
- 5. Docker containers are portable
- 6. Introduction to Flocker • Open source volume manager • Multi-node “cluster” • 0.1 Released on 13th August (0.2 due on 2nd)
- 7. Flocker
- 8. Proxy layer • Moving an application can be disruptive • Application port responds on any node • Routes reconfigured on migration
- 9. ZFS • Similarities to Docker containers • ZFS on Linux • Resilient • Continuous replication
- 10. Migration • Container stops • Volumes (ZFS dataset) is sent • Routes altered • Container re-started on new host 0.2 • Incremental send
- 12. Application configuration "version": 1 "applications": "mongodb-volume-example": "image": "clusterhq/mongodb" "ports": - "internal": 27017 "external": 27017 "volume": "mountpoint": "/data/db"
- 13. Deployment configuration "version": 1 "nodes": "172.16.255.250": ["mongodb-volume-example"] "172.16.255.251": []
- 14. Deployment configuration flocker-deploy deployment.yml application.yml
- 15. Demo Lets see what these configuration files do!
- 16. Real world uses • Moving applications to production • Replacing servers • Scalability Future • Clone datasets
- 17. The Future Ideas from our past and possible directions
- 18. Contribute! • GitHub https://github.com/ClusterHQ/flocker • Email Marcus@ClusterHQ.com
- 19. Questions?
Editor's Notes
- Berlin specific stuff – After the 8 months in production maybe we can help with some of the issues encountered
- Introduction - English only - Name, About ClusterHQ and HybridCluster
- What We Learnt From HybridCluster - What was HybridCluster - Commercial Product - Pro - Containers over VMs (Lightweight [1]) - Reference IBM research paper (July 21st) - ZFS (supporters of openzfs project) Knew people would like containers and still believe this “All enterprises will use containers” [5] Lack of public documentation This portability would be more useful in general rather than limited to LAMP
- Lets talk Docker Docker - Extremely portable ... but! - Lack of Data Management in Docker - Volumes Caught out eye because containers! Originally based off of LXC and now have own containerisation engine, libContainer Quickly spawn an application (seconds), almost nil boot
- Docker captures the application very well. Makes it easy to provision a new instance elsewhere using “Docker run” as long as you don’t intent to carry over the state. We believe you should be able to carry over the state also. Welcome, Flocker.
- Flocker handled the data of your cluster. Define “cluster”. A group of servers running the same software (Flocker). - Luke described as “homogeneous”. - one node can be in your office, one on digitalocean, one AWS … irrelevant - servers be able to stand in for each other in the even of a failure - One server can fill in for another, more examples at end 0.2 introduces a couple of important features which are explained further
- We feel like we’re able to pick up on the portability where Docker left off. Using technologies before I mentions we can capture the volumes and make them equally as portable. ZFS Proxy – next slide
- Moving data means you need to be careful about inbound connections being routed correctly A port will route to the correct destination (where the docker container is live) When a container is migrated, the route to the port is altered to the new master Future, pause connections when a migration is happening and resume upon completion with the new route (NOT in this version).
- Contrary to belief, ZFS is stable (in citations) Luke liked ZFS to Docker containers Lightweight Portable Supporters of ZFSonLinux project. We employee two developers, one from the original FreeBSD implementation, and another Linux dev. We used ZFS on FreeBSD for HybridCluster and it worked very well. Self healing Pools are great, explain datasets Docker community seems to have this love for Btrfs. Richard Yao made an article about advantages of ZFS (Cite 2). Now supported by Debian. Continuous replication, can take snapshots up until a point in time and send them. (In 0.1.1 this is useful, explain on next slide).
- What happens when we want to move a container from server A to B? - Stop container - Replicate data with ZFS send/receive - Future - Snapshot, send data, stop container, send remaining data (due for 0.2) Future (undefined) Currently Geard manages router HybridCluster had a PythonProxy which is much more configurable Pause TCP connections Unleash upon completion
- All this together we have 3 layers Network Proxy Containers ZFS (local storage) When a migration happens the container is started (downloaded from repo if necessary) and dataset copied Routes recreated User requests handled by ALL nodes in the cluster
- Main thing to start with here, is like Docker… Flocker is split into a node and a CLI. You can run the CLI locally and connect to production flocker nodes. “fig style” application configuration. - Ties application components together - Makes it easy to spawn a full application and all services together Run through each stage of the configuration Coming in 0.2 Environment variables The main feature of Flocker however is to cluster the nodes together.
- So the deployment configuration is where magic happens. This defines which application gets defined on which of the generic servers. It’s a super simple configuration, and explain moving from one node to another is just changing the value of the node
- Super simple one line! Put all this together and have one line which handles your application deployment - routes Containers Datasets
- Follow notes in notes.txt at /home/clusterhq/deployments
- Moving from dev to staging to production Replace server by moving application to another node, removing old node, and maybe adding a new one? LAMP example, all on one node, it gets busy and you want a dedicated MySQL server, easily moved with Flocker The ability to clone configurations, the containers and their associated data. Useful for Dev environments Disposable applications for testing etc, maybe CI?
- Failover Continuous replication - point in time restore? Generic RPM so can be easily installed on existing systems Python proxy for holding requests More application configuration options - Env variables coming soon (0.2) - Building Dockerfiles? Many possibilities, it’s important you contribute Clone data
- Give it a try! Vargant image Tutorial on next slide We always need contributors Give us your thoughts What seems bad? What would you like to see? What direction do you think we should head Fork and contribute code