Introducing Project Longhorn - April 2016 Rancher Online Meetup
•Download as PPTX, PDF•
5 likes•1,550 views
In April, Rancher Labs introduced Project Longhorn, a new way to deliver persistent storage for cloud- and container-based deployments. In this online meetup, Rancher labs founders Sheng Liang, Darren Shepherd, and Shannon Williams to introduce and discuss Project Longhorn, a microservices-based solution for distributed, resilient block storage for containers. To try out Longhorn visit http://github.com/rancher/longhorn
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Introducing Project Longhorn - April 2016 Rancher Online Meetup
Similar to Introducing Project Longhorn - April 2016 Rancher Online Meetup (20)
More from Shannon Williams
More from Shannon Williams (11)
Recently uploaded
Recently uploaded (20)
Introducing Project Longhorn - April 2016 Rancher Online Meetup
- 1. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Introducing Project Longhorn April 27, 2017 #ranchermeetup
- 2. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Shannon Williams Co-Founder/VP Sales @smw355 1 Darren Shepherd Co-Founder/Chief Architect @ibuildthecloud #ranchermeetup
- 3. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Sheng Liang Co-Founder/CEO Rancher Labs @shengliang 2 #ranchermeetup Sheng Yang Storage Architect Rancher @yasker
- 4. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . First things first… This is a not a ! 3 #ranchermeetup
- 5. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .4 There are rules for a meetup! • We won’t be done on time • Questions are always welcome • Demo, then demo some more • Things will break, be patient #ranchermeetup
- 6. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .5 Join the conversation on Twitter #ranchermeetup
- 7. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Agenda • Rancher Overview – Shannon • Introducing Project Longhorn – Sheng • Latest Rancher Release Details – Darren 6 #ranchermeetup
- 8. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Rancher Labs 7 #ranchermeetup The most complete container management platform A simplified Linux distribution built from containers, for containers OUR PRODUCTS A project for microservices-based distributed block storage
- 9. © 2017 Rancher Labs, Inc. A complete container management platform that makes it easy to… 8 INNOVATE WITH CONTAINERS without compromising flexibility by empowering developers with fast access to the latest tools MANAGE APPLICATIONS by simplifying day to day application lifecycle management RUN CONTAINERS with the most complete set of container and infrastructure management capabilities Production ready ✔ 20 million+ downloads ✔ Open platform for innovating ✔ Easy to use interface ✔ Multi-tenant ✔ Role based access ✔ 24X7 support ✔ And more….
- 10. © 2017 Rancher Labs, Inc. Complete Container Management Platform 9 Application Catalog Container Orchestration and SchedulingUser Mgmt RBAC AD/LDAP SAML Ops Mgmt CI/CD Registries Monitoring Networking Multi-tenant Environments Environment 1 Environment N Infrastructure Services Storage ……. ..Environment 2 Security DNS/LB
- 11. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .#ranchermeetup Introducing Project Longhorn
- 12. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Choices of container storage Docker – Infinit Red Hat – Ceph, Gluster VMware – vSAN Nutanix – ACS Docker volume plugin NetApp – Docker volume plugin for SolidFire, ONTAP, and E-series EMC – REX-Ray AWS – EBS and EFS Nexenta, Portworx, Quobyte, StorageOS – Containerized storage 11
- 13. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Scalable storage clusters 12 Compute Servers Storage Servers
- 14. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . 100,000 volumes 100 IOPs, 100MB/s per volume 10M IOPs, 10TB/s in aggregate 13
- 15. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . SanDisk (Fusion-io) HA configuration 14
- 16. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . One controller for 100,000 volumes A separate controller for each of the 100,000 volumes Use modern orchestration tools to manage these controllers 15
- 17. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .16 Host 1 Host 2 Container 3 Controller Replica Replica Container 1 Controller ReplicaReplica Container 2 Controller ReplicaReplica Docker volume Docker volume Docker volume Longhorn data plane How do we manage these?
- 18. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .17 Longhorn Flexvolume Plugin Longhorn Docker Volume Plugin Longhorn UI Kubernetes Cluster Cattle/Swarm Cluster Longhorn Manager (Orchestrates all the controllers) Longhorn API Longhorn API Longhorn API Docker Volume Plugin API Kubernetes Flex-volume Plugin API Controller Replica 1 Replica 2 Controller Replica 1 Replica 2 Controller Replica 1 Replica 2 Longhorn control plane etcd …
- 19. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .18
- 20. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . We are probably not that far away from 1PB SSD drives 19
- 21. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Focus on simplicity, reliability, and performance 20 Controller • Mirroring • Rebuild • Encryption Replica • Snapshot • Backup • QoS Does Not Support • Controller HA • Tiering • Striping • Dedup • Compression
- 22. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Project Longhorn: distributed block storage system built using containers and microservices 21
- 23. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .22 Demo #ranchermeetup
- 24. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Longhorn Under the Hood 23
- 25. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .24 Oldest Snapshot Newest Snapshot Live Data Use Linux sparse files to store differencing disks 4K block size Read: lazily fill up a read index Write: always to live data, update read index if needed Read index 0 1 2 3 4 5 6 7 0 4 1 5 2 6 3 7
- 26. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Backup 25 snap2 snap3 snap2snap3Live Data snap1 Secondary Storage (S3, NFS, …) Primary Storage Backup Restore AWS EBS-style backup Only changed blocks are copied 2M block size
- 27. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . How backups are stored 26 volume.cfg backups/ snap2.cfg snap3.cfg blocks/ c0facb6ba3102d29e8d847f32982a030028369020fd5ab6dfc99e63f8a1af903.blk f1af6a6aa6410a1eea5a1ba2a8856cc7bb01b302483e819f3ff4ca46bb17bb16.blk 21935af9e15f5c32c843fbfb6fa01369cc7c0aa0c589f7d1e930bf351f8650c7.blk 731859029215873fdac1c9f2f8bd25a334abf0f3a9e1b057cf2cacc2826d86b0.blk 965b2b6871ebb1b57d1bad2c087aeebc3f7052487b38fac939d655a493b49d06.blk
- 28. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Add a new replica (replica rebuild) • Pause controller • Take snapshot of existing replica • Add new replica in WO mode • Unpause controller • Sync snapshots • Set new replica to RW 27 Controller Sync NewReplica LiveData Snapshot Snapshot New Replica NewReplica LiveData Snapshot Existing Replica LiveData Snapshot PAUSED RWWO
- 29. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .28 Frontend: TCMU or Open-iSCSI/tgt /dev/longhorn/vol-name Controller Replica 1 Replica 2 Controller Replica 1 Replica 2 Live upgrade Disk 1 Disk 2
- 30. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Two deployment models 29 Hyper-Converged Dedicated Storage Servers Schedule replicas on the same set of hosts as controllers Schedule replicas on dedicated storage servers
- 31. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . What works now 1. Distributed volumes on a Docker Swarm cluster 2. Fault detection and replica rebuild 3. Snapshots, backups, and recurring snapshots and backups 4. UI and API 30
- 32. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Upcoming work 1. Kubernetes flexvolume driver 2. Deploy Longhorn clusters from Rancher catalog 3. Controller and replica live upgrade 4. Event log for Longhorn orchestration activities (e.g., replica rebuild) 5. Ability to backup to S3 6. Replica scheduling based on disk capacity and IOPS 7. Multiple disks on the same host 8. Volume stats, including throughput and IOPS 9. Authentication and user management of the Longhorn UI and API 10. Volume encryption 11. Performance tuning 31
- 33. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Latest Release 32 Rancher 1.5 – March 3, 2017 Key Features: - Templating support for catalog items - API Interceptor - Enhanced Network Policies - Webhooks – Host Scaling, DockerHub upgrade - Metadata Improvements - Portainer.io for Swarm UI
- 34. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Next Releases 33 Rancher 1.6 – Late April Key features: - Catalog Enhancements – Ability to add catalogs per environment - HAProxy Enhancements - - Certifying Rancher-EBS - - Scheduling Enhancements - schedule containers evenly across pools of hosts - K8S 1.6 Support with Docker 1.13/Docker 17.03 and etcdv3
- 35. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Getting Started Rancher and RancherOS are in GitHub – Get Involved! 34 #ranchermeetup http://github.com/rancher
- 36. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Even better - try.rancher.com… 35
- 37. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Then join a free training class… 36 http://rancher.com/training
- 38. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Thank you rancher.com #ranchermeetup
Editor's Notes
- Rancher is a complete container management platform that makes it easy to… Run containers in production with the most complete set of container and infrastructure management capabilities Manage applications by simplifying day to day application lifecycle management Innovate with containers without compromising flexibility by empowering developers with fast access to the latest tools And Rancher is production ready Enterprise-class features such as role-based access control, integration with LDAP and Active Directories, detailed audit logs, high-availability management servers and encrypted networking are available out of the box. With over 2.4 million downloads and optional 24x7x365 support, Rancher has quickly become the platform of choice for devops and IT teams who are serious about running containers in production More than 30 customers are using Rancher in production including large enterprise such as US Bank, IBM, Cisco, Invisalign and more.