Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introducing Project Longhorn - April 2016 Rancher Online Meetup

758 views

Published on

In April, Rancher Labs introduced Project Longhorn, a new way to deliver persistent storage for cloud- and container-based deployments. In this online meetup, Rancher labs founders Sheng Liang, Darren Shepherd, and Shannon Williams to introduce and discuss Project Longhorn, a microservices-based solution for distributed, resilient block storage for containers.

To try out Longhorn visit http://github.com/rancher/longhorn

Published in: Technology
  • Be the first to comment

Introducing Project Longhorn - April 2016 Rancher Online Meetup

  1. 1. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Introducing Project Longhorn April 27, 2017 #ranchermeetup
  2. 2. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Shannon Williams Co-Founder/VP Sales @smw355 1 Darren Shepherd Co-Founder/Chief Architect @ibuildthecloud #ranchermeetup
  3. 3. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Sheng Liang Co-Founder/CEO Rancher Labs @shengliang 2 #ranchermeetup Sheng Yang Storage Architect Rancher @yasker
  4. 4. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . First things first… This is a not a ! 3 #ranchermeetup
  5. 5. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .4 There are rules for a meetup! • We won’t be done on time • Questions are always welcome • Demo, then demo some more • Things will break, be patient #ranchermeetup
  6. 6. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .5 Join the conversation on Twitter #ranchermeetup
  7. 7. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Agenda • Rancher Overview – Shannon • Introducing Project Longhorn – Sheng • Latest Rancher Release Details – Darren 6 #ranchermeetup
  8. 8. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Rancher Labs 7 #ranchermeetup The most complete container management platform A simplified Linux distribution built from containers, for containers OUR PRODUCTS A project for microservices-based distributed block storage
  9. 9. © 2017 Rancher Labs, Inc. A complete container management platform that makes it easy to… 8 INNOVATE WITH CONTAINERS without compromising flexibility by empowering developers with fast access to the latest tools MANAGE APPLICATIONS by simplifying day to day application lifecycle management RUN CONTAINERS with the most complete set of container and infrastructure management capabilities Production ready ✔ 20 million+ downloads ✔ Open platform for innovating ✔ Easy to use interface ✔ Multi-tenant ✔ Role based access ✔ 24X7 support ✔ And more….
  10. 10. © 2017 Rancher Labs, Inc. Complete Container Management Platform 9 Application Catalog Container Orchestration and SchedulingUser Mgmt RBAC AD/LDAP SAML Ops Mgmt CI/CD Registries Monitoring Networking Multi-tenant Environments Environment 1 Environment N Infrastructure Services Storage ……. ..Environment 2 Security DNS/LB
  11. 11. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .#ranchermeetup Introducing Project Longhorn
  12. 12. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Choices of container storage Docker – Infinit Red Hat – Ceph, Gluster VMware – vSAN Nutanix – ACS Docker volume plugin NetApp – Docker volume plugin for SolidFire, ONTAP, and E-series EMC – REX-Ray AWS – EBS and EFS Nexenta, Portworx, Quobyte, StorageOS – Containerized storage 11
  13. 13. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Scalable storage clusters 12 Compute Servers Storage Servers
  14. 14. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . 100,000 volumes 100 IOPs, 100MB/s per volume 10M IOPs, 10TB/s in aggregate 13
  15. 15. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . SanDisk (Fusion-io) HA configuration 14
  16. 16. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . One controller for 100,000 volumes A separate controller for each of the 100,000 volumes Use modern orchestration tools to manage these controllers 15
  17. 17. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .16 Host 1 Host 2 Container 3 Controller Replica Replica Container 1 Controller ReplicaReplica Container 2 Controller ReplicaReplica Docker volume Docker volume Docker volume Longhorn data plane How do we manage these?
  18. 18. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .17 Longhorn Flexvolume Plugin Longhorn Docker Volume Plugin Longhorn UI Kubernetes Cluster Cattle/Swarm Cluster Longhorn Manager (Orchestrates all the controllers) Longhorn API Longhorn API Longhorn API Docker Volume Plugin API Kubernetes Flex-volume Plugin API Controller Replica 1 Replica 2 Controller Replica 1 Replica 2 Controller Replica 1 Replica 2 Longhorn control plane etcd …
  19. 19. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .18
  20. 20. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . We are probably not that far away from 1PB SSD drives 19
  21. 21. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Focus on simplicity, reliability, and performance 20 Controller • Mirroring • Rebuild • Encryption Replica • Snapshot • Backup • QoS Does Not Support • Controller HA • Tiering • Striping • Dedup • Compression
  22. 22. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Project Longhorn: distributed block storage system built using containers and microservices 21
  23. 23. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .22 Demo #ranchermeetup
  24. 24. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Longhorn Under the Hood 23
  25. 25. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .24 Oldest Snapshot Newest Snapshot Live Data Use Linux sparse files to store differencing disks 4K block size Read: lazily fill up a read index Write: always to live data, update read index if needed Read index 0 1 2 3 4 5 6 7 0 4 1 5 2 6 3 7
  26. 26. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Backup 25 snap2 snap3 snap2snap3Live Data snap1 Secondary Storage (S3, NFS, …) Primary Storage Backup Restore AWS EBS-style backup Only changed blocks are copied 2M block size
  27. 27. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . How backups are stored 26 volume.cfg backups/ snap2.cfg snap3.cfg blocks/ c0facb6ba3102d29e8d847f32982a030028369020fd5ab6dfc99e63f8a1af903.blk f1af6a6aa6410a1eea5a1ba2a8856cc7bb01b302483e819f3ff4ca46bb17bb16.blk 21935af9e15f5c32c843fbfb6fa01369cc7c0aa0c589f7d1e930bf351f8650c7.blk 731859029215873fdac1c9f2f8bd25a334abf0f3a9e1b057cf2cacc2826d86b0.blk 965b2b6871ebb1b57d1bad2c087aeebc3f7052487b38fac939d655a493b49d06.blk
  28. 28. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Add a new replica (replica rebuild) • Pause controller • Take snapshot of existing replica • Add new replica in WO mode • Unpause controller • Sync snapshots • Set new replica to RW 27 Controller Sync NewReplica LiveData Snapshot Snapshot New Replica NewReplica LiveData Snapshot Existing Replica LiveData Snapshot PAUSED RWWO
  29. 29. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc .28 Frontend: TCMU or Open-iSCSI/tgt /dev/longhorn/vol-name Controller Replica 1 Replica 2 Controller Replica 1 Replica 2 Live upgrade Disk 1 Disk 2
  30. 30. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Two deployment models 29 Hyper-Converged Dedicated Storage Servers Schedule replicas on the same set of hosts as controllers Schedule replicas on dedicated storage servers
  31. 31. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . What works now 1. Distributed volumes on a Docker Swarm cluster 2. Fault detection and replica rebuild 3. Snapshots, backups, and recurring snapshots and backups 4. UI and API 30
  32. 32. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Upcoming work 1. Kubernetes flexvolume driver 2. Deploy Longhorn clusters from Rancher catalog 3. Controller and replica live upgrade 4. Event log for Longhorn orchestration activities (e.g., replica rebuild) 5. Ability to backup to S3 6. Replica scheduling based on disk capacity and IOPS 7. Multiple disks on the same host 8. Volume stats, including throughput and IOPS 9. Authentication and user management of the Longhorn UI and API 10. Volume encryption 11. Performance tuning 31
  33. 33. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Latest Release 32 Rancher 1.5 – March 3, 2017 Key Features: - Templating support for catalog items - API Interceptor - Enhanced Network Policies - Webhooks – Host Scaling, DockerHub upgrade - Metadata Improvements - Portainer.io for Swarm UI
  34. 34. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Next Releases 33 Rancher 1.6 – Late April Key features: - Catalog Enhancements – Ability to add catalogs per environment - HAProxy Enhancements - - Certifying Rancher-EBS - - Scheduling Enhancements - schedule containers evenly across pools of hosts - K8S 1.6 Support with Docker 1.13/Docker 17.03 and etcdv3
  35. 35. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Getting Started Rancher and RancherOS are in GitHub – Get Involved! 34 #ranchermeetup http://github.com/rancher
  36. 36. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Even better - try.rancher.com… 35
  37. 37. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Then join a free training class… 36 http://rancher.com/training
  38. 38. © 2017 Rancher Labs, Inc.© 2017 Rancher Labs, Inc . Thank you rancher.com #ranchermeetup

×