Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BlackStor - World's fastest & most reliable Cloud Native Software Defined Storage

602 views

Published on

SDS for OpenStack and other cloud platforms - a way faster than CEPH!

Published in: Internet
  • Be the first to comment

  • Be the first to like this

BlackStor - World's fastest & most reliable Cloud Native Software Defined Storage

  1. 1. BlackStor (Based on DRBD) World's fastest & most reliable Cloud Native Software Defined Storage How it came into the world, what’s available now and where it’s headed 1
  2. 2. Copyright © 2016 PROZETA & LINBIT We were simply unhappy with SDS solutions that are available on the market these days… including Ceph. 22
  3. 3. Copyright © 2016 PROZETA & LINBIT 33 Is it even possible to build a new SDS standard?
  4. 4. 4 Who we are
  5. 5. Copyright © 2016 PROZETA & LINBIT ⬢ Since 1991 ⬢ Group annual turnover: 20M EUR (2015) ⬢ Two divisions ➥ Hardware: Value Added Distributor & Reseller ➥ Infrastructure services: IaaS, SaaS, PaaS CE data center infrastructure In-house development About PROZETA 5
  6. 6. Copyright © 2016 PROZETA & LINBIT About PROZETA 6
  7. 7. Copyright © 2016 PROZETA & LINBIT About LINBIT 7 •Open Source DRBD supported by proprietary LINBIT products / services •Hundreds of thousands of DRBD downloads •OpenStack comes with the DRBD Cinder driver •100% founder owned •Offices in Europe and US Best performing SDS block device using common off the shelf hardware Partners References Only replication technology excelling at both synchronous short distance and asynchronous long distance
  8. 8. Copyright © 2016 PROZETA & LINBIT 8 LINBIT’s team is made up of International Open Source experts and has successfully expanded globally with offices in Europe and the US and partnerships across 5 continents • LINBIT founded 2001 • European HQ; Vienna (Austria). US HQ; Portland • Founders are High Availability pioneers • International OSS, kernel and HA lecturers • Most recognized Linux HA team worldwide Incorporation of LINBIT HA 2001 2000 2003 2010 2008 2015 Release of DRBD Open Source Lars Ellenberg joins team as Head of Programming 2007 Release of DRBD Proxy US Office DRBD in Linux Kernel DRBD in thousands of clusters worldwide 2016+ DRBD9 release 9 DRBD in OpenStack PROXY About LINBIT
  9. 9. Copyright © 2016 PROZETA & LINBIT PROZETA & LINBIT together 9 PROZETA cloud experts LINBIT storage & HA gurus COMBINED KNOW-HOW BEST POSSIBLE STARTING POINT FOR A NEW SDS
  10. 10. 10 So what do we really need?
  11. 11. Copyright © 2016 PROZETA & LINBIT SDS Requirements We wanted SDS on the level of Enterprise storage BUT based on a cost-effective open-source technology we understand. 11 ⬢ High Stability ⬢ High Reliability ⬢ High and Predictable Performance ⬢ Scaling options - both Horizontal & Vertical ⬢ Modest resource use ⬢ Simple to manage (plus repair in case of disaster) ⬢ Possible to get info for deep monitoring & metrics ⬢ Extensible: features like Compression, Deduplication, etc.
  12. 12. 12 What’s needed for serious cloud storage? What’s available on the market?
  13. 13. Copyright © 2016 PROZETA & LINBIT What Are The Options? 13 ⬢ IaaS = Volumes / Block Storage ⬢ Servers + Disks + Software ⬢ Local storage ➥ not scalable ⬢ iSCSI ➥ software defined & redundant: hard to find ⬢ GlusterFS, LizardFS, … ➥ too complex to operate... Goal: “Next year I’d like to try out a different storage software solution… ...and I can!”
  14. 14. Copyright © 2016 PROZETA & LINBIT Situation on the Storage market 14 Open-source: ⬢ Often re-inventing wheel ⬢ Too old solutions ⬢ Too complex solutions ⬢ Over-engineered ⬢ Often unstable Proprietary: ⬢ Often very expensive ⬢ Closed source ⬢ Not future proof (Software-defined) ⬢ What if something goes wrong?
  15. 15. Copyright © 2016 PROZETA & LINBIT Look at Ceph... 15 ● No compression ● No deduplication ● Ineffective thin provisioning ● Extremely CPU hungry ● Bad write performance / high latency - SLOOOOOOOOOW… High TCO due to the need for extra hardware, unpredictable performance Perfect integration with OpenStack VERY stable BUT very difficult to fix potential issues in a timely manner 15
  16. 16. Copyright © 2016 PROZETA & LINBIT Need Software Defined Everything... 16
  17. 17. Copyright © 2016 PROZETA & LINBIT Need True Cloud Storage... 17
  18. 18. Copyright © 2016 PROZETA & LINBIT ...that is Fast, Reliable, Scalable 18
  19. 19. Copyright © 2016 PROZETA & LINBIT 19 Definitely requires a solid underlying base technology … Something new, right? 19
  20. 20. Copyright © 2016 PROZETA & LINBIT 20 And then we (re-) discovered DRBD! 20
  21. 21. 21 What is DRBD?
  22. 22. Copyright © 2016 PROZETA & LINBIT What is DRBD? 22 ⬢ 15+ years on the market ⬢ Developed by a stable company with great reputation ⬢ More than 250.000 production installations worldwide ⬢ Kernel modules (DRBD core) are standard part of Linux kernel ⬢ Super-reliable and super-fast!
  23. 23. Copyright © 2016 PROZETA & LINBIT DRBD in 3 minutes 23
  24. 24. Copyright © 2016 PROZETA & LINBIT Why is it so freakin' fast? 24 ⬢ Native Linux block device - not a block device emulation on top of an object store ⬢ Highly optimized meta data layout ⬢ The SDS solution leverages on proven technology ➥ LVM or ZFS’ zVols ⬢ Very well suited for hyperconverged compute and storage
  25. 25. Copyright © 2016 PROZETA & LINBIT DRBD - Key Features 25 ⬢ automatic resync after node or connectivity failure ➥ direction, amount, no full resync needed ⬢ performs under a Linux kernel implementation ➥ 160k IOPs measured (on SSDs, of course) ⬢ multiple volumes per resource (replication group) ➥ write order fidelity within resource ⬢ comes with Pacemaker integration ⬢ synchronous and async replication (LAN and WAN) ⬢ In Linux upstream since 2.6.33 (released 2010)
  26. 26. Copyright © 2016 PROZETA & LINBIT DRBD - New Features in 9.x 26 ⬢ Up to 32 nodes per resource ➥ Fixes the drawbacks of stacking ⬢ Auto promote ⬢ Transport abstraction (TCP, SCTP, RDMA) ⬢ DRBD Manage
  27. 27. Copyright © 2016 PROZETA & LINBIT DRBD Manage 27 ⬢ Provisioning solution for DRBD ⬢ Integrated with OpenStack, OpenNebula, Docker, Proxmox, … ⬢ Volumes ⬢ Snapshots ⬢ Replica count ⬢ Unlimited number of nodes ⬢ Distributed DRBD Kernel driver drbdsetup drbdadm drbdmanage daemon device mapper dmsetup LVM tools drbd cinder driver Drbdmanage CLI tool
  28. 28. Copyright © 2016 PROZETA & LINBIT DRBD Roadmap 28
  29. 29. 29 Let's rock!
  30. 30. Copyright © 2016 PROZETA & LINBIT First implementation 30 ⬢ DRBD Kernel modules + DRBD Utils + DRBD Manage ⬢ DRBD Cinder driver ⬢ Monitoring & Metrics ⬢ Backup
  31. 31. Copyright © 2016 PROZETA & LINBIT Stability & Reliability We have tried to totally break it a million times... with NO SUCCESS! Just what you would expect in a production environment 31
  32. 32. 32 So what now?
  33. 33. BlackStor World's fastest & most reliable Cloud Native Software Defined Storage 33
  34. 34. Copyright © 2016 PROZETA & LINBIT BlackStor 34 JSON HTTP CLI WEB UI API CONTROL ENGINE MONITORING ENGINE MESSAGE BROKER AGENT DRBD AGENT DRBD AGENT DRBD AGENT DRBD BI-DIRECTIONAL INTEGRATIONS TASK MANAGEMENT SCALING DISASTER RECOVERY STATE & PERFORMANCE METRICS ANALYTICS
  35. 35. 35 What more to add?
  36. 36. Copyright © 2016 PROZETA & LINBIT Policy-driven storage 36 ⬢ Use simple policies to control: ➥ Replication strategy ➥ Data placement ➥ Balance speed & data protection (RPO) ➥ QoS & SLA ➥ Backup ➥ … and various other knobs ⬢ Assign policies to / by ➥ Storage objects ➥ Consumer objects (OpenStack tenant, Cinder volume metadata, ...)
  37. 37. Copyright © 2016 PROZETA & LINBIT Multi-tenant & Multi-cloud 37 ⬢ Use the same storage pool for ➥ Multiple OpenStack tenants ➥ Multiple OpenStack clusters ➥ OpenStack & Docker ➥ ….. ⬢ We have QoS, so no issues with multi-tenant environment ⬢ Linked authentication ➥ Use Keystone for storage authentication ➥ Allows OpenStack users to manage resources (if allowed)
  38. 38. Copyright © 2016 PROZETA & LINBIT Scaling 38 ⬢ Horizontally: Scale-out ➥ Add nodes to grow cluster ➥ Hyper-converged nodes (more nodes, lower capacity per node) ⬢ Vertically: Scale-up ➥ Low CPU usage ➥ High density storage (high capacity, high iops) ➥ 76 TB SSD & 1-2M IOPs in one node? Why not? ⬢ Single volume striped across multiple nodes w/ read balancing ➥ Extra boost in performance
  39. 39. Copyright © 2016 PROZETA & LINBIT QoS 39 ⬢ Limit IOPs per ➥ Volume ➥ Volume group ➥ Consumer object (OpenStack tenant, …) ➥ Consumer (OpenStack cluster for multi-cluster environment) ⬢ Set priority ➥ Set priority per policy ⬢ Alpha version - many limitations but usable ⬢ Bandwidth limitation not yet implemented
  40. 40. Copyright © 2016 PROZETA & LINBIT Backup & “Recycle Bin” 40 ⬢ Ability to roll-back without any actual recovery process ➥ Storage snapshots (volume roll-back) ➥ Deferred delete (volume undelete) ⬢ Backup to another host ➥ Running in background ➥ Simple policies ➥ No performance degradation ➥ Encrypted over the wire ⬢ Recover anything ➥ Single volume ➥ Full storage
  41. 41. Copyright © 2016 PROZETA & LINBIT Fully Cloud Aware Storage 41 Everyone's talking about it, but there is none as of now! What it means? ● Bi-directional interfaces (OpenStack, Docker, etc…) ● BlackStor keeps all informations about your volumes & VMs inside ● Simply define QoS/SLA for OpenStack tenant or VM …. …. and see if SLA in not breached in the Web UI
  42. 42. Copyright © 2016 PROZETA & LINBIT Deep insight 42 ⬢ Cloud Native - Data + all metadata in one place ⬢ CLI, Web UI ⬢ Real-time status ⬢ Metrics ➥ IOPs, bandwidth ➥ Latency distribution ➥ Storage health (software, hardware) ➥ … and many more
  43. 43. Copyright © 2016 PROZETA & LINBIT BlackStor - Roadmap 43 ⬢ Short-term ➥ First BlackStor release (incl. new Cinder driver) ➥ Multi-tenant, Multi-pool, ACLs, Policy driven ➥ Web-based interface, single API, CLI tools ➥ Backup, Scale-up & Scale-out ➥ Simple installation (DEB/RPM packages, Puppet/Salt recipes) ⬢ Mid/Longer-term ➥ Advanced features (QoS) ➥ Encryption (both over the wire & data at rest) ➥ New drivers (Nova, Docker, Kubernetes, Windows ?) ➥ Backup policies
  44. 44. 44 How does it work in the real world? (It works in theory… does it work in practice?)
  45. 45. Copyright © 2016 PROZETA & LINBIT Performance - DRBD by itself 45 ● HW ○ 2x IBM 8247-22L ○ Power 8 2 sockets ○ 128 GB RAM ○ Mellanox 100GBps InfiniBand ○ HGST Ultrastart NVMe SSDs ● Ubuntu Xenial on bare metal ● DRBD 9.0.1 & RDMA Transport 2.0 ● fio 2.2.10 ● Random IO
  46. 46. Copyright © 2016 PROZETA & LINBIT Performance - DRBD itself Write 2 node DRBD 46 Write Baseline
  47. 47. Copyright © 2016 PROZETA & LINBIT Performance - DRBD itself Read through DRBD 47 Read baseline
  48. 48. Copyright © 2016 PROZETA & LINBIT Performance - DRBD itself DRBD with read balancing 48 Read baseline
  49. 49. Copyright © 2016 PROZETA & LINBIT DRBD vs CEPH 49 ⬢ 5-node clusters ➥ 2 Class-D SSDs each ➥ 2x E5-2650-L CPUs each ➥ 128GB RAM each ➥ 10GbE each ⬢ Attached VM - 8-core CPU, 16GB RAM ⬢ Ubuntu 14.04.5, Kernel 3.16 ⬢ DRBD9 vs Ceph 10.2.3 ⬢ fio 2.14
  50. 50. Copyright © 2016 PROZETA & LINBIT DRBD vs CEPH 50 Sequential write (IOPS) CPU usage
  51. 51. Copyright © 2016 PROZETA & LINBIT DRBD vs CEPH 51 Random write (IOPS) CPU usage
  52. 52. Copyright © 2016 PROZETA & LINBIT DRBD vs CEPH 52 Sequential read (IOPS) CPU usage
  53. 53. Copyright © 2016 PROZETA & LINBIT DRBD vs CEPH 53 Random read (IOPS) CPU usage
  54. 54. 54 Okay, I’ll launch this into production… but who’ll integrate it? what if something happens?
  55. 55. Copyright © 2016 PROZETA & LINBIT Support? 55 Yes - end-to-end! Including professional services
  56. 56. 56 Let's sum it up!
  57. 57. Copyright © 2016 PROZETA & LINBIT 57 Building a new cloud-native SDS is not an easy task... 57
  58. 58. Copyright © 2016 PROZETA & LINBIT 58 ...but we went for it anyway. Why? Because... ➥ nothing reliable out there ➥ great performance & production ready ➥ sweet spot between open source CEPH & expensive Enterprise storage 58
  59. 59. Copyright © 2016 PROZETA & LINBIT 59 What do you get with this solution? ⬢ Production ready ⬢ Stable & Reliable ⬢ Great & Predictable Performance ⬢ Simple to implement & manage ⬢ Includes deep metrics information ⬢ With additional support options ...and last but not least…. ⬢ Fantastic TCO - it doesn't cost the Earth!
  60. 60. Copyright © 2016 PROZETA & LINBIT Q&A 6060
  61. 61. Copyright © 2016 PROZETA & LINBIT Thank you! PRO-ZETA a.s. Prague, Czech Republic prozeta@prozeta.eu www.prozeta.eu www.tier5.cloud 61 LINBIT HA Solutions GmbH Vienna, Austria sales@linbit.com www.linbit.com 61

×