Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RedisConf17 - Building Large High Performance Redis Databases with Redis Enterprise

1,326 views

Published on

RedisConf17 breakout session

Published in: Technology
  • Be the first to comment

RedisConf17 - Building Large High Performance Redis Databases with Redis Enterprise

  1. 1. Building Large, High Performance Databases with Redis Enterprise using Flash Memory Cihan Biyikoglu VP Product Management - Redis Labs cihan@redislabs.com Frank Ober Solution Architect - Intel frank.ober@intel.com
  2. 2. Agenda 1. Introduction to Redis Labs 2. Building Large Databases with Redis • Redise and Redise Flash Architecture 3. Redise Flash Performance with Intel Optane • Emerging HW from Intel and Redise Flash Benchmark
  3. 3. Introduction To Redis Labs
  4. 4. Redis Labs – Home of Redis • Founded in 2011 • HQ in Mountain View CA, R&D center in Tel-Aviv IL The commercial company behind Open Source Redis Provider of the Redis Enterprise (Redise) technology, platform and products
  5. 5. Redise Cloud Private Redis Labs Products Redise Cloud Redise Pack ManagedRedise Pack SERVICES SOFTWARE Fully managed Redise service in VPCs within AWS, MS Azure, GCP & IBM Softlayer Fully managed Redise service on hosted servers within AWS, MS Azure, GCP, IBM Softlayer, Heroku, CF & OpenShift Downloadable Redise software for any enterprise datacenter or cloud environment Fully managed Redise Pack in private data centers && &
  6. 6. Building Large Database With Redis
  7. 7. Scaling Data with In-Memory Databases Redise In-Memory Database Benefits Fastest data access – faster than disk based databases! Stable & consistent performance as you scale! But RAM is Expensive! >10-20x RAM vs Flash $ Cost/GB
  8. 8. Price/Performance – Memory Technology $9 >$2 $1.0 $0.4 DRAM NV-DIMM/PM NVME SSD SATA SSD $perGBofStorage $ Cost of 1 GB
  9. 9. Why Redise Flash? Massive Datasets with Near-Ram Latency at a Drastically Lower Cost • Optimized Read/Writes with RAM-Extension approach • Gain speed with smart data placement between RAM & Flash • Built for Cloud – take full advantage of “Ephemeral Storage” • Future proof for upcoming persisted-memory technology
  10. 10. Why Redise Flash? Lower cost for large data sets 1024 GB RAM >80% Lower Cost with RAM + Flash Compared to all-in-RAM 100 GB RAM 924 GB Flash
  11. 11. Why Redise Flash Redis on RAM Redise Flash Dataset size 10 TB 10 TB Database size with replication 30 TB 20 TB* AWS instance type x1.32xlarge** i3.16xlarge*** Actual instance size (RAM, and RAM+Flash) 1.46 TB 3.66 TB # of instances needed 21 6+1 Persistent Storage (EBS) 154 TB 110 TB 1 year cost (reserved instances) $1,595,643 $298,896 Savings - 81.27% * Redis Enterprise only needs 1 copy of the data because quorum issues are solved at the node level ** x1 EC2 instances on AWS are optimized for memory $s with the cheapest RAM/GB *** i3 instances on AWS are optimized for flash access with NVMe Storage 10TB with AWS-EC2
  12. 12. Redise Architecture
  13. 13. Redise Technology – Cluster Architecture Redise Cluster Architecture • Shared nothing cluster architecture ◦ Single node type for simple scalability • Fully compatible with open source commands & data structures ◦ Simply change your Redis application connection endpoint to Redise
  14. 14. Redise Technology – Node Architecture Redise Node Architecture Cluster Manager Govern Cluster, Orchestrate Failure Detection, Failover, Stats Collection & more Redise Shards Based on Open Source Redis Secure UI & REST API Allow programmable and visual administration over HTTPS Proxy Scale Connections & Improve Application Performance
  15. 15. Redise Architecture - Single Threaded, In-memory Engine with Persistence - “Lock Free” architecture for fast execution Connection Handler Command Parser Expi ry Evicti on Modules Dispatcher Process Space Disk IO (AOF, Snapshots) Command Dispatcher Background Services Replicati on Listener Redis Event Loop
  16. 16. Redise Architecture - Single Threaded, In-memory Engine with Persistence - “Lock Free” architecture for fast execution - In-memory, optimized for high speed access - Persistence with AOF or Snapshot disk durability “Strings” “Hash” “List” “Sorted Set” “Sets” “Module Types” - … … Key Key Key Key Key Key Key Key Key DISK Storage Space Listener Connection Handler Command Parser Expi ry Evicti on Modules Dispatcher Process Space Disk IO (AOF, Snapshots) Command Dispatcher Background Services Replicati on Redis Event Loop
  17. 17. Redise Architecture – Redise Flash Shard: ◦ Ability to extend RAM to Flash for cheaper storage of data (Redise Flash) Redise Flash Shard “Sting” “Sorted Set” “Set” - - Key Key Key Key Key Key Key Key Key “List” “Module Types” - “Hash” DISK Storage Space Process Space Listener Connection Handler Command Parser Expi ry Evicti on Modules Dispatcher Disk IO (AOF, Snapshots) Command Dispatcher Background Services Replicati on Redis Event Loop
  18. 18. Reading And Writing Data
  19. 19. Read/Write Operation with Redise Proxy 23 1 4 Redise Redis Apps Redis Apps Master Shards Slave Shards 1. App submits the operation. One of the proxies in Redise Receive the Operation - Single Key Ops (GET,STRLEN,HSTRLEN etc) - Multi Key Ops (MGET, BRPOP, EXISTS, TOUCH, etc) 2. Proxy distributes the operations to the corresponding shards in parallel 3. All shards involved in the execution return data to proxy - Fetch values from Flash if not already in RAM - Replication triggers writes to slave shards 4. Proxy assemble responses back to App
  20. 20. DEMO
  21. 21. Redise Flash Performance 21
  22. 22. Redise Flash vs Disk Based Databases? Redise Flash Disk Based Databases Hot Value Handling No IO Required Keep hot values in RAM Heavy IO Required Keeps writing to disk Write Performance Faster Writes Non-Durable Writes with RAM Extension approach* Slower Writes Durable Writes (WAL, Redo logs etc) Cloud Optimized Fast Local Writes to Ephemeral Drive Utilizes the Ephemeral Drive for fast local IO and Network IO for durability Slow Writes to Network Attached Storage CANNOT Utilizes the Ephemeral Drive for fast local IO Future Proof Ready for Persistent Memory Systems like Intel 3D-XPoint Needs Re-Architecting *Redis has durable writes configurable as part of the database configuration as well independent off of the RAM-Extended Flash writes
  23. 23. Redise Flash on Intel® Optane™ SSD vs P3700 2040 1380 590 728 142 64 0 500 1000 1500 2000 2500 95% 85% 50% KOps/sec RAM Hit Ratio % Optane P3700 Up to 9x Higher Throughput item size = 1000B; read/write = 50%/50%
  24. 24. Intel Optane & 3DXpoint Frank Ober Solution Architect - Intel
  25. 25. CPU DELAY MORELESS COST HIGHERLOWER Intel® 3D NAND technology lower cost & higher density “Warm Data” Higher Performance “HOT DATA” Intel® Optane™ technology
  26. 26. 26 Intel® Optane™ SSD DC P4800X Throughput (IOPS) Quality of Service Latency Breakthrough Performance Predictably Fast Service Responsive Under Load Endurance Ultra Endurance
  27. 27. 27 Intel® Optane™ SSD Use Cases DRAM PCIe* PCIe Intel® 3D NAND SSDs Intel® Optane™ SSD Fast Storage Intel® Xeon® ‘memory pool’DRAM PCIe Intel® 3D NAND SSDs Intel® Optane™ SSD DDR DDR PCIe Extend Memory Intel® Xeon® *Other names and brands names may be claimed as the property of others
  28. 28. Engage • Get Started with Redis Enterprise? Signup for Redise Cloud: https://redislabs.com/products/redis-cloud/ Download Redise Pack: https://redislabs.com/downloads • Participate in Previews of Upcoming Technology? Email: pm.group@redislabs.com • Questions on Redis or Redis Enterprise (Redise)? StackOverflow: Tag with “Redis” https://stackoverflow.com/questions/tagged/redis • Find Local Redis Meetups Meetup.com: https://www.meetup.com/San-Francisco-Redis-Meetup/
  29. 29. Thank You! Cihan Biyikoglu VP Product Management - Redis Labs cihan@redislabs.com Frank Ober Solution Architect - Intel frank.ober@intel.com

×