Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Day 2 General Session Presentations RedisConf

611 views

Published on

Yiftach Shoolman
Adrian Colyer
Laura Merling
Reynold Xin

Published in: Technology
  • Be the first to comment

Day 2 General Session Presentations RedisConf

  1. 1. Salvatore Sanfilippo Creator of Redis
  2. 2. Yiftach Shoolman Co-Founder & CTO Redis Labs
  3. 3. 2 Hot Topics + 1 New Trend
  4. 4. Redis on New Memory Technologies
  5. 5. 0.10 250.00 0.10 500.00 0.10 1.00 10.00 100.00 1,000.00 $0 $1 $2 $3 $4 $5 $6 $7 $8 $9 $10 DRAM NV-DIMM/PM NVMe SSD SATA SSD microsec $/GB 1 GB cost vs. Read/Write Performance 1 GB cost Read latency Write latency $9 $0.4 High DRAM Cost and Low Flash Performance
  6. 6. High DRAM Cost and Low Flash Performance 0.10 90.00 250.00 0.10 25.00 500.00 0.10 1.00 10.00 100.00 1,000.00 $0 $1 $2 $3 $4 $5 $6 $7 $8 $9 $10 DRAM NV-DIMM/PM NVMe SSD SATA SSD microsec $/GB 1 GB cost vs. Read/Write Performance 1 GB cost Read latency Write latency $9 $0.4 $1
  7. 7. High DRAM Cost and Low Flash Performance 0.10 1.00 90.00 250.00 0.10 1.00 25.00 500.00 0.10 1.00 10.00 100.00 1,000.00 $0 $1 $2 $3 $4 $5 $6 $7 $8 $9 $10 DRAM NV-DIMM/PM NVMe SSD SATA SSD microsec $/GB 1 GB cost vs. Read/Write Performance 1 GB cost Read latency Write latency $9 $0.4 $1 $2
  8. 8. Bring Redis Performance on Persistent Memories Close to Redis Performance on DRAM The Challenge
  9. 9. Before dealing with this challenge…
  10. 10. Is Persistent Memory Really Persistent ? NVMe NV-DIMM Cloud Instance
  11. 11. NVMe NV-DIMM Failed Instance Is Persistent Memory Really Persistent ?
  12. 12. Is Persistent Memory Really Persistent ? NVMe NV-DIMM Failed Instance NVMe NV-DIMM New Empty Instance
  13. 13. Data Loss Is Persistent Memory Really Persistent ? NVMe NV-DIMM Failed Instance NVMe NV-DIMM New Empty Instance
  14. 14. Alternative Solution Redundant Flash Array Cloud Instance
  15. 15. Alternative Solution New non-empty Instance Redundant Flash Array Failed Instance
  16. 16. Another Alternative • Requires ultra high-speed/low- latency network like Fiber-Channel or Infiniband • Latency is still at ~ 100microsec • Expensive, complex and not supported by cloud vendorsNew non-empty Instance Redundant Flash Array Failed Instance
  17. 17. Persistent Memory = Ephemeral Memory Persistent Storage AOF, Snapshot NVMe NV-DIMM Cloud Instance
  18. 18. Redis on Flash (RoF) • Flash used as a RAM extender (not as a persistent storage) • In RAM: dict & keys & hot values • On Flash: cold values • Values are either in RAM or on Flash • Multi-shard; multi-threaded; asynchronous • Pluggable storage engine (RocksDB and IBM’s ark) • 100% compatible with the OSS Redis
  19. 19. Flexible RAM/Flash Ratios
  20. 20. Behind the Scene
  21. 21. Increase RAM ratio  Scale-out Cluster 2500 GB2500 GB 50%50%
  22. 22. RoF is Fast Avg: 2.04M ops/sec Max: 2.14M ops/sec Avg: 0.91msec Max: 0.98 msec % below 1msec:100% Avg: 313RMB / 9.4WMB Max: 1.71RGB / 96WMB Avg: 1.45Gbps (Tx) / 0.97Gbps (Rx) Max: 1.6Gbps (Tx) / 1.2Gbps (Rx) >2M Ops/sec, > 1GB disk bandwidth, @ <1 msec latency read write read write
  23. 23. Master-Master Replication
  24. 24. Customer Perspective • Keep the speed of Redis • Maintain all Redis commands - simple and complex • Flexible configuration across regions • Any # of regions, single/multi-az/rack, any combination of nodes/shards • Fully-managed service
  25. 25. How it Should Work? Start Synced Not synced
  26. 26. How it Should Work? Write Locally Synced Not synced App App
  27. 27. How it Should Work? Distribute P2P Asynchronously Synced Not synced App App
  28. 28. Partially Converged How it Should Work? Synced Not synced App App
  29. 29. App App Fully Converged How it Should Work? Synced Not synced
  30. 30. Our Approach to Master-Master • Behaves like local Redis, performs like local Redis • Based on CRDT (Conflict-free Replication Data Type) • Supports Strong Eventual Consistency • No consensus protocol; no need for R + W > N • Any two nodes that have received the same (unordered) set of updates will be in the same state • More to come…
  31. 31. 1 New Trend
  32. 32. Foreword
  33. 33. If you are here …
  34. 34. You’re probably a geek
  35. 35. Or a geekette (the female geek)
  36. 36. • Dealing with simple problems • Writing code in shitty languages • Using silly npm packages • Not refactoring your poor old architecture because of politics You are tired of…
  37. 37. And you really want to solve real problems
  38. 38. So what are your options ?
  39. 39. Option #1 – Create your own startup
  40. 40. 50% of startups fail in the first year and 95% percent fail within five years
  41. 41. Option #2 – Create a Redis Module
  42. 42. Get respect from the community
  43. 43. And (hopefully) get many stars on GitHub
  44. 44. How do you make $$$ ?
  45. 45. • A marketplace for OSS and closed source modules • All modules are certified by Redis Labs for these platforms: Module Hub 1M+ instances 350K+ instances 100+ F500 Companies www.redismodules.com
  46. 46. Geo Clustering
  47. 47. Geo Clustering
  48. 48. Adding a Module to Module Hub Upload A few clicks
  49. 49. Adding a Module to Module Hub Upload Certification Process One monthA few clicks
  50. 50. Adding a Module to Module Hub Upload Alpha Certification Process One monthA few clicks Still not shown in the marketplace. Test by 10 selected customers
  51. 51. Adding a Module to Module Hub Upload Alpha Beta Certification Process One month At least 1 month free A few clicks Still not shown in the marketplace. Test by 10 selected customers
  52. 52. Adding a Module to Module Hub Upload Alpha Beta GACertification Process One month Still not shown in the marketplace. Test by 10 selected customers At least 1 month free A few clicks
  53. 53. So – can you make $$$ here ?
  54. 54. 3.15 2.40 21.00 8.70 24.57 10.61 0.00 5.00 10.00 15.00 20.00 25.00 30.00 Full text search Prefix search Average Latency (msec) RLEC Elasticsearch Solr 20,045 6,831 690 3,686 621 3,133 0 5,000 10,000 15,000 20,000 25,000 Full text search Prefix search Ops/sec RLEC Elasticsearch Solr 85% higher 32x higher 7.8x faster 4.1x faster redisearch The world fastest text search engine
  55. 55. For only $0.114/hr redisearch – the world fastest text search engine 3.15 2.40 21.00 8.70 24.57 10.61 0.00 5.00 10.00 15.00 20.00 25.00 30.00 Full text search Prefix search Average Latency (msec) RLEC Elasticsearch Solr 20,045 6,831 690 3,686 621 3,133 0 5,000 10,000 15,000 20,000 25,000 Full text search Prefix search Ops/sec RLEC Elasticsearch Solr 85% higher 32x higher 7.8x faster 4.1x faster
  56. 56. The Possibilities are Truly Endless Thank You @yiftachsh
  57. 57. Salvatore Sanfilippo Creator of Redis
  58. 58. Adrian Colyer Venture Partner at Accel Author of the Morning Paper Blog Former CTO, VMWare, Pivotal and Spring Source
  59. 59. As I get older, my memory just keeps getting better and better!
  60. 60. blog.acolyer.org 350 Foundations Frontiers
  61. 61. A new era of hardware (and what it might mean for Redis) A new era of data structures? Talk outline a b
  62. 62. Human computers at Dryden by NACA (NASA) - Dryden Flight Research Center Photo Collection http://www.dfrc.nasa.gov/Gallery/Photo/Places/HT ML/E49-54.html. Licensed under Public Domain via Commons - https://commons.wikimedia.org/wiki/File:Human_c omputers_- _Dryden.jpg#/media/File:Human_computers_- _Dryden.jpg
  63. 63. Computing on a human scale 63 10ns 70ns 10ms 10s 1:10s 116d Registers & L1-L3 File on desk Main memory Office filing cabinet HDDTrip to the warehouse
  64. 64. Compute HTM Persistent Memory NI FPGA GPUs Memory NVDIMMs Persistent Memory Networking 100GbE RDMA Storage NVMe Next-gen NVM Next generation hardware All change please 64
  65. 65. 2-10m Computing on a human scale 65 10s 1:10s 116d File on desk Office filing cabinet Trip to the warehouse 4x capacity fireproof local filing cabinets 23-40m Phone another office (RDMA) 3h20m Next-gen warehouse
  66. 66. The new ~numbers everyone should know 66 Latency Bandwidth Capacity/IOPS Register 0.25ns L1 cache 1ns L2 cache 3ns 8MB L3 cache 11ns 45MB DRAM 62ns 120GBs 6TB - 4 socket NVRAM’ DIMM 620ns 60GBs 24TB - 4 socket 1-sided RDMA in Data Center 1.4us 100GbE ~700K IOPS RPC in Data Center 2.4us 100GbE ~400K IOPS NVRAM’ NVMe 12us 6GBs 16TB/disk,~2M/600K NVRAM’ NVMf 90us 5GBs 16TB/disk, ~700/600K
  67. 67. NVMe Example: Redis on Flash 67 • 2M IOPS • Sub-ms latency • 1:4 DRAM:NVM split (40x SATA SSD)
  68. 68. NVMf Example: E8 Storage 68 $12M Series B led by Accel • 10M IOPS in single appliance, accessed via RDMA • 100us/50us read/write latency • ...and this is with NAND Flash!
  69. 69. Data tiering in heterogeneous memory systems DDR-DRAM NVM Capacity per CPU 100s of GBs Terabytes Read latency 1x 2x to 4x Write bandwidth 1x ⅛ x to ¼ x Estimated cost 5x 1x Endurance 1016 106 to 108 Dulloor et al. 2016 (EuroSys)
  70. 70. What will it mean? 70 this! All DRAM (baseline) All NVM tiered memory, with increasing NVM proportion Impact on cost? Impact on performance? Impact on performance/$?
  71. 71. Key-value store results - performance impact Proportion of DRAM Performance relative to all DRAM NVM only 1.15 - 1.45x 1/8 1.14 - 1.4x 1/4 1.06 - 1.13x 1/2 1.05x beware!
  72. 72. Key-value store results - cost impact Proportion of DRAM Performance/$ relative to all DRAM NVM only 3.45 - 4.5x 1/8 3.1x 1/4 2.3x 1/2 1.6x
  73. 73. And don’t forget about capacity! Proportion of DRAM Performance relative to all DRAM Performance/$ relative to all DRAM Capacity relative to all DRAM NVM only 1.15 - 1.45x 3.45 - 4.5x 4 - 10x 1/8 1.14 - 1.4x 3.1x 3.6 -8.9x 1/4 1.06 - 1.13x 2.3x 3.3 - 7.8x 1/2 1.05x 1.6x 2.5 - 5.5x
  74. 74. 74 Ultra low latency, PBs of data • The RAMCloud storage system - Ousterhout et al. 2015 • Implementing linearizability at large scale and low latency - Lee et al. 2015 1000+ Servers PB+ Memory RDMA RIFL
  75. 75. 75 RAMCloud Performance Operation End-to-end Latency median (99.9%-ile) Read 4.7 (9.2)μs Write (durable) 13.4 (148)μs Transaction 20μs 5-object transaction 27μs TPC-C throughput (10 nodes) 35K tps
  76. 76. 76 No compromises Distributed transactions with consistency, availability, and performance Dragojević et al. 2015 “This paper demonstrates that new software in modern data centers can eliminate the need to compromise. It describes the transaction, replication, and recovery protocols in FaRM, a main memory distributed computing platform. FaRM provides distributed ACID transactions with strict serializability, high availability, high throughput and low latency. These protocols were designed from first principles to leverage two hardware trends appearing in data centers: fast commodity networks with RDMA and an inexpensive approach to providing non- volatile DRAM.”
  77. 77. 77 FaRM Performance Performance TPC-C tps (90 nodes) 4.5M 99%-ile latency 1.9ms K-V Store* qps (20 nodes) 146M latency (at peak throughput) 35μs (* FaRM: Fast Remote Memory - Dragojević et al. 2014)
  78. 78. 78 Exploiting HTM... Persist. Memory RDMA HTM DrTM Fast in-memory transaction processing using RDMA & HTM - Wei et al. 2015 5.5M tps (TPC-C, 6 node cluster)
  79. 79. Redis is way more than just a ‘K-V store’... b A new era of data structures?
  80. 80. Evolution of the humble data structure 80 Collection classes & stdlibs ? single thread/core Concurrent data types multi- thread/core Distributed data types multi-server Shared concurrent data types shared-server PACELC
  81. 81. 81 Libraries, Servers, & PACELC App Instance App InstanceApp Instance CRDT Library DS Server DS Server Client Client Client Client Client CRDT Server
  82. 82. 82 CRDT Evolution Operation Based ● prepare and effect ● smaller messages ● require reliable exactly-once causal broadcast ● membership management ● no batching State Based ● state shipping and idempotent join ● larger messages ● support unreliable communication channels ● accommodate batching Delta-State Based ● local execution, ship representation of the effect & idempotently join ● smaller messages ● support unreliable communication channels ● accommodate batching
  83. 83. Data Structures! E.g. GCounter and PNCounter { replica_id value } 83 What are CRDTs made of? {“A”->3, “B”->5} HINCRBY gcounter A 1 {“A”->4, “B” -> 5} HVALS gcounter… 4+5 = 9 {“A” -> 4, “B” -> 7} {“A”->3, “B”->5} HINCRBY gcounter B 1 HINCRBY gcounter B 1 {“A”->3, “B” -> 7} {“A” -> 4, “B” -> 7} delta B -> 7 delta A -> 4 A B
  84. 84. Causal Context X Dot Store 84 Causal ∂-CRDTs version vector (Redis hash) Dot.Event (Redis set or hash) Dot = replica_id:sequence_no INCR + a delta queue for propagation... WATCH my_crdt:seq_no next = INCR my_crdt:seq_no MULTI HSET my_crdt:ccontext replica_id next SADD my_crdt:dotstore “replica_id:next:val” RPUSH my_crdt:deltaq delta EXEC
  85. 85. A new era of hardware (and what it might mean for Redis) A new era of data structures Summary a b
  86. 86. Thank You @adriancolyer
  87. 87. Salvatore Sanfilippo Creator of Redis
  88. 88. Laura Merling Former VP and GM of IoT Initiative at SAP
  89. 89. The Internet of Things Real Time Reality
  90. 90. The IoT Landscape PLATFORMS AND ENABLEMENT SOFTWARE, FULL-STACK, DEVELOPER, ANALYTICS, OPEN SOURCE
  91. 91. Big Data with an IOT View Volume Velocity Data streams and near-real time; RFID tags, sensors and smart metering. Variety Structured and unstructured. Variability Inconsistent; event triggered. Complexity Multiple sources; connect and correlate relationships, hierarchies and multiple data linkages. Volume Billions of sensors and machine to machine transactions.
  92. 92. Analytics Continuum DiagnosticDescriptive PrescriptivePredictive What Happened Why Did it Happen What Will Happen Make it Happen In Real Time
  93. 93. Geospatial The Redis IOT Opportunity PubSub DatabaseCaching Time Series
  94. 94. Challenges • Latency • Automation • Complexity Oil and Gas
  95. 95. Challenges • Transformer • Distribution Electric Vehicle Charging
  96. 96. Manufacturing Challenges • Unpredictable • Adherence • Unavailable
  97. 97. Construction Challenges • Awareness • Congestion • Slow
  98. 98. Connected Home Challenges • Latency • Interaction • Simplification
  99. 99. Retail Challenges • Latency • Interaction/Engagement • Inventory
  100. 100. The Data MINING IoT FOR THE UNFILTERED TRUTH
  101. 101. Thank You @magicmerl
  102. 102. Salvatore Sanfilippo Creator of Redis
  103. 103. Reynold Xin Chief Architect Spark at Databricks
  104. 104. Spark and Redis: In-situ analytics beyond Hadoop Reynold Xin @rxin 2016-05-11 RedisConf
  105. 105. About Databricks Founded by creators of Spark in 2013 Cloud enterprise data platform - Managed Spark clusters - Interactive data science - Production pipelines - Data governance, security, …
  106. 106. Please put up your hand if you know what Spark is?
  107. 107. Please put up your hand if you think your significant other knows what Spark is?
  108. 108. open source data processing engine built around speed, ease of use, and sophisticated analytics largest open source data project with 1000+ contributors
  109. 109. “Spark is the Taylor Swift of big data software.” - Derrick Harris, Fortune
  110. 110. Analytics: a traditional approach Hadoop is complex to operate Data consistency is difficult to maintain
  111. 111. Analytics in-situ SQL ML Streaming
  112. 112. Analytics in-situ SQL Streaming MLEnable SQL analytics over Redis Use Redis to store streaming data Use Redis to serve results generated by Spark
  113. 113. Demo by Richard Garris
  114. 114. Discount code: Meetup16SF
  115. 115. Thank You @rxin
  116. 116. Salvatore Sanfilippo Creator of Redis
  117. 117. Enjoy RedisConf Updates • Your morning break is located downstairs in the Sponsor Showcase • Three concurrent Breakout Sessions will begin at 11:15am • The Developer Workshop begins at 1:00pm in the Developer Café • Save your Redis Community Raffle ticket for today’s drawing for a chance to win a Phantom Drone
  118. 118. 11:15am-12:00pm New Redis Capabilities: Own, Make, Share 1:00pm-1:45pm SSH I/O Streaming Via Redis-based Persistent Message Queue 2:00pm-2:45pm Walmart and IBM Revisit the Linear Road Benchmark 3:15pm-4:00pm Back your App with MySQL & Redis, the Cloud Foundry Way 4:15pm-5:00pm Postgres and Redis Sitting in a Tree 122 BREAKOUTS: WEDNESDAY, MAY 11
  119. 119. Salvatore Sanfilippo Creator of Redis

×