Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Beyond Caching: Extending Redis Enterprise for Real-Time Streams Processing

72 views

Published on

SpringOne Platform 2019
Session Title: Beyond Caching: Extending Redis Enterprise for Real-Time Streams Processing
Speaker: Sheryl Sage, Director of Partner Marketing, Redis Labs
Youtube: https://youtu.be/ldEqSZFm1kY

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Beyond Caching: Extending Redis Enterprise for Real-Time Streams Processing

  1. 1. Beyond Caching: Extending Redis to Real-time Streams Processing October 7–10, 2019 Sheryl Sage, Director Marketing Redis Labs Beyond Caching: Extending Redis to Real-time Streams Processing October 7–10, 2019 Sheryl Sage, Director Marketing Redis Labs
  2. 2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ • What is a data stream? • What are the typical challenges you face while managing data streams? • What is Redis Stream? • How Redis Stream addresses the challenges? • How to use Redis Streams? 2
  3. 3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Who We Are Open source. The leading in-memory NoSQL database, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise technology, platform, products & services.
  4. 4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Redis Top Differentiators
  5. 5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Key "I'm a Plain Text String!" Strings Hyperlog-logs Cache Counting Data Structures - Redis’ Building Blocks [ A → B → C → D → E ] Lists Queues {23334}{112345569}{766538}{665455} Bit field Efficient Integers { A: “foo”, B: “bar”, C: “baz” } Hashes Sessions/profiles { A , B , C , D , E } Sets Recommendation Engine { A: 0.1, B: 0.3, C: 100, D: 1337 } Sorted Sets Leader Board 00110101011001110010101010 Bitmaps Bitmap encoding { A: (51.5, 0.12), B: (32.1, 34.7) } Geospatial Indexes Location Services 00110101 11001110 10101010 {id1=time1.seq1(A:“xyz”), id2=time2.seq2(D:“abc”, )} Streams
  6. 6. strings, ttl lists, sets, pub/sub streams search ML + Graph Relationships Graph RAM + Flash + Disk Client Location / Proximity Geo Intelligent Session Management Hash, Sets API Rate Limiting Hash User SessionsHashes, Sets Web App Architecture 101 Pub/Sub Messaging Pub/Sub JSON Storage JSON medium.com
  7. 7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Redis Enterprise for Production Deployments
  8. 8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Node Architecture
  9. 9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Delivered in Any Form
  10. 10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is a Data Stream?
  11. 11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simplest form of a Data Stream ConsumerProducer
  12. 12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ But Often You Have M Producers and N Consumers Producer 2 Producer m Producer 1 Producer 3 Consumer 1 Consumer n Consumer 2 Consumer 3
  13. 13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ It’s a bit more complex than this
  14. 14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data Stream Components
  15. 15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Good News
  16. 16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Redis Supports All the Functions of a Data Stream
  17. 17. Understanding Redis Streams
  18. 18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Let’s look at the challenges first
  19. 19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How to Support a Variety of Consumers Analytics Data Backup Consumers Producer Messaging Real-time Real-time or Periodic Lookup Periodic Read
  20. 20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Catch Up Problem Image ProcessorProducer Consumption Rate: 100/secArrival Rate: 500/sec Redis Stream Fast Slow Backlog
  21. 21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How to Address The Catch Up Problem? Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor Redis Stream Scale Out New Problem: Mutual Exclusivity
  22. 22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data Recovery from Consumer Failures Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor Redis Stream Scale Out Yet Another Problem: Failure Scenarios
  23. 23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Connection Failures Producer Image Processor Image Processor Image Processor Image Processor Image Processor Redis Stream The solution must be resilient to failures
  24. 24. Here comes Redis Streams
  25. 25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is Redis Streams? Pub/Sub Lists Sorted Sets It is like Pub/Sub, but with persistence It is like Lists, but decouples producers and consumers It is like Sorted Sets, but asynchronous + • Lifecycle management of streaming data • Built-in support for timeseries data • A rich choice of options to the consumers to read streaming and static data • Super fast lookback queries powered by radix trees • Automatic eviction of data based on the upper limit
  26. 26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1. It enables asynchronous data exchange between producers and consumers MessagingProducer Consumer
  27. 27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Analytics Data Backup Consumers Producer Messaging 2. You can consume data in real-time as it arrives or lookup historical data
  28. 28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor Redis Stream 3. With consumer groups, you can scale out and avoid backlogs
  29. 29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Classifier 1 Classifier 2 Classifier n Consumer Group XREADGROUP XREAD Consumers Producer 2 Producer m Producer 1 Producer 3 XADD XACK Deep Learning-based Classification Analytics Data Backup Messaging 4. Simplify data collection, processing and distribution to support complex scenarios
  30. 30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Quick Reference Guide https://redislabs.com/docs/getting-started-redis-streams/ Redis Streams Commands
  31. 31. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Commands: XADD, XREAD Asynchronous producer-consumer message transfer MessagingProducer Consumer XADD XREAD XADD mystream * name Anna XADD mystream * name Bert XADD mystream * name Cathy XREAD COUNT 100 STREAMS mystream 0 XREAD BLOCK 10000 STREAMS mystream $
  32. 32. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Commands: XRANGE, XREVRANGE Lookup historical data Analytics Data Backup Consumers Producer Messaging XADD XREAD XRANGE XREVRANGE XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10 XRANGE mystream - + COUNT 10
  33. 33. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Scaling out using consumer groups Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Redis Stream
  34. 34. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Scaling out using consumer groups Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Redis Stream Consumer Group Consumer 1 Consumer 2 Consumer 3 Consumer n Unconsumed List Consumers XADD XGROUP XREADGROUP XACK XPENDING XCLAIM
  35. 35. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Command: XGROUP CREATE Create a consumer group Unconsumed List mygroup Alice Bob edcba mystream edcba App A App B XGROUP CREATE XGROUP CREATE mystream mygroup $ MKSTREAM
  36. 36. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Command: XREADGROUP Read the data Unconsumed List mygroup Alice Bob App A App B a ed cb XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream >
  37. 37. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Command: XACK Consumers acknowledge that they consumed the data Unconsumed List mygroup Alice Bob App A App B a cb XACK XACK mystream mygroup 1526569411111-0 1526569411112-0
  38. 38. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Command: XREADGROUP Repeat the cycle Unconsumed List mygroup Alice Bob App A App B a cb XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >
  39. 39. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How to claim the data from a consumer that failed while processing the data? Unconsumed List mygroup Alice Bob App A App Bcb Unconsumed List mygroup Alice Bob App A App B cb XCLAIM
  40. 40. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Commands: XPENDING, XCLAIM Claim pending data from other consumers Unconsumed List mygroup Alice Bob App A App B cb XPENDING mystream mygroup - + 10 Bob XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0
  41. 41. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Java Jedis https://github.com/xetorthio/jedis Lettuce https://lettuce.io/ Python redis-py https://github.com/andymccurdy/redis-py JavaScript ioredis https://github.com/luin/ioredis node_redis https://github.com/NodeRedis/node_redis .NET StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis Go redigo https://github.com/gomodule/redigo go-redis https://github.com/go-redis/redis C Hiredis https://github.com/redis/hiredis PHP predis https://github.com/nrk/predis phpredis https://github.com/phpredis/phpredis Ruby redis-rb https://github.com/redis/redis-rb Redis Clients that Support Redis Streams 41
  42. 42. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Sample Use Case: Social Media Analytics https://github.com/redislabsdemo/IngestRedisStreams
  43. 43. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Redis Streams Twitter Ingest Stream Influencer Classifier https://github.com/redislabsdemo/IngestRedisStreams
  44. 44. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 44
  45. 45. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Questions ? ? ? ? ? ? ? ? ? ? ?

×