Recommended
More Related Content
What's hot
What's hot (20)
Similar to Redis Streams
Similar to Redis Streams (20)
More from Redis Labs
More from Redis Labs (20)
Recently uploaded
Recently uploaded (20)
Redis Streams
- 1. Redis Streams Overview SIMON PRICKETT
- 2. What is Redis? 2 ● Redis – (REmote DIctionary Server) ● Open source. ● The leading in-memory database platform ● Created in 2009 by Salvatore Sanfilippo (a.k.a @antirez) ● Source: https://github.com/antirez/redis ● Website: http://redis.io What’s Redis? 2 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise technology, platform, products & services. Redis – (REmote DIctionary Server)
- 3. Data Structures - Redis’ Building Blocks Lists [ A → B → C → D → E ] Hashes { A: “foo”, B: “bar”, C: “baz” } Bitmaps 0011010101100111001010 Strings "I'm a Plain Text String!” Bit field {23334}{112345569}{766538} Key Streams {id1=time1.seq1(A:“xyz”, B:“cdf”), d2=time2.seq2(D:“abc”, )} Hyperloglog 00110101 11001110 Sorted Sets { A: 0.1, B: 0.3, C: 100 } Sets { A , B , C , D , E } Geospatial Indexes { A: (51.5, 0.12), B: (32.1, 34.7) }
- 4. Redis Streams: High Level Overview Consumer Producer 1532556560197-0 location: "99" temp: "86.2" humidity: "45.9" 1532556560195-1 location: "97" temp: "72.3" humidity: "21.8" 1532556560195-0 location: "32" temp: "91.4" humidity: "75.8" Key:ingest:temphumidity
- 5. Redis Streams: Connect Many Producers and Consumers Producer 2 Producer m Producer 1 Producer 3 Consumer 1 Consumer n Consumer 2 Consumer 3 Streams enable asynchronous data exchange between producers and consumers and historical range queries
- 6. Producers and Messages Producer 1532556560197-0 location: "99" temp: "86.2" humidity: "45.9" Key:ingest:temphumidity XADD ingest:temphumidity *location 99 temp 86.2 humidity 45.9
- 8. Consumer: Range Queries Consumer 1532556560197-0 location: "99" temp: "86.2" humidity: "45.9" 1532556560195-1 location: "97" temp: "72.3" humidity: "21.8" 1532556560195-0 location: "32" temp: "91.4" humidity: "75.8" Key:ingest:temphumidity 1532556560199-0 location: "19" temp: "56.8" humidity: "14.3" XRANGE ingest:temphumidity 15325565601971532556560199
- 9. Consumer: Read Queries 1532556560197-0 location: "99" temp: "86.2" humidity: "45.9" 1532556560195-1 location: "97" temp: "72.3" humidity: "21.8" 1532556560195-0 location: "32" temp: "91.4" humidity: "75.8" Key:ingest:temphumidity 1532556560199-0 location: "19" temp: "56.8" humidity: "14.3" XREAD COUNT2 STREAMS ingest:temphumidity 0 XREAD COUNT2 STREAMS ingest:temphumidity 1532556560195-1
- 10. Classifier 1 Classifier 2 Classifier n Consumer Group XREADGROUP XREAD Consumers Producer 2 Producer m Producer 1 Producer 3 XADD XACK Deep Learning-based Classification Analytics Data Backup Messaging Consumer Groups
- 11. Stream Management Producer XADD ingest:temphumidity MAXLEN 25 * location 99 temp 86.2... XX XTRIM ingest:temphumidity MAXLEN 25 XTRIM ingest:temphumidity MAXLEN ~ 25 XLEN ingest:temphumidity EXPIRE ingest:temphumidity 86400
- 12. redis.io • Streams documentation university.redislabs.com • RU202 Redis Streams • Free online course • Earn a Certificate of Completion YouTube: Streams overview with Salvatore • youtube.com/watch?v=qXEyuUxQXZM Would You Like to Know More?
Editor's Notes
- * Introduce yourself
- Created in 2009 by Salvatore Sanfillipo. Everything in this talk applies equally to open source Redis and Redis Enterprise When we say "in memory", all reads will come from memory Data can be persisted to disk
- Show of hands, who has any experience with Redis? Who has experience using Redis as cache? Who has experience using Redis as a data structure store / non cache applications? Everything is about keys Streams is a newer data type, added in Redis 5.0 about a year ago
- Are a Redis data structure, built into the product Store and share event data Can also provide communication between processes A stream is an append-only list of entries called messages. The order of messages is immutable. Messages can be read from the stream by ID or by ranges. Consuming from the stream doesn't remove the message from it, it's still there for other consumers
- Streams decouple multiple producers from multiple consumers Useful when there is fast data production and slower consumption – as Redis can ingest very quickly Also useful for implementing architectural patterns such as fan out – one input to multiple consumers A good way to implement communication between microservices too
- Producers add messages to a stream using XADD command There is no explicit stream creation step, first XADD to the key creates it and starts a stream Explain timestamp format Explain that you can supply own ID but next message ID must always be greater than the last you added Message bodies are a series of name/value pairs There is no schema, message bodies can vary between messages in the same stream
- Python example using a redis client library Libraries available for pretty much all programming languages Most libraries map each Redis command to a function call in the target language e.g. redis.xadd here In a real case, we'd poll the temperature sensor periodically or whatever application logic needs going
- As message IDs are numeric, streams naturally form a sort of time series Range queries can be made with the XRANGE command, by supplying a start and end timestamp Messages returned in oldest to newest order For newest to oldest order, use XREVRANGE command
- XREAD allows us to read from multiple streams at once Designed to allow consumer to work through a stream by fetching messages whose ID is greater than the last highest ID the consumer read XREAD can block waiting for new messages to appear in the stream This is how we can implement a blocking pub/sub type consumer
- For expensive processing tasks or greater throughput we may want a set of consumers to be able to work together to process the stream, splitting workload between them Consumer groups allow consumers to do this by reading using the XREADGROUP command, similar syntax to XREAD Each consumer in the group would see only a subset of the messages, but the group as a whole processes the entire stream between them The Redis server keeps track of which consumer in a group has seen each message Once a consumer in a group has processed a message, it uses XACK to tell Redis it was processed But what if a consumer takes a message and fails before processing it? As the consumer group shares the consumption of the stream amongst its members, no other consumer in the group would see this message Redis provides commands that allow for the detection of failed consumers in a group If a consumer fails after reading a message but before acknowledging it was successfully processed, actions can be taken to reassign the message elsewhere in the group Note that independently other consumers can use XREAD or XRANGE as before to consume the same stream
- So far it seems that streams can grow forever Obviously, the Redis server has a finite amount of memory to store the data in Streams can be trimmed to a specified length Either when adding Or by another process using XTRIM command Length can be an exact number, or an approximation (Radix trie implementation) Redis will compress repeating a sequence of message bodies that are the same Because streams are just a Redis key you can also set expiry – useful for time partitioning?
- Documentation is online, organized by command and topic Redis University is free online training. Currently 5 different courses, all apply to open source Redis Can earn certificates of completion Redis Streams course covers streams exclusively, uses a small amount of Python coding Salvatore's half hour video on streams gives a longer introduction with examples in Ruby