2. Hello World
Open source. The leading in-memory database platform,
supporting any high performance operational, analytics or
hybrid use case.
The open source home and commercial provider of Redis
Enterprise (Redise) technology, platform, products & services.
2
TalshkolnikSolution Architect at Redis Labs.
3. A Brief Overview of Redis
● Key => Data Structure server
● In memory
● Disk backed
● Scalable
● Modular
● Fast
4. Redis Top Differentiators
Simplicity ExtensibilityPerformance
NoSQL Benchmark
1
Redis Data Structures
2 3
Redis Modules
4
Lists
Hashes
Bitmaps
Strings
Bit field
Streams
Hyperloglog
Sorted Sets
Sets
Geospatial Indexes
5. Performance: The Most Powerful Database
Highest Throughput at Lowest Latency
in High Volume of Writes Scenario
Least Servers Needed to
Deliver 1 Million Writes/Sec
Benchmarks performed by Avalon Consulting Group Benchmarks published in the Google blog
5
1
Serversusedtoachieve1Mwrites/sec
6. Simplicity: Data Structures - Redis’ Building Blocks
Lists
[ A → B → C → D → E ]
Hashes
{ A: “foo”, B: “bar”, C: “baz” }
Bitmaps
0011010101100111001010
Strings
"I'm a Plain Text String!”
Bit field
{23334}{112345569}{766538}
Key
6
2
Streams
→{id1=time1.seq1(A:“xyz”, B:“cdf”),
d2=time2.seq2(D:“abc”, )}→
Hyperloglog
00110101 11001110
Sorted Sets
{ A: 0.1, B: 0.3, C: 100 }
Sets
{ A , B , C , D , E }
Geospatial Indexes
{ A: (51.5, 0.12), B: (32.1, 34.7)
}
7. • Add-ons that use a Redis API to seamlessly support
additional use cases and data structures.
• Enjoy Redis’ simplicity, super high performance, infinite
scalability and high availability.
Extensibility: Modules Extend Redis Infinitely
• Leverage existing data structures or introduce new ones.
7
3
9. • All that is cool, but sometimes you want a bit more
– New Data Structures
– Space Efficiency
– Complex Calculation
– Complex Queries
• Lua scripting not always enough
Why Modules?
10. Enter Redis Modules
● Since 4.0
● Dynamic libraries loaded to Redis
● With a C API
● Can extend Redis with:
New Capabilities
New Commands
New Data Types
11. Adapt your database to your data
ReBloom Redis-Graph
Scalable Bloom Filters for
fast and memory efficient
membership testing
Graph database on Redis
based on Cypher language
ReJSON
JSON Engine on Redis.
Pre-released
Neural Redis Redis-ML RediSearch
Full Text Search Engine in
Redis
Machine Learning Model
Serving
Simple Neural Network
Native to Redis
13. RediSearch: Redis Based Search Index
● Completely from-scratch, in C
● Open source (AGPL)
● Optimized data structures
● Text, numeric and geo filters
● Fast concurrent updates and deletes
● Scalable to 100s of servers, billions of docs
14. Working in Real Time
● Being 100% in RAM allows:
– ~O(1) insertions
– No batch vs. incremental indexing
– Concurrent operation
● Fast indexing, immediately searchable
● Fast updates / deletes
● Garbage collection and background optimization
15. Building an Index
doc1
{
title: “LCD TV”,
price: 500,
description: “foo”
}
Tokenizing /
Indexing
Term Posting List
lcd doc1, doc3, doc5...
tv doc1, doc7, doc19...
foo doc1
bar doc1, doc100, ...
doc1
{
title: “LCD TV”,
price: 500,
description: “foo”
}
doc1
{
title: “LCD TV”,
price: 500,
description: “foo”
}
Documents / Entities
Index
Price Index
…. ….
500 doc1, doc7, doc19...
503 doc3
…. ...
16. Searching The Index
Term Posting List
lcd doc1, doc3, doc5...
tv doc1, doc7, doc19...
foo doc1
bar doc1, doc100, ...
Query
Index
Price Index
…. ….
500 doc1, doc7, doc19...
503 doc3
…. ...
Text: “lcd tv”
Price: 500
Doc1
Intersect Index Entries
Result
17. Let’s Model Something
● A product database
● Products have:
– Title
– Description
– Price
– Categories
– Brand
● Sort by: Relevance, Title, Price
18. Defining Our Model
FT.CREATE products
SCHEMA
title TEXT SORTABLE
body TEXT
cats TEXT NOSTEM
brand TEXT NOSTEM
price NUMERIC SORTABLE
26. JSON and Redis
• Everybody uses JSON
• Everybody uses Redis
• Atwood's Law:
"any application that can be written in JavaScript, will eventually be
written in JavaScript"
• Which leads to:
"...any database will eventually store JSON" :)
26
27. Storing JSON values in Redis (pre v4)
Two approaches using the core Redis data structures:
• Raw JSON in String keys: the document is stored in serialized form
– Lua may be used to encode/decode on the fly
– Fast fetching
– Manipulations are slow and non atomic
• Decomposed in Hash keys: the data is deserialized to key-value pairs
– Fast, atomic updates
– Supports only flat documents with string/number values
27
28. ReJSON
• A native JSON data type for Redis
• Keys can contain any valid JSON value
• Data is stored decoded (tree, binary format)
• JSONPath-like syntax for direct access to elements
• Strongly-typed values and commands
28
30. ReJSON commands
General
JSON.DEL deletes value
JSON.GET gets value
JSON.MGET … from multiple keys
JSON.SET sets value
JSON.TYPE reports type value
Number operations
JSON.NUMINCRBY increments
number
JSON.NUMMULTBY multiplies
number
String operations
JSON.STRAPPEND appends to string
JSON.STRLEN reports string's length
30
Array operations
JSON.ARRAPPEND appends to array
JSON.ARRINDEX searches in array
JSON.ARRINSERT inserts to array
JSON.ARRLEN reports array's length
JSON.ARRPOP pops value from end
JSON.ARRTRIM trims an array
Object operations
JSON.OBJKEYS the keys in object
JSON.OBJLEN the number of keys
Other commands
JSON.RESP returns a JSON value using
Redis Serialization Protocol
31. Performance: 380 bytes, 3 nesting levels
31
Throughput
Average latency
ReJSON Raw JSON & Lua
MessagePack & Lua
32. Performance: 3468 bytes, 3 nesting levels
32
Throughput
Average latency
ReJSON Raw JSON & Lua
MessagePack & Lua
33. Main benefits
Now
• Intuitive for anybody who uses JSON
• Fully integrated in Redis, functionally- and design-wise
• Works with any Redis client, no extra coding required
• Designed for maximal element access performance
Future
Data compression, schema validation, secondary indices,
querying & much more (feature requests are welcome!)
33
35. What is a bloom filter?
● Probabilistic structure for membership testing
● Trades memory for accuracy
●No false negatives
● False positives - trade-off with space
● RAM - a fraction of a normal set
36. How Bloom Filters Work
● Bloom Filters work by ‘storing’ each item in a sparse array of bits.
● The item is mapped to n bit positions, and each position is set
● The position is determined through n hash functions.
37. When to use Bloom Filters?
● Large (possibly remote) database.
– Bloom filters can be used to check if an entry exists before querying.
– Query time in bloom filter on the order of a few µs
● Can tolerate false positives
– Sometimes (how often is configurable) a bloom filter can claim an item
really exists, but it doesn’t. The use case must be able to tolerate this.
● Datastore is append-only (for example, a blacklist)
– Items cannot be deleted from a bloom filter.
38. ReBloom: Bloom + Redis
● Redis Module by Redis Labs
● Implements Bloom Filter as Redis type
● Simple API
● Possibly the fastest BF implementation out there.
● Can scale automatically
39. Real Users
● Reddit uses ReBloom to weed out hacked passwords. A list of
300M passwords are loaded into a bloom filter, and existing/new
passwords checked against it.
● Scopely - Filtering in a real-time gaming application
40. How to use ReBloom
● Functions like a normal Redis data type (can be deleted, saved,
and replicates/persists)
● Items can be added to the filter:
– BF.ADD filterName value1
● Filter can be queried for existence of an item
– BF.EXISTS filterName value1
● Some advanced functionality
– Bulk API
– Customizing filter settings
– Debugging
