Redis Labs is a company that provides Redis Enterprise, a fully managed Redis database. Redis is an in-memory database but lacks search capabilities. Redis Labs developed RediSearch, a Redis module that allows for fast searching of billions of documents stored in Redis. RediSearch uses inverted indexes and custom data structures to enable full-text search, numeric filtering, and other search features in Redis in a memory-efficient way. It can also scale to handle large indexes by partitioning documents across multiple Redis servers and coordinating queries.

1. Home of RedisSearching billions of documents with Redis
(Dvir Volk)|(Itamar Haber)

2. Redis Labs – Home of Redis
• Founded in 2011
• HQ in Mountain View CA, R&D center in Tel-Aviv IL
The commercial company behind Open Source Redis
Provider of the Redis Enterprise (Redise
) technology,
platform and products

3. Redise
Cloud Private
Redis Labs Products
Redise
Cloud Redise
Pack ManagedRedise
Pack
SERVICES SOFTWARE
Fully managed Redise
service in
VPCs within AWS, MS Azure, GCP
& IBM Softlayer
Fully managed Redise
service on
hosted servers within AWS, MS
Azure, GCP, IBM Softlayer,
Heroku, CF & OpenShift
Downloadable Redise
software
for any enterprise datacenter or
cloud environment
Fully managed Redise
Pack in
private data centers
&& &

4. So… Searching in Redis?
We all know and love Redis as a data store, but...
● GET user1? no problemo!
● HGETALL user1? sure thing!
● GET users WHERE name contains “jeff”? NOPE!
Redis has no built in
concept of indexing
or search, duh!
4

5. Search 101
The core concept: The Index
● Take a document
● Break it apart
● Map terms/properties ⇒ docs in a Posting List
Searching is getting documents that are linked to
the terms
5

6. Building an Index
doc1
{
title: “LCD TV”,
price: 500,
description: “foo”
}
Indexing
Process
Term Posting List
lcd doc1, doc3, doc5...
tv doc1, doc7, doc19...
foo doc1
bar doc1, doc100, ...
doc1
{
title: “LCD TV”,
price: 500,
description: “foo”
}
doc1
{
title: “LCD TV”,
price: 500,
description: “foo”
}
Documents
Index
Price Index
…. ….
500 doc1, doc7, doc19...
503 doc3
…. ...
6

7. So… Searching With Redis?
Redis is just Key=>Value, right?
But can we use its advanced data types?
Key
"I'm a Plain Text String!"
{ A: “foo”, B: “bar”, C: “baz” }
Strings/Blobs/Bitmaps
Hash Tables (objects!)
Linked Lists
Sets
Sorted Sets
Geo Sets
HyperLogLog
{ A , B , C , D , E }
[ A → B → C → D → E ]
{ A: 0.1, B: 0.3, C: 100, D: 1337 }
{ A: (51.5, 0.12), B: (32.1, 34.7) }
00110101 11001110 10101010
Looks like a
set will do!
7

8. Searching With Redis - Vanilla Style
● Two most suitable structures: Sorted Sets & Sets
● Remember, the core of searching is:
term ⇒ [doc_id, doc_id, ….]
● So how about this: redis> SADD hello doc1
redis> SADD world doc1
redis> SINTER hello world
8

9. Searching With Redis - Vanilla Style
Even better with Sorted Sets - we can have scores!
> ZADD hello 1.0 doc1 0.5 doc2
> ZADD world 1.0 doc 0.5 doc2
> ZINTERSTORE hw 2 hello world
> ZRANGE hw 0 -1
9

10. Vanilla Style Only Goes So Far...
● LOTS of memory - ZSETs take about 60B
overhead per record!
● Intersections and unions can get slow
● No exact phrase matching
● No proximity boosting
● etc, etc...
10

11. Enter Redis Modules
● Since 4.0
● Dynamic libraries loaded to Redis
● With a C API
● Can extend Redis with:
New Capabilities
New Commands
New Data Types
11

12. Search With Modules
● So now Everything Is PossibleTM
!
● We implement our own structures & algorithms!
● With less memory!
● And less CPU time!
● And more capabilities!
● And use existing libraries!
12

13. Why Write From Scratch?
● Most widely used search libraries are either:
○ In Java
○ Very complex and bloated
○ Written with disk in mind
Plus, where is the fun in that?
13

14. Introducing RediSearch
● Completely from-scratch, in C
● Optimized data structures
● Stores documents as Hashes
● Extremely fast indexing and search
● Text, numeric and geo filters
● Fast non blocking updates and deletes
● Scalable distributed mode up to billions of docs
14

15. RediSearch In Action
> FT.CREATE products SCHEMA title TEXT price NUMERIC
> FT.ADD products prod1 1.0 FIELDS title "Toshiba 32 LCD TV" price 250
> FT.ADD products prod2 1.0 FIELDS title "Samsung 42 LCD TV" price 350
> FT.SEARCH products "lcd tv"
1) (integer) 2
2) "prod1"
…
4) "prod2"
…
> FT.SEARCH products "lcd tv" FILTER price 300 400
1) (integer) 1
"prod2"
15

16. RediSearch Data Structures
Key
[prod1, prod2, …]
{title:“Toshiba 32 LCD TV”, …}
Document Table
Posting Lists (Inverted Index)
Documents (HASHes)
AutoComplete (Trie)
Numeric Index (Range tree)
Geo Sets
{200..400: [prod1, prod2, …]… }
Toshib*
{ A: (51.5, 0.12), B: (32.1, 34.7) }
{1: prod1, 2: prod2, …}
16

17. Processing Search Queries

18. RediSearch Search Request
FT.SEARCH {index} {query} [NOCONTENT] [VERBATIM]
[NOSTOPWORDS] [WITHSCORES] [WITHPAYLOADS]
[FILTER {numeric_field} {min} {max}] ...
[GEOFILTER {geo_field} {lon} {lat} {raius} m|km|mi|ft]
[INKEYS {num} {key} ... ]
[INFIELDS {num {field} ... ]
[SLOP {slop}] [INORDER]
[LANGUAGE {language}]
[EXPANDER {expander}]
[SCORER {scorer}]
[PAYLOAD {payload}]
[SORTBY {field} [ASC|DESC]]
[LIMIT offset num]
18

19. Search Request Flow
Request/Query
Parsing
Query
Expansion
Query
Execution
Document
Ranking
Top-n + Paging
Document
Loading
Response
Request
19

20. Evaluating Queries
● Document at-a-time approach
● Indexes are decoded as iterators, one object per
round
● Top N results copied into a
heap & sorted on the fly INTERSECT
UNION Obama
Barack Barrack
Date: 2010-2016
Scoring function
20

21. Scoring Results
● Each document in the query's result is evaluated by a
Scoring Function
● Top-N scored documents are selected
● The default uses:
○ Term Frequency–Inverse Document Frequency (TF-IDF)
○ User given a priori document score
○ Proximity boosting
● And you can write your own function!
21

22. Query Language

23. Simple Text Search
Hello world
equivalent to:
(Hello world)
23

24. Exact Phrase Search
"Hello world"
24

25. Or Search (Union)
Hello|hola world|mundo
(hello world)|(hola mundo)
25

26. Specific Field Matching
@title:Hello world
26

27. Multi Field Matching
@title|body:Hello world @pic:jpg
modifiers are implicitly "AND"ed
27

28. Numeric Filtering
@title:lcd tv @price:[100 300]
@title:lcd tv @price:[(100 +inf]
28

29. Optional Clauses
@title:tv ~@description:lcd
optional matches just increase score
29

30. Negative Clauses
@title:tv -plasma
30

31. Prefix Queries
Hello wo*
31

32. Complex Query Example
@title:(lcd tv)
~@tags|description:”42 inch”
@price:[100 500]
-@type:4K|Plasma
32

33. (a demo would be nice)

34. Extending RediSearch
● Has its own module system
○ C/C++ API
○ Minimal, simple API
● Supports custom query
expanders (e.g. synonyms)
● Supports custom ranking
functions
34

35. Auto-Complete
● Custom data type - trie based
● Fuzzy completions
● Manual dictionary building and
scoring
● Unicode aware case-folding
35

36. Scaling RediSearch

37. Scaling RediSearch
● What do we do when the index is too big for one
machine?
● We need to split the index on multiple machines
● But it’s not that simple!
37

38. Scaling RediSearch
● Partition by document id
● Each partition contains a full index of 1/N of the
documents
● All terms for a sub-index are stored on the same
shard
● This means we need a query coordinator:
○ Distribute the queries to all shards
○ Merge the results
○ Top-N of Top-Ns
38

39. RediSearch Shard
Redis Search Engine Coordinator
+ + =
RediSearch Shard

40. RediSearch Cluster
Node 1
Node 2
Node 3

41. RediSearch Cluster
Node 1
Node 2
Node 3
ClientFT.SEARCH idx “foo”

42. RediSearch Cluster Fully-Connected Fan-Out/In
Node 1 Node 3
“foo” {3}
“foo” {4}
“foo” {5}
“foo” {6}
“foo” {2}
“foo” {1}
ClientFT.SEARCH idx “foo”
Node 2

43. When RediSearch Cluster Becomes Larger
Node 1
Node 2
Node 3

44. RediSearch Cluster Tree-Based Fan-Out/In
Node 1
Node 2
Node 3
“foo” {5},{6},{7},{8} “foo” {9},{10},{11},{12}
ClientFT.SEARCH idx “foo”

45. RediSearch Cluster Tree-Based Fan-Out/In
Node 1
Node 2
Node 3
“foo” {5},{6},{7},{8} “foo” {9},{10},{11},{12}
ClientFT.SEARCH idx “foo”
“foo” {5}
“foo” {7}
“foo” {6}
“foo” {8}
“foo” {1}
“foo” {3}
“foo” {2}
“foo” {4}
“foo” {9}
“foo” {11}
“foo” {10}
“foo” {12}

46. Fastest Search in Town
46

47. http://redisearch.io - Thank You!
Dvir Volk
Senior Architect
Redis Labs
@dvirsky
Itamar Haber
Chief OSS Education Officer
Redis Labs
@itamarhaber