RedisConf18 - Introducing RediSearch Aggregations
•
2 likes•1,203 views
This document introduces RediSearch aggregations, which allow transforming search results into statistical insights using a pipeline of grouping, reducing, applying transformations, and sorting. It provides examples of counting questions by month from Stack Overflow data and comparing tag popularity over time. RediSearch aggregations can run distributed queries across shards and push operations down to shards for efficiency. Future work may include improving expression capabilities and supporting cursors.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to RedisConf18 - Introducing RediSearch Aggregations
Similar to RedisConf18 - Introducing RediSearch Aggregations (20)
More from Redis Labs
More from Redis Labs (20)
Recently uploaded
Recently uploaded (20)
RedisConf18 - Introducing RediSearch Aggregations
- 1. Introducing RediSearch Aggregations Super fast data processing pipeline!
- 2. RediSearch 101 •High Performance Search Index •Redis Module •Written From Scratch in C •Supports multiple index types •Open Source / Commercial – Commercial is scalable and distributed – Several commercial customers – Largest cluster: 200 shards, ~2B documents, 8TB RAM
- 3. What Can RediSearch Do? •Full-Text, Numeric, Geo and Tag indexes •Real-time indexing and updates •Distributed search on Billions of documents •Complex query language •Auto-Complete •Highlighting •And….
- 5. Aggregations In Practice • Transform plain search results into statistical insights •Using a dynamic pipeline of: – Grouping and Reducing – Applying Transformations – Sorting GROUP Result Stream REDUCE REDUCE APPLY SORT
- 6. Aggregate vs Search Search Request •Top N relevant results •Processing is minimal if any •Focus on filtering/ranking “People named Homer Simpson” Aggregate Request • Reduce and transform results • Complex processing pipeline • Used for analytics and insights “Average age of people by surname”
- 7. • Let’s take all Stack Overflow questions history • Count the number of questions asked by month • We have – Timestamp – Tags – Question Text – Etc A Little Taste
- 8. Count Questions By Month FT.AGGREGATE idx “*” APPLY month(@time) AS month GROUPBY 1 @month REDUCE COUNT 0 AS count SORTBY 1 @month
- 10. Building The Aggregation Pipeline
- 11. Step 1: Filter Filter A regular RediSearch Query
- 12. Step 2: Group Group by one or more fields Filter Group
- 13. Step 2: Group Group by one or more fields Filter Group Reduce Reduce
- 14. Step 3: Apply projections Functions, expressions, whatever Filter Group Reduce Apply
- 15. Step 4: Sort By one or more fields Or output of previous steps Filter Apply Sort Group Reduce
- 16. Add More Steps Re-group, apply more, re-sort, page, etc. Filter Apply Sort ? Group Reduce
- 17. Aggregate Filters • Determine what results we process • Can be used to create complex selections @country:{Israel | Italy} @age:[25 52] @profession:”1337 h4x0r” -@languages:{perl | php}
- 18. GROUPBY and Reduce • Grouping by any number of fields / upstream properties • A GROUPBY is associated with many reducers • Reducers flatten the a group’s stream into one result Group Results REDUCE Single Result GROUP Group Results REDUCE Single Result Result Stream Result Stream
- 19. Available Reducers • count • cont_distinct(ish) • min/max • avg • sum • stddev • quantile • first_value • tolist • random_sample Result Stream REDUCER Single Result
- 20. APPLY Projections • APPLY applies 1:1 transformations on results • Evaluate a complex expression per result • Operate on previous output APPLY Single ResultSingle Result
- 21. APPLY Complex Expressions APPLY “sqrt((@a + @b)/2)” AS avg APPLY “timefmt(day(@time))” AS date APPLY “format(‘Hello, %s’, @userName)” AS msg
- 22. Numeric Functions and Expressions • Numeric operators: + - / * % ^ • log(x), log2(x) • floor(x), ciel(x) • abs(x) • sqrt(x) • exp(x)
- 23. String Functions and Expressions • lower(s) • upper(s) • substr(s, offset, len) • format(fmt, …) • matched_terms([max])
- 24. Time Utility Functions • timefmt(unix_timestamp, [fmt]) → string • parsetime(str, [fmt]) → timestamp • minute(timestamp) → timestamp • hour(timestamp) → timestamp • day(timestamp) → timestamp • year(timestamp) → integer • dayofmonth(timestamp) → integer • dayofweek(timestamp) → integer • monthofyear(timestamp) → integer
- 25. SORTBY •Sort by one or more properties •Each can be ASC/DESC •MAX limits to top-N results
- 26. LIMIT vs. SORTBY MAX • SORTBY … MAX {n} • LIMIT {offset} {num} • Getting results 10-20 will work best as: SORTBY … MAX 20 LIMIT 10 10
- 28. What If Our Data Is Split Across Nodes?
- 29. Classic Search Approach: Coordinator Index Shard Top K Top K Top K Search: “Hello”
- 30. That Doesn’t Translate to Aggregation Pipelines Coordinator Index Shard count_distinctx(x) count_distinctx(x) count_distinctx(x) count_distinct(x)
- 31. That Doesn’t Translate to Aggregation Pipelines Coordinator Index Shard count_distinctx(x) count_distinctx(x) count_distinctx(x) count_distinct(x)
- 32. Distributing Aggregations: Naive Approach Filter: “redis” Group: by date Reduce: count Sort: by count Original Pipeline
- 33. Distributing Aggregations: Naive Approach Filter: “redis” Group: by date Reduce: count Sort: by count Coordinator Pipeline Shard Pipeline Distribute To Shards Merge
- 34. Problem: Every Record Must Be Transferred COUNT_DISTINCT(x) Index Shard GET ALL (x) GET ALL (x) GET ALL (x) count_distinct(x)
- 35. Push Down: 1 Record Per Group Transferred Filter: “redis” Group: by date Reduce: COUNT Sort: by sum(counts) Coordinator Pipeline Shard Pipeline Group: by date Reduce: SUM(COUNTS) Distribute To Shards Merge
- 36. Example: Simple Counting Client COUNT() SUM(COUNT) COUNT
- 37. Example: MIN/MAX Client MAX(x) MAX(MAX(x)) MAX(x)
- 38. Example: Average Client AVG(x) SUM(SUM(x)) → A SUM(COUNT) → B A/B → AVG(x) SUM(x) COUNT
- 39. Example: Count Distinct Elements Client COUNT_DISTINCT(x) HLL_SUM(HLL(x)) HLL(x)
- 40. Example: Median Client MEDIAN(x) MEDIAN(SAMPLE(x)) RESERVOIR SAMPLE(x)
- 41. FT.AGGREGATE idx “*” APPLY month(@time) AS month GROUPBY 1 @month REDUCE COUNT 0 AS count SORTBY 1 @month The Query Plan Translator FT.AGGREGATE idx “*” APPLY month(@time) AS month GROUPBY 1 @month REDUCE COUNT 0 AS count FT.AGGREGATE {DISTRIBUTE(...)} GROUPBY 1 @month REDUCE SUM 1 @count AS count SORTBY 1 @month
- 42. • Two or more GROUPBYs - not possible / easy • High number of groups - still slow • Exact count distinct and quantiles are slow • If you really want to parallelize huge workloads - use Spark et al Limitations
- 44. The Dataset • All Stack Overflow Questions • The Schema: – Title / body – Tags – Answers – Asking User Id – Rating – Timestamp
- 45. Count Questions By Month FT.AGGREGATE idx “*” APPLY month(@time) AS month GROUPBY 1 @month REDUCE COUNT 0 AS count SORTBY 1 @month
- 46. Compare Tag Popularity Over Time FT.AGGREGATE idx “@tags:{java|python|c}” APPLY month(@time) AS month APPLY matched_terms() AS tag GROUPBY 2 @month @tag REDUCE COUNT 0 AS count SORTBY 2 @month @tag
- 47. Top Tags By Approx. Number of Askers FT.AGGREGATE idx “*” APPLY split(@tags) AS tag GROUPBY 1 @tag REDUCE COUNT_DISTINCTISH 1 @user AS users SORTBY 2 @users DESC MAX 10
- 48. Histogram Of Scores FT.AGGREGATE idx “*” APPLY floor(@score/10)*10 AS score GROUPBY 1 @score REDUCE COUNT 0 SORTBY 1 @score MAX 100
- 49. • Port all search functionality to the FT.AGGREGATE API • Streaming time-window based aggregations • Improve expression language – Add post filters – Expressions as groupby / sortby properties • Support cursored operation in the coordinator Now open to requests! Future Challenges
- 50. Thank You