Practical examples written in PHP using nested aggregations to quickly segment and provide data for charting among millions of records within a web request.

1. THE POWER OF ELASTIC
AGGREGATIONS
DAN FEY

2. ABOUT DAN FEY
▸ Grew up in northern NJ
▸ Back-end engineer at Crowdskout in DC
▸ Work on API and data layers in PHP using Laravel
▸ Work with MySQL, Mongo, and Elasticsearch

3. CROWDSKOUT
▸ Data collection, analytics, and outreach platform
▸ Collects, normalizes, and matches data for customers
▸ Tools for segmenting audiences, building dynamic
charting, and acting on segments

4. IN THIS PRESENTATION
▸ Brief introduction to Elasticsearch
▸ Practical examples of aggregations
▸ Crowdskout’s journey to provide dynamic real-time
charting among millions of records within a web request

5. CHARTS SCREENSHOT

6. ELASTICSEARCH
Elasticsearch is a distributed, RESTful search and
analytics engine capable of solving a growing number
of use cases.

7. ELASTICSEARCH CONCEPTS
• Cluster - Collection of one or more nodes
• Node - One server - stores/indexes data, serves queries
• Index - Collection of documents with mappings
• Type - Category of documents within an index
• Document - JSON unit of information
• Shards & Replicas - Subdivision of an index and copies

8. ELASTIC AGGREGATIONS
▸ Somewhat similar to SQL Group By
▸ Can perform operations on large datasets:
▸ Terms - unique terms with counts
▸ Range - counts within a number range
▸ Date Histogram - counts by a given time interval
▸ Sum/Average/Statistics - performed on the given
dataset

9. ELASTIC AGGREGATIONS
▸ Enabled by stored columnar document values
▸ Can be nested within each other*

10. INDEX MAPPING DOCUMENT

11. MY INTRODUCTION TO ELASTICSEARCH
▸ Started at Crowdskout over a year and a half ago
▸ Used search queries to create segments of profiles
▸ Used terms aggregations to get unique string options for
profile fields
▸ Wanted to provide charting capabilities for our data points

12. SEARCH QUERY EXAMPLE
SELECT COUNT(*) FROM genders WHERE value = 'female'

13. TERMS AGGREGATION EXAMPLE
SELECT value, COUNT(*) FROM genders GROUP BY value

14. FIRST - SIMPLE CHARTING
▸ Provide simple charting data while supporting segment
querying
▸ This meant combining our segment search queries with
our options terms aggregations

16. COMBINING SEARCH AND SEGMENT QUERIES
Count of profiles by gender for people with an
undergraduate education
SELECT genders.value, COUNT(*)
FROM genders
JOIN educations USING (profile_id)
WHERE educations.level = "undergraduate"
GROUP BY genders.value

18. NEXT - DATE HISTOGRAM SUPPORT
▸ We needed to replace the terms aggregation with a date
histogram aggregation
▸ We also needed to add a filter aggregation to the date to
limit the time period

20. DATE HISTOGRAM QUERY
▸ Count page views by month
SELECT when, MONTH(when) AS month, COUNT(*)
FROM pageviews
WHERE when >= "2017-06-01"
GROUP BY month

22. MANAGING COMPLEX REQUESTS AND RESPONSES
▸ Built a library: https://github.com/crowdskout/es-search-
builder
▸ Simpler, less verbose queries
▸ Assists in building aggregations and parsing results

23. ES-SEARCH-BUILDER QUERY

24. ES-SEARCH-BUILDER AGG

25. USE ES-SEARCH-BUILDER WITH ELASTICSEARCH-PHP

26. NEXT - COMPARING TWO COLLECTIONS
▸ What if we want to view Education within Gender
▸ This required nesting two terms aggregations

28. EXAMPLE REQUEST

29. EXAMPLE RESPONSE

31. THERE’S NO NESTING LIMIT IN AGGREGATIONS

32. OTHER AGGREGATIONS
▸ Geo Distance, Geo Bounds
▸ Histogram, Range, Date Range
▸ Min, Max, Percentile
▸ Scripted

33. THE VALUE OF ELASTIC AGGREGATIONS
▸ Performant on large datasets for a wide variety of dynamic
charting
▸ Charts can be requested frequently, making them real-
time and always up to date
▸ Customers can build their own charts through a simple UI
and get immediate results
▸ After charts are built, customers can apply filters using
dates or segment queries

34. SOME DIFFICULTIES
▸ Keeping Elasticsearch up to date with the databases of record
▸ Complex nested aggregations tricky to get right
▸ Some aggregations are less performant
▸ High entropy fields (lots of unique values)
▸ Very large amount of documents (i.e. billions)
▸ Not sure the query limits, i.e. requests per hour, number of
concurrent requests

35. QUESTIONS

36. CROWDSKOUT CHARTS DEMO

37. CROWDSKOUT QUERY LANGUAGE
▸ Simple SQL-like language for Elasticsearch queries
▸ (Gender = "Male") AND (EducationLevel = "Graduate")

38. CROWDSKOUT TRAIT QUERIES FOR CHARTING
▸ Trait queries use Crowdskout Query Language criteria

39. TRAITS DEMO