This document summarizes Jon Hyman's presentation on using MongoDB for analytics at the NY MongoDB User Group. It discusses Appboy's use of pre-aggregated analytics documents to track time series data like app opens over time with breakdowns by dimension. It also covers Appboy's technique for quickly estimating the size of user segments by sampling random subsets of documents and extrapolating the results.

1. Appboy Analytics
Jon Hyman
NY MongoDB User Group, November 19, 2013
eBay NYC
@appboy @jon_hyman

2. A LITTLE BIT ABOUT
US & APPBOY
(who we are and what we do)
Appboy is a mobile relationship
management platform for apps
Jon Hyman
CIO :: @jon_hyman
!
Harvard
Bridgewater

3. Appboy improves
engagement by helping you
understand your app users
•
IDENTIFY - Understand demographics,
social and behavioral data
•
SEGMENT - Organize customers into
groups based on behaviors, events, user
attributes, and location
•
ENGAGE - Message users through
push notifications, emails, and multiple
forms of in-app messages

4. Use Case: Customer engagement begins with onboarding
Urban Outfitters
textPlus
Shape Magazine

5. Agenda
•
How to quickly store time series data in
MongoDB using flexible schemas
•
Learn how flexible schemas can easily
provide breakdowns across dimensions
•
Counting quickly: statistical analysis on top
of MongoDB queries

6. What kinds of analytics does Appboy track?
•
Lots of time series data
•
App opens over time
•
Events over time
•
Revenue over time
•
Marketing campaign stats and efficacy over time

7. What kinds of analytics does Appboy track?
•
Breakdowns*
•
Device types
•
Device OS versions
•
Screen resolutions
•
Revenue by product
* We also care about this over time!

8. What kinds of analytics does Appboy track?
•
User segment membership
•
How many users are in each
segment?
•
How many can be emailed or
reached via push notifications?
•
What is the average revenue
per user in the segment?
•
Per paying user?

9. Pre-aggregated Analytics:
APP OPENS OVER TIME

10. Typical time series collection
Log a new row for each open received
!
{!
timestamp: 2013-11-14 00:00:00 UTC,!
app_id: App identifier!
}!
!
db.app_opens.find({app_id: A, timestamp: {$gte: date}})!
Pro: Really, really simple. Easy to add attribution to users.
Con: You need to aggregate the data before
drawing the chart; lots of documents read into
memory, lots of dirty pages

11. Fewer documents with pre-aggregation iteration 1
Create a document that groups by the time period
!
{!
app_id: App identifier,!
date: Date of the document,!
hour: 0-23 based hour this document represents,!
opens: Number of opens this hour!
}!
!
db.app_opens.update({date: D, app_id: A, hour: 0},
{$inc: {opens:1}})
Pro: Really easy to draw histograms
Con: We never care about an hour by itself. We lose attribution.

12. Fewer documents with pre-aggregation iteration 2
Create a document by day and have each hour be a field
!
{!
app_id: App identifier,!
date: Date of the document,!
total_opens: Total number of opens this day,!
0: Number of opens at midnight,!
1: Number of opens at 1am,!
...!
23: Number of opens at 11pm!
}!
!
db.app_opens.update(!
{date: D, app_id: A}, !
{$inc: {“0”:1, total:1}}!
)
Pro: Document count is low, easy to use aggregation framework
for longer spans, fast: document should be in working set

13. Fewer documents with pre-aggregation iteration 2
•
What about looking at different dimensions?
•
App opens by device type (e.g., how do iPads
compare to iPhones?)
•
Demographics (gender, age group)

14. Solution!
FLEXIBLE SCHEMAS!

15. Fewer documents with pre-aggregation iteration 3
Dynamically add dimensions in the document
!
{!
app_id; App identifier,!
date: Date of the document,!
totals: {!
app_opens: Total number of opens this day,!
devices: {!
"iPad Air": Total number of opens on the iPad Air,!
"iPhone 4": Total number of opens on the iPhone 4,!
},!
genders: {!
male: Total number of opens from male users,!
female: Total number of opens from female users!
},!
...!
},!
0: {!
app_opens: Number of opens at midnight,!
devices: {!
"iPad Air": Number of opens on the iPad Air at midnight,!
"iPhone 4": Number of opens on the iPhone 4 at midnight,!
},!
...!
},!
...!
}!
!
db.app_opens.update({date: D, app_id: A}, {$inc: {“0”:1, total:1}})

16. Pre-aggregated analytics
Pros
•
•
Easily extensible to add other dimensions
•
Still only using one document, therefore you can create
charts very quickly
•
You get breakdowns over a time period for free
!
Cons
•
•
Pre-aggregated data has no attribution
•
Have to know questions ahead of time
Follow up: What if we wanted to look at a graph by age group?

17. Pre-aggregated analytics summary
•
Get started tracking time series
data quickly
•
You get breakdowns for free
•
Adding dimensions is super simple
•
No attribution, need to know
questions ahead of time
•
Don’t just rely on pre-aggregated
analytics

18. Counting quickly:
USER SEGMENTATION &
STATISTICAL ANALYSIS

19. User Segmentation
•A
group of users who match some set of filters

20. Counting quickly
Appboy shows you segment membership in real-time
as you add/edit/remove filters.
!
How do we do it quickly?
!
We estimate the population sizes of segments when
using our web UI.

21. Counting quickly
Goal: Quickly get the
count() of an arbitrary
query
!
Problem: MongoDB
counts are slow,
especially unindexed
ones

22. Counting quickly
10 million documents that represent people:
{!
favorite_color: “blue”,!
age: 27,!
gender: “M”,!
favorite_food: “pizza”,!
city: “NYC”,!
shoe_size: 11,!
attractiveness: 10,!
...!
} !

23. Counting quickly
10 million documents that represent people:
{!
favorite_color: “blue”,!
age: 27,!
gender: “M”,!
favorite_food: “pizza”,!
city: “NYC”,!
shoe_size: 11,!
attractiveness: 10,!
...!
} !
•
How many people like blue?
•
How many live in NYC and love pizza?
•
How many men have a shoe size less than 10?

24. Answer:
Big Question:
How do you estimate
counts?
The same way news
networks do it.
!
With confidence.

25. Counting quickly
Add a random number in a known range to each document. Say,
between 0 and 9999.
{!
random: 4583,!
favorite_color: “blue”,!
age: 27,!
gender: “M”,!
favorite_food: “pizza”,!
city: “NYC”,!
shoe_size: 11,!
attractiveness: 10,!
...!
} !
Add an index on the random number:
!
db.users.ensureIndex({random:1})

26. Counting quickly
Step 1: Get a random sample
!
I have 10 million documents. Of my 10,000 random “buckets”, I
should expect each “bucket” to hold about 1,000 users.
!
E.g.,
!
db.users.find({random: 123}).count() == ~1000!
db.users.find({random: 9043}).count() == ~1000!
db.users.find({random: 4982}).count() == ~1000

27. Counting quickly
Step 1: Get a random sample
!
Let’s take a random 100,000 users. Grab a random range that
“holds” those users. These all work:
!
db.users.find({random: {$gt: 0, $lt: 101})!
db.users.find({random: {$gt: 503, $lt: 604})!
db.users.find({random: {$gt: 8938, $lt: 9039})!
db.users.find({$or: [!
{random: {$gt: 9955}}, !
{random: {$lt: 56}}!
])
Tip: Limit $maxScan to 100,000 just to be safe

28. Counting quickly
Step 2: Learn about that random sample
!
db.users.find(!
{!
random: {$gt: 0, $lt: 101},!
gender: “M”,!
favorite_color: “blue”,!
size_size: {$gt: 10}!
}, !
)!
._addSpecial(“$maxScan”, 100000)!
.explain()
Explain Result:
!
{!
nscannedObjects: 100000,!
n: 11302,!
...!
} !

29. Counting quickly
Step 3: Do the math
!
Population: 10,000,000
!
Sample size: 100,000
!
Num matches: 11,302
!
Percentage of users who matched: 11.3%
!
Estimated total count: 1,130,000 +/- 0.2%
with 95% confidence

30. Counting quickly
Step 4: Optimize
!
Limit $maxScan to (100,000/numShards) to be even
faster
•
!
Cache the random range for a few hours
•
!
Add more RAM (or shards)
•
!
Cache results to not hit the database for the same
query
•

31. Counting quickly
Step 5: Improve
!
Get more than one count: use the aggregation
framework on top of the population’s sample size
•
•
Work around all sorts of Mongo bugs :-(

32. Summarize
•
Pre-aggregated analytics
•
Create a document that represents event occurrences
in some time period
•
Takes full advantage of MongoDB’s flexible schemas
•
Not a catch-all for analytics, you should still store event
data

33. Summarize
•
Counting quickly
•
Estimate results of arbitrary queries using population
sample sizes
•
Depending on your app, this could be a great way to
keep response time predictable as you scale

34. Thanks! Questions?
jon@appboy.com
@appboy @jon_hyman