1) MongoDB is used to collect analytics data from GitHub pages in real-time with over 10-15 million page views per day stored across 13 servers. 2) Data is stored in a denormalized manner across multiple collections to optimize for space, RAM, and read performance while live querying is supported. 3) As data volume grows over time, the data will need to be partitioned either by time frame, functionality, or individual servers to support the increased load.

1. MongoDB for Analytics
A loving conversation with @jnunemaker
MongoChicago 2012 John Nunemaker
November 12, 2012 GitHub

2. Background
How hernias can be good for you

5. 1 month
Of evenings and weekends

6. 18 months
Since public launch

7. 10-15 Million
Page views per day

8. 2.7 Billion
Page views to date

9. 13 tiny servers
2 web, 6 app, 3 db, 2 queue

10. requests/sec

11. ops/sec

12. cpu %

13. lock %

14. Implementation
How we do what we do

15. Doing It (mostly) Live
No aggregate querying

18. get('/track.gif') do
track_service.record(...)
TrackGif
end

19. class TrackService
def record(attrs)
message = MessagePack.pack(attrs)
@client.set(@queue, message)
end
end

20. class TrackProcessor
def run
loop { process }
end
def process
record @client.get(@queue)
end
def record(message)
attrs = MessagePack.unpack(message)
Hit.record(attrs)
end
end

21. http://bit.ly/rt-kestrel

22. class Hit
def record
site.atomic_update(site_updates)
Resolution.record(self)
Technology.record(self)
Location.record(self)
Referrer.record(self)
Content.record(self)
Search.record(self)
Notification.record(self)
View.record(self)
end
end

23. class Resolution
def record(hit)
query = {'_id' => "..."}
update = {'$inc' => {}}
update['$inc']["sx.#{hit.screenx}"] = 1
update['$inc']["bx.#{hit.browserx}"] = 1
update['$inc']["by.#{hit.browsery}"] = 1
collection(hit.created_on)
.update(query, update, :upsert => true)
end
end
end

24. Pros

25. Pros
Space

26. Pros
Space
RAM

27. Pros
Space
RAM
Reads

28. Pros
Space
RAM
Reads
Live

29. Cons

30. Cons
Writes

31. Cons
Writes
Constraints

32. Cons
Writes
Constraints
More Forethought

33. Cons
Writes
Constraints
More Forethought
No raw data

34. http://bit.ly/rt-counters
http://bit.ly/rt-counters2

35. Time Frame
Minute, hour, month, day, year, forever?

36. # of Variations
One document vs many

37. Single Document
Per Time Frame

39. {
"t" => 336381,
"u" => 158951,
"2011" => {
"02" => {
"18" => {
"t" => 9,
"u" => 6
}
}
}
}

40. {
'$inc' => {
't' => 1,
'u' => 1,
'2011.02.18.t' => 1,
'2011.02.18.u' => 1,
}
}

41. Single Document
For all ranges in time frame

43. {
"_id" =>"...:10",
"bx" => {
"320" => 85,
"480" => 318,
"800" => 1938,
"1024" => 5033,
"1280" => 6288,
"1440" => 2323,
"1600" => 3817,
"2000" => 137
},
"by" => {
"480" => 2205,
"600" => 7359,

44. "600" => 7359,
"768" => 4515,
"900" => 3833,
"1024" => 2026
},
"sx" => {
"320" => 191,
"480" => 179,
"800" => 195,
"1024" => 1059,
"1280" => 5861,
"1440" => 3533,
"1600" => 7675,
"2000" => 1279
}
}

45. {
'$inc' => {
'sx.1440' => 1,
'bx.1280' => 1,
'by.768' => 1,
}
}

46. Many Documents
Search terms, content, referrers...

48. [
{
"_id" => "<oid>:<hash>",
"t" => "ruby class variables",
"sid" => BSON::ObjectId('<oid>'),
"v" => 352
},
{
"_id" => "<oid>:<hash>",
"t" => "ruby unless",
"sid" => BSON::ObjectId('<oid>'),
"v" => 347
},
]

49. Writes
{'_id' => "#{sid}:#{hash}"}

50. Reads
[['sid', 1], ['v', -1]]

51. Growth
Don’t say shard, don’t say shard...

52. Partition Hot Data
Currently using collections for time frames

53. [
"content.2011.7",
"content.2011.8",
"content.2011.9",
"content.2011.10",
"content.2011.11",
"content.2011.12",
"content.2012.1",
"content.2012.2",
"content.2012.3",
"content.2012.4",
]

54. [
"resolutions.2011",
"resolutions.2012",
]

55. Move

56. Move
BigintMove

57. Move
BigintMove
MakeYouWannaMove

58. Move
BigintMove
MakeYouWannaMove
DaMove

59. Move
BigintMove
MakeYouWannaMove
DaMove
SmoothMove

60. Move
BigintMove
MakeYouWannaMove
DaMove
SmoothMove
NightMove

61. Move
BigintMove
MakeYouWannaMove
DaMove
SmoothMove
NightMove
DanceMove

62. Bigger, Faster Server
More CPU, RAM, Disk Space

63. Users
Sites
Users
Sites
Content
Content
Referrers Referrers
Terms
Engines Terms
Resolutions
Locations Engines
Resolutions
Locations

64. Partition by Function
Spread writes across a few servers

65. Users
Sites
Content
Referrers
Engines
Terms
Locations
Resolutions

66. Partition by Server
Spread writes across a ton of servers,
way down the road, not worried yet

67. Thank you!
john@github.com
@jnunemaker
MongoChicago 2012 John Nunemaker
November 12, 2012 GitHub