The document discusses Project Rubicon, a software analytics tool from New Relic. It summarizes Rubicon's ability to capture raw event data from applications, allowing users to ask complex questions. It then demonstrates how to write NRQL queries to analyze metrics like page views and custom events over time. NRQL makes it easy to aggregate large amounts of data through functions, time windows, time series, and facets. The document also provides an overview of Rubicon's architecture and how it handles billions of events through techniques like using memory efficiently and building for failure.

1. Software Analytics with Project Rubicon
ALEX KROMAN
OCTOBER 25TH, 2013
Wednesday, November 6, 13

2. Today’s agenda
1. Learn how to write NRQL queries
2. Aggregating billions of events
3. Building features on top of Rubicon
Wednesday, November 6, 13

3. New Relic captures performance data from
a bunch of places
Wednesday, November 6, 13

4. But performance data is a small part of
application data
Clicks
Wednesday, November 6, 13
Transactions Log messages
API calls

5. Rubicon captures the raw events from your
applications
Ruby
App
iOS app
Rubicon
Wednesday, November 6, 13
Python
App

6. By capturing raw events you can ask your
application complex questions
How many customers logged in today?
What is my average revenue per sale?
What is my most used feature?
How many people are live on the app now?
Wednesday, November 6, 13

7. An introduction to NRQL
Wednesday, November 6, 13

8. Four questions you can ask Rubicon
1. How many people used the site today?
2. How many people are using a new feature?
3. How many people are using a new feature over
time?
4. What kind of customers are using this feature?
Wednesday, November 6, 13

9. 1. How many people used the site today?
SELECT
uniqueCount(user)
FROM PageView
SINCE 1 day ago
Wednesday, November 6, 13

10. Time windows in NRQL
SINCE 8 hours ago
SINCE 2 months ago
SINCE 1 week ago UNTIL 2 hours ago
Wednesday, November 6, 13

11. 2. How many people are using a new
feature?
SELECT uniqueCount(user)
FROM PageView
WHERE name LIKE '%x_rays%'
SINCE 1 month ago
Wednesday, November 6, 13

12. 3. How many people are using the feature
over time?
SELECT uniqueCount(user) FROM PageView
WHERE name LIKE ‘%x_rays%’
SINCE 7 days ago TIMESERIES day
Wednesday, November 6, 13

13. 4. What kind of customers are using this
feature?
SELECT uniqueCount(user)
FROM PageView
WHERE name LIKE '%x_rays%'
SINCE 7 days ago FACET apm_product
Wednesday, November 6, 13

14. NRQL is designed to make it easy to
aggregate data over time
•Plenty
of aggregation functions
•Time
windows
•Time
series
•Facets
Wednesday, November 6, 13

15. How do we aggregate a billion events?
Wednesday, November 6, 13

16. Four things we’ve learned handling large
amounts of data
Build a wide cluster
Memory is cheap so use it
Build for failure
Send the cluster real traffic
Wednesday, November 6, 13

17. A brief tour of the architecture...
Queries
Load balancer
Metadata
Router
Router
Router
Worker A
Worker B
Backup
Storage
Wednesday, November 6, 13
Worker C

18. Building features on top of Rubicon
Wednesday, November 6, 13

19. Wednesday, November 6, 13

20. Wednesday, November 6, 13

21. Wednesday, November 6, 13

22. Making a histogram chart
Rubicon.nrql(“SELECT
histogram(duration) FROM
PageView SINCE 7 days ago”)
Wednesday, November 6, 13

23. JSON response
{
"results": [
{
"histogram": [
1618,
4748,
3736,
4526,
etc...
],
"bucketSize": 0.25
}
Wednesday, November 6, 13

24. Making a percentile chart
Rubicon.nrql(“SELECT
percentile(duration, 50, 95,
99) FROM PageView SINCE 7
days ago TIMESERIES hour”)
Wednesday, November 6, 13

25. JSON response
{
"timeSeries": [
{
"results": [
{
"percentiles": {
"50": 1.66,
"95": 5.445,
"99": 11.942
}
}
],
"beginTimeSeconds": 1382050194,
"endTimeSeconds": 1382053794
},
Wednesday, November 6, 13

26. Adding a custom event
data = {‘salesTeam’:
‘Enterprise’, ‘amount’: 100000}
Rubicon.insert(‘RevenueChange’,
data)
Wednesday, November 6, 13

27. Querying a custom event
SELECT count(*)
FROM RevenueChange
FACET salesTeam
SINCE 7 days ago
Wednesday, November 6, 13

28. Recap
Learned a bit of NRQL
Overview of the architecture
Explored the Rubicon API’s
Wednesday, November 6, 13

29. Come visit us in the hacker lounge!
Wednesday, November 6, 13

30. Find out more!
newrelic.com/analytics
Wednesday, November 6, 13