Software Analytics with Project Rubicon

ALEX KROMAN
OCTOBER 25TH, 2013

Wednesday, November 6, 13
Today’s agenda
1. Learn how to write NRQL queries
2. Aggregating billions of events
3. Building features on top of Rubicon...
New Relic captures performance data from
a bunch of places

Wednesday, November 6, 13
But performance data is a small part of
application data

Clicks

Wednesday, November 6, 13

Transactions Log messages

AP...
Rubicon captures the raw events from your
applications
Ruby
App

iOS app

Rubicon

Wednesday, November 6, 13

Python
App
By capturing raw events you can ask your
application complex questions
How many customers logged in today?
What is my aver...
An introduction to NRQL

Wednesday, November 6, 13
Four questions you can ask Rubicon
1. How many people used the site today?
2. How many people are using a new feature?
3. ...
1. How many people used the site today?
SELECT
uniqueCount(user)
FROM PageView
SINCE 1 day ago

Wednesday, November 6, 13
Time windows in NRQL
SINCE 8 hours ago
SINCE 2 months ago
SINCE 1 week ago UNTIL 2 hours ago

Wednesday, November 6, 13
2. How many people are using a new
feature?
SELECT uniqueCount(user)
FROM PageView
WHERE name LIKE '%x_rays%'
SINCE 1 mont...
3. How many people are using the feature
over time?
SELECT uniqueCount(user) FROM PageView
WHERE name LIKE ‘%x_rays%’
SINC...
4. What kind of customers are using this
feature?
SELECT uniqueCount(user)
FROM PageView
WHERE name LIKE '%x_rays%'
SINCE ...
NRQL is designed to make it easy to
aggregate data over time
•Plenty

of aggregation functions

•Time

windows

•Time

ser...
How do we aggregate a billion events?

Wednesday, November 6, 13
Four things we’ve learned handling large
amounts of data
Build a wide cluster
Memory is cheap so use it
Build for failure
...
A brief tour of the architecture...
Queries

Load balancer

Metadata

Router
Router
Router

Worker A

Worker B

Backup
Sto...
Building features on top of Rubicon

Wednesday, November 6, 13
Wednesday, November 6, 13
Wednesday, November 6, 13
Wednesday, November 6, 13
Making a histogram chart
Rubicon.nrql(“SELECT
histogram(duration) FROM
PageView SINCE 7 days ago”)

Wednesday, November 6,...
JSON response
{
"results": [
{
"histogram": [
1618,
4748,
3736,
4526,
etc...
],
"bucketSize": 0.25
}
Wednesday, November 6...
Making a percentile chart
Rubicon.nrql(“SELECT
percentile(duration, 50, 95,
99) FROM PageView SINCE 7
days ago TIMESERIES ...
JSON response
{
"timeSeries": [
{
"results": [
{
"percentiles": {
"50": 1.66,
"95": 5.445,
"99": 11.942
}
}
],
"beginTimeS...
Adding a custom event
data = {‘salesTeam’:
‘Enterprise’, ‘amount’: 100000}
Rubicon.insert(‘RevenueChange’,
data)

Wednesda...
Querying a custom event
SELECT count(*)
FROM RevenueChange
FACET salesTeam
SINCE 7 days ago

Wednesday, November 6, 13
Recap
Learned a bit of NRQL
Overview of the architecture
Explored the Rubicon API’s

Wednesday, November 6, 13
Come visit us in the hacker lounge!

Wednesday, November 6, 13
Find out more!
newrelic.com/analytics

Wednesday, November 6, 13
Upcoming SlideShare
Loading in …5
×

FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engineering Manager at New Relic

889 views
669 views

Published on

This week we've released two new features: Histograms and Percentiles. Come learn about the analytics product that powers both of these features and how you'll be able to use Project Rubicon to get answers to complex questions about your application.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
889
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engineering Manager at New Relic

  1. 1. Software Analytics with Project Rubicon ALEX KROMAN OCTOBER 25TH, 2013 Wednesday, November 6, 13
  2. 2. Today’s agenda 1. Learn how to write NRQL queries 2. Aggregating billions of events 3. Building features on top of Rubicon Wednesday, November 6, 13
  3. 3. New Relic captures performance data from a bunch of places Wednesday, November 6, 13
  4. 4. But performance data is a small part of application data Clicks Wednesday, November 6, 13 Transactions Log messages API calls
  5. 5. Rubicon captures the raw events from your applications Ruby App iOS app Rubicon Wednesday, November 6, 13 Python App
  6. 6. By capturing raw events you can ask your application complex questions How many customers logged in today? What is my average revenue per sale? What is my most used feature? How many people are live on the app now? Wednesday, November 6, 13
  7. 7. An introduction to NRQL Wednesday, November 6, 13
  8. 8. Four questions you can ask Rubicon 1. How many people used the site today? 2. How many people are using a new feature? 3. How many people are using a new feature over time? 4. What kind of customers are using this feature? Wednesday, November 6, 13
  9. 9. 1. How many people used the site today? SELECT uniqueCount(user) FROM PageView SINCE 1 day ago Wednesday, November 6, 13
  10. 10. Time windows in NRQL SINCE 8 hours ago SINCE 2 months ago SINCE 1 week ago UNTIL 2 hours ago Wednesday, November 6, 13
  11. 11. 2. How many people are using a new feature? SELECT uniqueCount(user) FROM PageView WHERE name LIKE '%x_rays%' SINCE 1 month ago Wednesday, November 6, 13
  12. 12. 3. How many people are using the feature over time? SELECT uniqueCount(user) FROM PageView WHERE name LIKE ‘%x_rays%’ SINCE 7 days ago TIMESERIES day Wednesday, November 6, 13
  13. 13. 4. What kind of customers are using this feature? SELECT uniqueCount(user) FROM PageView WHERE name LIKE '%x_rays%' SINCE 7 days ago FACET apm_product Wednesday, November 6, 13
  14. 14. NRQL is designed to make it easy to aggregate data over time •Plenty of aggregation functions •Time windows •Time series •Facets Wednesday, November 6, 13
  15. 15. How do we aggregate a billion events? Wednesday, November 6, 13
  16. 16. Four things we’ve learned handling large amounts of data Build a wide cluster Memory is cheap so use it Build for failure Send the cluster real traffic Wednesday, November 6, 13
  17. 17. A brief tour of the architecture... Queries Load balancer Metadata Router Router Router Worker A Worker B Backup Storage Wednesday, November 6, 13 Worker C
  18. 18. Building features on top of Rubicon Wednesday, November 6, 13
  19. 19. Wednesday, November 6, 13
  20. 20. Wednesday, November 6, 13
  21. 21. Wednesday, November 6, 13
  22. 22. Making a histogram chart Rubicon.nrql(“SELECT histogram(duration) FROM PageView SINCE 7 days ago”) Wednesday, November 6, 13
  23. 23. JSON response { "results": [ { "histogram": [ 1618, 4748, 3736, 4526, etc... ], "bucketSize": 0.25 } Wednesday, November 6, 13
  24. 24. Making a percentile chart Rubicon.nrql(“SELECT percentile(duration, 50, 95, 99) FROM PageView SINCE 7 days ago TIMESERIES hour”) Wednesday, November 6, 13
  25. 25. JSON response { "timeSeries": [ { "results": [ { "percentiles": { "50": 1.66, "95": 5.445, "99": 11.942 } } ], "beginTimeSeconds": 1382050194, "endTimeSeconds": 1382053794 }, Wednesday, November 6, 13
  26. 26. Adding a custom event data = {‘salesTeam’: ‘Enterprise’, ‘amount’: 100000} Rubicon.insert(‘RevenueChange’, data) Wednesday, November 6, 13
  27. 27. Querying a custom event SELECT count(*) FROM RevenueChange FACET salesTeam SINCE 7 days ago Wednesday, November 6, 13
  28. 28. Recap Learned a bit of NRQL Overview of the architecture Explored the Rubicon API’s Wednesday, November 6, 13
  29. 29. Come visit us in the hacker lounge! Wednesday, November 6, 13
  30. 30. Find out more! newrelic.com/analytics Wednesday, November 6, 13

×