Why measure anything?We want to continually improve, to further the              MoFo mission.Humans, and also organizatio...
Absent of any data, everything looks fine
First Requirement of ImprovementRealize there is something that needs            improvement       __________________
Knowing the truth is often scary and uncomfortable,       but at least you know when to run!
Fast, Confident, and AwesomeIf an org, team, or individual:-Knows, definitively, how they are doing now-Knows, definitivel...
Then that org, team, or individual:-will be much faster to determine success vs. failure, making it possible toresolve fas...
Had Good Metrics Indicating Dogs Were Tame...                  Confident.
What kind of stats are we talking?  Not just CPU and App Performance!       metrics << app / product      metrics << proje...
Proposed Layers of Metrics1. Organization Layer - "How is Mozilla Foundation doing   at its mission?"2. Team Layer - "How ...
Organizational Metrics ExamplesHow many visits to all of our sites this year vs. last year?How many people have donated to...
Team Metrics ExamplesHow many hours were worked/PTOd on theteam?How many projects is that team involved in thisweek?How ma...
Application Metrics ExamplesWhat was the average response time for popcorn.webmaker.org this week?How many videos were edi...
Operations Metrics ExamplesHow much disk is currently being used?How much memory and CPU utilization do we have inproducti...
What tools will we be using?Primary Tools● Newrelic● StatsD/Graphite● Google Analytics● GeckoboardSecondary Tools (Not cov...
New Relic (Server Monitoring)
New RelicReal time data on your server and appperformanceExamples:● what code is running slowly or erroring?● how long are...
Google Analytics
Google AnalyticsNumbers on who is using your app, how andwhen they are using it● what content is popular with who?● which ...
statsd/graphiteMultiple stats, one stream
statsd/graphitestatsd - "Collects data by increment counters and timers,but does not show, sort, or average(etc....)"graph...
StatsD examples (counting new vids in popcorn)(note the data structure, env.app.what)Time something (PHP example)   $start...
Geckoboard
GeckoboardA friendly view into all of our high level stats.● this is where Mark/Ryan will come in● this is where we highli...
Metrics Flow
Things to Beware Of● Presenting data out of context for audience● Instrumentation of anything to New Relic,  Google Analyt...
Go fourth and metric!See what you can do with these client libs:● statsd:   ○ https://django-statsd.readthedocs.org/en/lat...
Links/ReferencesGood read with a further intro for statsd/graphiteHow we will setup Opsview to watch StatsD/GraphiteGoogle...
Mozilla Foundation Metrics - presentation to engineers
Mozilla Foundation Metrics - presentation to engineers
Upcoming SlideShare
Loading in …5
×

Mozilla Foundation Metrics - presentation to engineers

1,652 views
1,583 views

Published on

@rossbruniges and I talked with our fellow Mozilla Foundation engineers and development teams about getting the data for building a data driven operation using statsd, graphite, geckoboard, google analytics, and newrelic.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,652
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mozilla Foundation Metrics - presentation to engineers

  1. 1. Why measure anything?We want to continually improve, to further the MoFo mission.Humans, and also organizations, cant know if they can or should do something differentlyunless they get feedback or data to prompt the question: "How can we/I do better at this?"
  2. 2. Absent of any data, everything looks fine
  3. 3. First Requirement of ImprovementRealize there is something that needs improvement __________________
  4. 4. Knowing the truth is often scary and uncomfortable, but at least you know when to run!
  5. 5. Fast, Confident, and AwesomeIf an org, team, or individual:-Knows, definitively, how they are doing now-Knows, definitively, how they are doing after a change-Knows, definitively, what variables changed when
  6. 6. Then that org, team, or individual:-will be much faster to determine success vs. failure, making it possible toresolve faster.-will inevitably grow and learn, further and faster-generates more trust...with users, contributors, org, team and maybe most importantly...-Well be more confident to try more things, knowing we can quickly detect andresolve.-Thus, as we will try more things, it will lead more often to success.
  7. 7. Had Good Metrics Indicating Dogs Were Tame... Confident.
  8. 8. What kind of stats are we talking? Not just CPU and App Performance! metrics << app / product metrics << projects / teams metrics << users / customers metrics << contributors metrics << operations metrics << performance metrics << developers metrics << support
  9. 9. Proposed Layers of Metrics1. Organization Layer - "How is Mozilla Foundation doing at its mission?"2. Team Layer - "How well is this team working together and accomplishing goals? "3. Application/Service Layer - "How are things with X application or service?"4. Operations Layer - "How are we doing, logistically?"
  10. 10. Organizational Metrics ExamplesHow many visits to all of our sites this year vs. last year?How many people have donated to MoFo, and how much?How many times were we or any of our sites mentioned onsocial media and the news?How many new "webmakers" did we bring into ourecosystem across all applications and teams?
  11. 11. Team Metrics ExamplesHow many hours were worked/PTOd on theteam?How many projects is that team involved in thisweek?How many collaborator / volunteer / staff linesof code / features were added?How many milestones did the team meet thismonth?
  12. 12. Application Metrics ExamplesWhat was the average response time for popcorn.webmaker.org this week?How many videos were edited this week?How many new videos were created this week?What was the # of page visits for popcorn on average byhour?What was the ratio of errors to successes for popcorns keytransactions this week?
  13. 13. Operations Metrics ExamplesHow much disk is currently being used?How much memory and CPU utilization do we have inproduction?How many gigabytes of logs were generated this week?What was the slowest loading page per application thisweek?How much did our infrastructure cost?
  14. 14. What tools will we be using?Primary Tools● Newrelic● StatsD/Graphite● Google Analytics● GeckoboardSecondary Tools (Not covering today)● AWS, Pingdom, Opsview
  15. 15. New Relic (Server Monitoring)
  16. 16. New RelicReal time data on your server and appperformanceExamples:● what code is running slowly or erroring?● how long are end users waiting before they can interact with our content?● How many database writes per minute?● End user performance monitoring● Which deploy caused a problem?
  17. 17. Google Analytics
  18. 18. Google AnalyticsNumbers on who is using your app, how andwhen they are using it● what content is popular with who?● which parts of the site are causing people to leave?● where are our users coming from?● what things have they done while on our sites?● Can track campaigns, goals (ie, made a popcorn video), conversion rates
  19. 19. statsd/graphiteMultiple stats, one stream
  20. 20. statsd/graphitestatsd - "Collects data by increment counters and timers,but does not show, sort, or average(etc....)"graphite - Shows the data in a presentable way, allowingyou to line up multiple metrics and perform calculations,such as sort, avg, std deviation(Examples of a few things we might count/time)● someone made a project, remixed a project● someone has been issued a badge● someone has decided to delete their profile
  21. 21. StatsD examples (counting new vids in popcorn)(note the data structure, env.app.what)Time something (PHP example) $start = microtime(true); postvideo(); StatsD::timing("production.butter.postvideo.response.200", (microtime (true) - $start) * 1000);Count something (PHP) StatsD::increment("production.butter.postvideo.response.200");
  22. 22. Geckoboard
  23. 23. GeckoboardA friendly view into all of our high level stats.● this is where Mark/Ryan will come in● this is where we highlight the individual metrics we get judges against● one stop-shop over all MoFo/webmaker properties● encouraged to make public
  24. 24. Metrics Flow
  25. 25. Things to Beware Of● Presenting data out of context for audience● Instrumentation of anything to New Relic, Google Analytics, Geckoboard should go over ssl● Do not use New Relic auto instrumentation to capture any params in the database● Gameable Metrics
  26. 26. Go fourth and metric!See what you can do with these client libs:● statsd: ○ https://django-statsd.readthedocs.org/en/latest/ ○ https://github.com/sivy/node-statsd● geckoboard: ○ http://pypi.python.org/pypi/django-geckoboard ○ https://github.com/danjenkins/node-geckoboard- push ○ https://github.com/Hebo/node-geckoboard● newrelic custom instrumentation: ○ https://newrelic.com/docs/instrumentation/custom- metric-collection
  27. 27. Links/ReferencesGood read with a further intro for statsd/graphiteHow we will setup Opsview to watch StatsD/GraphiteGoogle Analytics: Setting up goals and tracking conversionsEasy setup instructions for statd/graphite server(if you want to test locally)

×