KPIs for APIs (and how API Calls are the new Web Hits, and you may be measuring all wrong)

36,688 views

Published on

How do you measure API success? What KPIs do APIs need? What mistakes should I avoid? Find out what you should, and shouldn't, be measuring as part of your API program in this Business of APIs Conference NYC talk. Dive into a breadth of API metrics, the 6 keys to better API metrics, and the traps to beware of (the important do's and don'ts). Also real-world API case studies show who measures what.

Published in: Internet
3 Comments
99 Likes
Statistics
Notes
No Downloads
Views
Total views
36,688
On SlideShare
0
From Embeds
0
Number of Embeds
19,272
Actions
Shares
0
Downloads
555
Comments
3
Likes
99
Embeds 0
No embeds

No notes for slide

KPIs for APIs (and how API Calls are the new Web Hits, and you may be measuring all wrong)

  1. 1. KPIs for APIs or, why API Calls are the new Web Hits and you may be measuring all wrong John Musser / @johnmusser / API Science / apiscience.com Business of APIs Conference, NYC, 2014
  2. 2. Why do we need a Web site? 1995 Of course we have a Web site Of course we have an API Why do we need an API? 2000 2005 2010
  3. 3. Why do we need a Web site? 1995 Of course we have a Web site You are here Of course we have an API Why do we need an API? 2000 2005 2010
  4. 4. Remember hit counters?
  5. 5. Evolution of web site metrics Hits Page Views Visitors Conversions
  6. 6. Evolution of API metrics? Calls ? ? ?
  7. 7. Great APIs get measured
  8. 8. But measure what?
  9. 9. Well, it depends…
  10. 10. Why do you have an API?
  11. 11. Marketing channel BizDev / LeadGen New line of business Extend product Drive innovation User acquisition Upsell opportunity Device and mobile support Content acquisition API as Product Distribution channel Increase stickiness Accelerate internal projects Partner opportunities Drive traffic Increase footprint
  12. 12. Who is your API for?
  13. 13. API consumer continuum You Your partners & customers Everyone else Internal Private Open
  14. 14. The answers to who & why should drive your KPIs
  15. 15. The answers to who & why should drive your KPIs
  16. 16. Many ways to think about API KPIs… Traffic Total calls Top methods Call chains Quota faults Developers Total developers Active developers Top developers Trending apps Retention Service Performance Availability Error rates Code defects Marketing Dev registrations Dev portal funnel Traffic sources Event metrics Support Support tickets Response times Business Direct revenue Indirect revenue Market share Costs
  17. 17. More ways to think about API KPIs… Customer Satisfaction NPS Churn Community Forum activity Social media (both for API and apps built on it) Quality Defects Error prone APIs Repair rate API Maturity Stability Maturity metric Change rate Innovation Number of products Number of apps Speed to market Channel Call volume: Mobile, web, others Revenue by channel
  18. 18. There are a lot of API KPIs you could measure
  19. 19. Don’t drown in API KPIs
  20. 20. KPI SECRET #1 Great APIs prioritize what to measure
  21. 21. Choose your “KPI lens” CFO lens CMO lens CEO lens Ops lens QA lens CRO lens PM lens PR lens
  22. 22. CFO lens: financial metrics Total direct and indirect revenue Margin ROI Marginal cost Capex / Opex investment CLTV per developer
  23. 23. DevOps lens: operational metrics Performance Availability Error rates Faults by API type Defect backlog API traffic trends
  24. 24. CMO lens: marketing metrics Developer acquisition cost Developer demographics Portal visitor count & traffic sources SEO / SEM metrics Social media metrics Evangelism and event metrics
  25. 25. KPI SECRET #2 Know your Developer Funnel
  26. 26. KPI SECRET #2 Know your Developer Funnel (for public APIs)
  27. 27. Web sites have funnels Acquisi9on Ac9va9on Reten9on Revenue Referral How do users find you? Does the user have a good first experience? Do users return? How do you make money? Do users tell others? Dave McClure’s “Startup Metrics for Pirates”
  28. 28. APIs have funnels too Acquisi9on Ac9va9on Reten9on Revenue Referral How do developers find you? Does the developer have a good first experience? Do developers return? How do you and developers make money? Do developers tell others?
  29. 29. Potential KPIs Acquisi9on Ac9va9on Reten9on Revenue Referral Dev portal site: sources, traffic vol, uniques # dev registraCons, API keys issued # acCve devs, API call volume by dev, # apps $ per dev, $ per app, # end users # dev referrals, dev/app growth, NPS
  30. 30. KPI SECRET #3 Measure your API ecosystem
  31. 31. App Users Apps Developers Your APIs You API 1 API 2 API 3
  32. 32. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total app users Revenue per user User growth rate User churn
  33. 33. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total apps Trending apps Revenue per app Channel: mobile/web
  34. 34. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total developers Active developers Revenue per developer Top developers Stuck developers
  35. 35. App Users Apps Developers Your APIs You Total API calls Top API calls API 1 Top API packages Internal usage API 2 API 3
  36. 36. App Users Apps Developers Your APIs You API 1 API 2 API 3 Total revenue Partnerships Market share Innovation
  37. 37. KPI SECRET #4 Most APIs obey the Power Law
  38. 38. The Power Law of APIs Developers API call volume
  39. 39. The Power Law of APIs Most of your API traffic, revenue, success, etc…. Developers API call volume
  40. 40. Think quality, not just quantity Use your API KPIs, to find who, how, where, when. Developers API call volume
  41. 41. KPI SECRET #5 API KPIs need buy-in
  42. 42. Don’t let API KPIs live in a vacuum Who are your stakeholders? Who’s your sponsor? Get executive buy-in on your top 3-5 KPIs
  43. 43. KPI SECRET #6 Measuring internal API usage often just as important as external
  44. 44. Guardian: 70% internal calls API usage from internal apps External usage
  45. 45. Evernote: 99% internal calls API usage from internal apps External usage
  46. 46. Netflix: 99.9% internal calls API usage from devices External usage
  47. 47. Netflix: 99.9% internal calls Why Enable Netflix experience Who Netflix device and UI teams
  48. 48. KPI WARNING #1 Beware of API vanity metrics
  49. 49. API Billionaire Club 13 billion API calls / day (May 2011) 5 billion API calls / day (April 2010) 5 billion API calls / day (October 2009) 1.1 billion API calls / day (April 2011) 1 billion API calls / day (May 2012) 1 billion API calls / day (Q1 2012) 1 billion API calls / day (January 2012)
  50. 50. 5 Billion API calls/day
  51. 51. “What if, all things being equal, instead of 30 billion API calls a month, we served 5 billion?” Daniel Jacobson, Netflix
  52. 52. KPI WARNING #2 Beware of API KPI traps
  53. 53. 3 common API KPI traps Not having a metrics process (have a metrics plan & a feedback loop) Using the wrong metrics (don’t be afraid to change what you track) Unrealistic expectations (“Absolutely, we’ll have X developers on-board by X date”)
  54. 54. One important metric you may have missed...
  55. 55. What’s your TTFHW? Time To First “Hello World” aka: how long from zero to 60?
  56. 56. What’s your TTFHW? Developer experience (DX) can make or break your API.
  57. 57. What’s your TTFHW? Too subjective to measure?
  58. 58. What’s your TTFHW? Don’t forget about qualitative metrics
  59. 59. What’s your TTFHW? User testing. Surveys. Forums. Social media…
  60. 60. Real world API KPIs
  61. 61. KPI: Conversions “We are seeing a trend where a person who uses another app with Evernote is 50% more likely to move on to our premium (paid) service”
  62. 62. KPI: Engagement “Apps that have an API integration with Evernote are seeing much more usage: Pocket readers normally read 15% of what they save, but Evernote users read 80% of what they save in Pocket.”
  63. 63. KPI: Revenue API now accounts for 20-25% of the charity’s annual revenue
  64. 64. KPI: Photo Edits
  65. 65. KPI: Apps 3,500 Apps Aviary, March, 2013
  66. 66. Goal: “more eyeballs on our content” API calls (requests) Response volume (ex: # stories) Impressions (via image beacon) Loyalty More KPIs KPI: Impressions
  67. 67. KPI: Recipe delivery “We sell Vodka. But people buy drinks.” Goal: “ensure top quality drinks consistently delivered to people regardless of where people encounter Absolut”
  68. 68. KPI: Recipe delivery APIs let you be where your customers are. Measure how. “We sell Vodka. But people buy drinks.” Goal: “ensure top quality drinks consistently delivered to people regardless of where people encounter where people encounter Absolut”
  69. 69. Evolution of API metrics? Calls ? ? ?
  70. 70. Who & why drives KPIs Have a KPI lens & process Prioritize & repeat
  71. 71. Thank You john@apiscience.com @johnmusser
  72. 72. Photo Credits Race car: hCp://www.flickr.com/photos/lim_lik_wei/3270522646/ Stopwatch: hCp://www.flickr.com/photos/purplemaMish/3020016417/ Stop sign: hCps://www.flickr.com/photos/9toperez/5791194358 Power law: hCp://en.wikipedia.org/wiki/Power_law

×