Cross-platform logging and analytics

1,510 views
1,420 views

Published on

This is the logging and analytics talk I gave at CocoaCoders.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,510
On SlideShare
0
From Embeds
0
Number of Embeds
416
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Cross-platform logging and analytics

    1. 1. Logging & Analytics For fun and profit
    2. 2. Let me tell you about a bug
    3. 3. Let me tell you about a bug• ~2% of our customers were affected
    4. 4. Let me tell you about a bug• ~2% of our customers were affected• Resolved (WORKSFORME)
    5. 5. Let me tell you about a bug• ~2% of our customers were affected• Resolved (WORKSFORME)• People will complain, but they won’t send me crash reports
    6. 6. Let me tell you about a bug• ~2% of our customers were affected• Resolved (WORKSFORME)• People will complain, but they won’t send me crash reports• Apple’s crash reporter shows no crashes
    7. 7. I wrote my own crash reporter
    8. 8. c
    9. 9. The confidence you get from finding out about every crash, anywhere in the world, is crucial to delivering a high quality product that needs to be used in the wild. For those of us in the consumer software business its absolutely critical.Youcant rely on your customers to tell you about crashes—manyof them may not be technical enough, and most of them wont bother to take time off of their own important work to give you a useful crash report unless you make it completely automatic. - Joel Spolsky
    10. 10. It’s 2012, don’t roll your own Use quincykit.net
    11. 11. And everybody lived happily ever after...
    12. 12. Until...
    13. 13. NSError *err = nil;[somethingWithErr:&err];if (err) { UIAlertView *errorAlertView = [[UIAlertView alloc] initWithTitle:@”Error” message:[err description] delegate:nil cancelButtonTitle:OK otherButtonTitles:nil]; [errorAlertView show]; [errorAlertView release];}
    14. 14. NSError *err = nil;[somethingWithErr:&err];if (err) { UIAlertView *errorAlertView = [[UIAlertView alloc] initWithTitle:@”Error” message:[err description] delegate:nil cancelButtonTitle:OK otherButtonTitles:nil]; [errorAlertView show]; [errorAlertView release];}
    15. 15. Or do I want one of these?
    16. 16. So I wrote an error logger
    17. 17. And everybody lived happily ever after...
    18. 18. Until...
    19. 19. I started maintaining non-iOS code http://github.com/drewcrawford/buildbot
    20. 20. I have too much noise
    21. 21. But not enough signal
    22. 22. But too much noise
    23. 23. But not enough signal
    24. 24. Within the same msg
    25. 25. Meanwhile...
    26. 26. How do I make money?1. Customer hears about me2. ????3. Profit!
    27. 27. How do I make money?1. Customer hears about me2. ???? What’s going on here?3. Profit!
    28. 28. Web Analytics
    29. 29. Web Analytics• How many different times does a person visit my site before they convert?
    30. 30. Web Analytics• How many different times does a person visit my site before they convert?• How do my blog readers’ habits change over time?
    31. 31. Web Analytics• How many different times does a person visit my site before they convert?• How do my blog readers’ habits change over time?• What indicators (referrers, time-on-site, months of loyalty) correlate to high quality users?
    32. 32. Web Analytics• How many different times does a person visit my site before they convert?• How do my blog readers’ habits change over time?• What indicators (referrers, time-on-site, months of loyalty) correlate to high quality users?• Can I prioritize sales leads automatically by their behavior on my site?
    33. 33. App Analytics
    34. 34. App Analytics• Do websites, SEM ads, or product videos matter?
    35. 35. App Analytics• Do websites, SEM ads, or product videos matter?• What causes people to make an IAP?
    36. 36. App Analytics• Do websites, SEM ads, or product videos matter?• What causes people to make an IAP?• Should the button be here or there?
    37. 37. App Analytics• Do websites, SEM ads, or product videos matter?• What causes people to make an IAP?• Should the button be here or there?• Is anyone really going to use this feature?
    38. 38. App Analytics• Do websites, SEM ads, or product videos matter?• What causes people to make an IAP?• Should the button be here or there?• Is anyone really going to use this feature?• How many of my app users still use after 3 days?
    39. 39. Nothing on this pageanswers any questions
    40. 40. Anatomy of a useful page
    41. 41. Anatomy of a useful page HOW???
    42. 42. About Flurry
    43. 43. [FlurryAnalyticsstartSession:@"1R1NQX1SPPWD7SDPI9WZ"]; for(int i = 0; i < 5; i++) { [FlurryAnalytics logEvent:@"In for loop"withParameters:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:i] forKey:@"i"]]; }
    44. 44. And then I waited several hours...
    45. 45. And all I got was this lousy summary
    46. 46. The world’s most worthless pie chart
    47. 47. In fairness, you canexport a few hundred records into Excel
    48. 48. But if you want full logs You’d better ask nicely
    49. 49. If only we had a machine that could process large quantities of data...
    50. 50. These are viewsNot models. They draw pretty pictures. They’re not your data.
    51. 51. It walks like raw data export It talks like raw data export But it’s not.Look for yourself: http://mixpanel.com/docs/api- documentation/data-export-api#libs-python
    52. 52. We dont provide raw log exports at thistime. It is something we are consideringadding to our pipeline, so thanks for writingin with the feature request. - Mixpanel
    53. 53. I’m not saying there’s a conspiracy tokeep you from getting real event logsBut don’t believe you can export your data until you actually download & check the damn logs
    54. 54. Meanwhile...
    55. 55. The epic unit test• I’m working on a complicated “big-data” project• The unit tests parse 100ks of points and take hours• When your compile-build-run-test loop is measured in hours, you debug from an execution trace, not by stepping through code• One day the 200MB+ execution traces wouldn’t upload to our bug tracker any more
    56. 56. The epic unit test• I was building custom tools to parse through the huge log files and give me the info I needed to debug after-the-fact• There’s a complicated system of macros controlling log levels of different components
    57. 57. The epic unit test• Even with all these tools, rarely would I log the right things.• Logging the wrong things costs me hours to re- run the test.• Over and over again.
    58. 58. Bion has a similar problem on a different project
    59. 59. Meanwhile...
    60. 60. I read this articlehttp://journal.paul.querna.org/articles/2011/12/26/log-for- machines-in-json/
    61. 61. What if instead of this[Mon Dec 26 09:14:46 2011] [info] [client 50.57.61.4] File doesnot exist: /var/www/no-such-file
    62. 62. We log this{    "timestamp": 1324830675.076,    "status": "404",    "short_message": "File does not exist: /var/www/no-such-file",    "host": "ord1.product.api0",    "facility": "httpd",    "errno": "ENOENT",    "remote_host": "50.57.61.4",    "remote_port": "40100",    "path": "/var/www/no-such-file",    "uri": "/no-such-file",    "level": 4,    "headers": {        "user-agent": "BadAgent/1.0",        "connection": "close",        "accept": "*/*"    },    "method": "GET",    "unique_id": ".rh-g2Tm.h-ord1.product.api0.r-axAIO3bO.c-9210.ts-1324830675.v-24e946e"}
    63. 63. And instead of logging to this
    64. 64. We log to this
    65. 65. Then I can query• By any element in the JSON dictionary• Across multiple tiers (client, server, web)• Across platforms• I can assign a unique ID to an iOS event and follow the API request across my whole stack
    66. 66. If it serves up all FB’s contentPretty sure it can handle a few million print statements
    67. 67. Have you figured outthe common problem? we’re talking about EVENTS
    68. 68. • printf / NSLog • e-mail received• server logs • sales inquiry• webapp logs • bug resolved• page view • User taps button• error • unit test run• download notification • patch committed
    69. 69. Why do we need 10 systems to solve one problem?
    70. 70. Only four kinds of events• Things logged only to stderr / console• Things logged to a unified network log file• Things somebody is notified about• Things somebody gets woken up about
    71. 71. So about that server...
    72. 72. I’ve been playing with loggly• So far, so good• I downloaded the damn logs• Events queryable in ~5s for me• High-perf log import (raw sock available)• Filter and query on a hadoop cluster console, a few gotchas but pretty neat• I’m routlinely pushing up 100k events per day and I’m a tiny user
    73. 73. Searching by bug # across platforms
    74. 74. Do you really expectme to build that dict? I just want to type NSLog(@”Something”);
    75. 75. The hardest part of this whole mess is figuring out what to log and actually logging it
    76. 76. Remember this? An event that happened three months ago can be themost important thing to understand the events of today.
    77. 77. We can add state to our log statement• dict.username = “Frank”• log(“something”) //username=Frank• log(“something”) //username=Frank To reduce the work
    78. 78. But we have to remember to undo it• dict.username = “bill”• begin_session(bill)• log(“something”)• begin_session(james)• log(“something else”) //oops!
    79. 79. Kind of like memory management• Balance malloc with free• Balance retain with release• Balance new with delete
    80. 80. But some memory is auto-managedint i = 0;{ int j = 0; //can access j and i}//can access only i
    81. 81. Can we use a similar trick to build our logging dictionary?
    82. 82. JUCHE • printf / NSLog • e-mail received • server logs • sales inquiry • webapp logs • bug resolved • page view • User taps button • error • unit test run • download • patch committed notificationOne library to rule them all
    83. 83. • iOS, Python
    84. 84. • iOS, Python• Builds the dict with ease
    85. 85. • iOS, Python• Builds the dict with ease• Logs to stderr, loggly
    86. 86. Events are nested
    87. 87. Events are nested • Today
    88. 88. Events are nested • Today • Got up
    89. 89. Events are nested • Today • Got up • Made coffee
    90. 90. Events are nested • Today • Got up • Made coffee • Closed bugs
    91. 91. Events are nested • Today • Got up • Made coffee • Closed bugs • Bug 1
    92. 92. Events are nested • Today • Got up • Made coffee • Closed bugs • Bug 1 • Bug 2
    93. 93. Events are nested • Today • Got up • Made coffee • Closed bugs • Bug 1 • Bug 2 • Had lunch
    94. 94. Events are scoped • Today • Got up • Made coffee • Closed bugs • Bug 1 • Bug 2 • Had lunchInner items inherit the settings of outer items
    95. 95. for(int i = 0; i < 3; i++) { [JucheLog revolt:@"i",[NSStringstringWithFormat:@"%d",i],^{ JUCHE(JINFO,@"My awesomeloop"); }]; }
    96. 96. for(int i = 0; i < 3; i++) { [JucheLog revolt:@"i",[NSString stringWithFormat:@"%d",i],^{ JUCHE(JINFO,@"My awesome loop"); }]; }| [INFO] 19:12:11 My awesome loop i=0 juchelog.py:142| [INFO] 19:12:11 My awesome loop i=1 juchelog.py:142| [INFO] 19:12:11 My awesome loop i=2 juchelog.py:142
    97. 97. for i in range(0,3): with juche.revolution(i=i,eternal_president="kim-il-sun"): juche.info("Outer loop!") for j in range(0,2): with juche.revolution(j=j): juche.info("Inner loop!")| [INFO] 19:12:11 Outer loop! i=0 eternal_president=kim-il-sun juchelog.py:147| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150| [INFO] 19:12:11 Outer loop! i=1 eternal_president=kim-il-sun juchelog.py:147| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150| [INFO] 19:12:11 Outer loop! i=2 eternal_president=kim-il-sun juchelog.py:147| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150
    98. 98. | [INFO] 19:12:11 Outer loop! i=0 eternal_president=kim-il-sun juchelog.py:147| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150| [INFO] 19:12:11 Outer loop! i=1 eternal_president=kim-il-sun juchelog.py:147| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150| [INFO] 19:12:11 Outer loop! i=2 eternal_president=kim-il-sun juchelog.py:147| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150{"function": "__block_global_0", "who": "G88014V4XYK", "indent": "2","thread": "main", "eternal_president": "kim-il-sun", "i": "2", "app":"com.dca.JucheLogTestMac", "j": "2", "version": "1", "file":"JucheLogTests.m", "msg": "Inner loop!", "line": "44", "level":"info"}
    99. 99. Call for contributorshttp://github.com/drewcrawford/JucheLog
    100. 100. Call for contributors Use the libraryhttp://github.com/drewcrawford/JucheLog
    101. 101. Call for contributors Use the library See if it solves your problemshttp://github.com/drewcrawford/JucheLog
    102. 102. Call for contributors Use the library See if it solves your problems Critique the syntaxhttp://github.com/drewcrawford/JucheLog
    103. 103. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal outputhttp://github.com/drewcrawford/JucheLog
    104. 104. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal output Port it to (PHP, RoR, JS...)http://github.com/drewcrawford/JucheLog
    105. 105. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal output Port it to (PHP, RoR, JS...) Write convenience functionshttp://github.com/drewcrawford/JucheLog
    106. 106. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal output Port it to (PHP, RoR, JS...) Write convenience functions “Log every Y times”http://github.com/drewcrawford/JucheLog
    107. 107. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal output Port it to (PHP, RoR, JS...) Write convenience functions “Log every Y times” Log+UIAlertViewhttp://github.com/drewcrawford/JucheLog
    108. 108. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal output Port it to (PHP, RoR, JS...) Write convenience functions “Log every Y times” Log+UIAlertView Log more types (NSError)http://github.com/drewcrawford/JucheLog
    109. 109. Call for contributors Use the library See if it solves your problems Critique the syntax Critique the terminal output Port it to (PHP, RoR, JS...) Write convenience functions “Log every Y times” Log+UIAlertView Log more types (NSError) streamlined opt-outhttp://github.com/drewcrawford/JucheLog
    110. 110. So how evil is this?
    111. 111. I am not a lawyer And this is not legal advice
    112. 112. 17.1 Apps cannot transmit data about a user withoutobtaining the users prior permission and providing theuser with access to information about how and where the data will be used
    113. 113. Your EULA says Consent to Use of Data:You agree that ApplicationProvider may collect and use technical data and related information, including but not limited to technical information about Your device, system and application software, and peripherals, that is gathered periodically to facilitate the provision of software updates, productsupport and other services to You (if any) related to the Licensed Application. Application Provider may use this information, as long as it is in a form that does not personally identify You, to improve its products or to provide services or technologies to You.
    114. 114. Don’t be evil
    115. 115. I use data
    116. 116. I use data• To diagnose and fix bugs and crashes
    117. 117. I use data• To diagnose and fix bugs and crashes• To learn where users are getting stuck
    118. 118. I use data• To diagnose and fix bugs and crashes• To learn where users are getting stuck• To learn what percentage of my customer base experiences issues
    119. 119. I use data• To diagnose and fix bugs and crashes• To learn where users are getting stuck• To learn what percentage of my customer base experiences issues• To plan new features and updates
    120. 120. I use data• To diagnose and fix bugs and crashes• To learn where users are getting stuck• To learn what percentage of my customer base experiences issues• To plan new features and updates• To A/B test and see what people like
    121. 121. I use data• To diagnose and fix bugs and crashes• To learn where users are getting stuck• To learn what percentage of my customer base experiences issues• To plan new features and updates• To A/B test and see what people like• To understand my customers’ needs
    122. 122. I use data• To diagnose and fix bugs and crashes• To learn where users are getting stuck• To learn what percentage of my customer base experiences issues• To plan new features and updates• To A/B test and see what people like• To understand my customers’ needs• Always for the benefit of the users
    123. 123. Not to sell to someone else
    124. 124. (Don’t even log personal data)
    125. 125. So
    126. 126. Get a good signal
    127. 127. Turn down the noise
    128. 128. Try out Juchehttp://github.com/drewcrawford/JucheLog
    129. 129. Look for common problems and build common solutions
    130. 130. • Tiny iOS Developer• Mix of contracts & products• Many other dev tools like JucheLog
    131. 131. Drew Crawfordhttp://drewcrawfordapps.comdrew@drewcrawfordapps.comhttp://sealedabstract.com <--you should be reading this

    ×