1. Logging & Analytics
For fun and profit

2. Let me tell you about a bug

3. Let me tell you about a bug
• ~2% of our customers were affected

4. Let me tell you about a bug
• ~2% of our customers were affected
• Resolved (WORKSFORME)

5. Let me tell you about a bug
• ~2% of our customers were affected
• Resolved (WORKSFORME)
• People will complain, but they won’t send
me crash reports

6. Let me tell you about a bug
• ~2% of our customers were affected
• Resolved (WORKSFORME)
• People will complain, but they won’t send
me crash reports
• Apple’s crash reporter shows no crashes

7. I wrote my own crash reporter

9. c

10. The confidence you get from finding out about every crash,
anywhere in the world, is crucial to delivering a high quality
product that needs to be used in the wild. For those of us in
the consumer software business it's absolutely critical.You
can't rely on your customers to tell you about crashes—many
of them may not be technical enough, and most of them won't
bother to take time off of their own important work to give
you a useful crash report unless you make it completely
automatic. - Joel Spolsky

11. It’s 2012, don’t roll your own
Use quincykit.net

12. And everybody lived
happily ever after...

13. Until...

14. NSError *err = nil;
[somethingWithErr:&err];
if (err) {
UIAlertView *errorAlertView =
[[UIAlertView alloc]
initWithTitle:@”Error” message:[err
description] delegate:nil
cancelButtonTitle:OK
otherButtonTitles:nil];
[errorAlertView show];
[errorAlertView release];
}

15. NSError *err = nil;
[somethingWithErr:&err];
if (err) {
UIAlertView *errorAlertView =
[[UIAlertView alloc]
initWithTitle:@”Error” message:[err
description] delegate:nil
cancelButtonTitle:OK
otherButtonTitles:nil];
[errorAlertView show];
[errorAlertView release];
}

16. Or do I want one of these?

17. So I wrote an error logger

18. And everybody lived
happily ever after...

19. Until...

20. I started maintaining
non-iOS code
http://github.com/drewcrawford/buildbot

24. I have too much noise

25. But not enough signal

26. But too much noise

28. But not enough signal

29. Within the same msg

31. Meanwhile...

32. How do I make money?
1. Customer hears about me
2. ????
3. Profit!

33. How do I make money?
1. Customer hears about me
2. ???? What’s going on here?
3. Profit!

34. Web Analytics

35. Web Analytics
• How many different times does a person
visit my site before they convert?

36. Web Analytics
• How many different times does a person
visit my site before they convert?
• How do my blog readers’ habits change
over time?

37. Web Analytics
• How many different times does a person
visit my site before they convert?
• How do my blog readers’ habits change
over time?
• What indicators (referrers, time-on-site,
months of loyalty) correlate to high quality
users?

38. Web Analytics
• How many different times does a person
visit my site before they convert?
• How do my blog readers’ habits change
over time?
• What indicators (referrers, time-on-site,
months of loyalty) correlate to high quality
users?
• Can I prioritize sales leads automatically by
their behavior on my site?

39. App Analytics

40. App Analytics
• Do websites, SEM ads, or product videos
matter?

41. App Analytics
• Do websites, SEM ads, or product videos
matter?
• What causes people to make an IAP?

42. App Analytics
• Do websites, SEM ads, or product videos
matter?
• What causes people to make an IAP?
• Should the button be here or there?

43. App Analytics
• Do websites, SEM ads, or product videos
matter?
• What causes people to make an IAP?
• Should the button be here or there?
• Is anyone really going to use this feature?

44. App Analytics
• Do websites, SEM ads, or product videos
matter?
• What causes people to make an IAP?
• Should the button be here or there?
• Is anyone really going to use this feature?
• How many of my app users still use after 3
days?

45. Nothing on this page
answers any questions

46. Anatomy of a useful page

47. Anatomy of a useful page
HOW???

49. About Flurry

50. [FlurryAnalytics
startSession:@"1R1NQX1SPPWD7SDPI9WZ"];
for(int i = 0; i < 5; i++) {
[FlurryAnalytics logEvent:@"In for loop"
withParameters:[NSDictionary dictionaryWithObject:
[NSNumber numberWithInt:i] forKey:@"i"]];
}

51. And then I waited
several hours...

52. And all I got was
this lousy summary

53. The world’s most
worthless pie chart

54. In fairness, you can
export a few hundred records into Excel

55. But if you want full logs
You’d better ask nicely

56. If only we had a machine
that could process large quantities of data...

57. These are views
Not models. They draw pretty pictures. They’re not
your data.

60. It walks like raw data export
It talks like raw data export
But it’s not.
Look for yourself: http://mixpanel.com/docs/api-
documentation/data-export-api#libs-python

61. We don't provide raw log exports at this
time. It is something we are considering
adding to our pipeline, so thanks for writing
in with the feature request. - Mixpanel

62. I’m not saying there’s a conspiracy to
keep you from getting real event logs
But don’t believe you can export your data until you
actually download & check the damn logs

63. Meanwhile...

64. The epic unit test
• I’m working on a complicated “big-data” project
• The unit tests parse 100ks of points and take
hours
• When your compile-build-run-test loop is
measured in hours, you debug from an execution
trace, not by stepping through code
• One day the 200MB+ execution traces wouldn’t
upload to our bug tracker any more

65. The epic unit test
• I was building custom tools to parse through the
huge log files and give me the info I needed to
debug after-the-fact
• There’s a complicated system of macros
controlling log levels of different components

66. The epic unit test
• Even with all these tools, rarely would I log the
right things.
• Logging the wrong things costs me hours to re-
run the test.
• Over and over again.

67. Bion has a similar problem
on a different project

68. Meanwhile...

69. I read this article
http://journal.paul.querna.org/articles/2011/12/26/log-for-
machines-in-json/

70. What if instead of this
[Mon Dec 26 09:14:46 2011] [info] [client 50.57.61.4] File does
not exist: /var/www/no-such-file

71. We log this
{
"timestamp": 1324830675.076,
"status": "404",
"short_message": "File does not exist: /var/www/no-such-file",
"host": "ord1.product.api0",
"facility": "httpd",
"errno": "ENOENT",
"remote_host": "50.57.61.4",
"remote_port": "40100",
"path": "/var/www/no-such-file",
"uri": "/no-such-file",
"level": 4,
"headers": {
"user-agent": "BadAgent/1.0",
"connection": "close",
"accept": "*/*"
},
"method": "GET",
"unique_id": ".rh-g2Tm.h-ord1.product.api0.r-axAIO3bO.c-9210.ts-1324830675.v-24e946e"
}

72. And instead of logging
to this

73. We log to this

74. Then I can query
• By any element in the JSON dictionary
• Across multiple tiers (client, server, web)
• Across platforms
• I can assign a unique ID to an iOS event
and follow the API request across my whole
stack

75. If it serves up all FB’s content
Pretty sure it can handle a few million print statements

76. Have you figured out
the common problem?
we’re talking about
EVENTS

77. • printf / NSLog • e-mail received
• server logs • sales inquiry
• webapp logs • bug resolved
• page view • User taps button
• error • unit test run
• download notification • patch committed

78. Why do we need 10 systems
to solve one problem?

79. Only four kinds of events
• Things logged only to stderr / console
• Things logged to a unified network log file
• Things somebody is notified about
• Things somebody gets woken up about

80. So about that server...

81. I’ve been playing with loggly
• So far, so good
• I downloaded the damn logs
• Events queryable in ~5s for me
• High-perf log import (raw sock available)
• Filter and query on a hadoop cluster console, a
few gotchas but pretty neat
• I’m routlinely pushing up 100k events per day
and I’m a tiny user

82. Searching by bug #
across platforms

83. Do you really expect
me to build that dict?
I just want to type NSLog(@”Something”);

84. The hardest part of this whole mess
is figuring out what to log
and actually logging it

85. Remember this?
An event that happened three months ago can be the
most important thing to understand the events of today.

86. We can add state to
our log statement
• dict.username = “Frank”
• log(“something”) //username=Frank
• log(“something”) //username=Frank
To reduce the work

87. But we have to
remember to undo it
• dict.username = “bill”
• begin_session(bill)
• log(“something”)
• begin_session(james)
• log(“something else”) //oops!

88. Kind of like memory
management
• Balance malloc with free
• Balance retain with release
• Balance new with delete

89. But some memory is
auto-managed
int i = 0;
{
int j = 0;
//can access j and i
}
//can access only i

90. Can we use a similar trick
to build our logging dictionary?

91. JUCHE
• printf / NSLog
• e-mail received
• server logs
• sales inquiry
• webapp logs
• bug resolved
• page view
• User taps button
• error
• unit test run
• download
• patch committed
notification
One library to rule them all

93. • iOS, Python

94. • iOS, Python
• Builds the dict with ease

95. • iOS, Python
• Builds the dict with ease
• Logs to stderr, loggly

96. Events are nested

97. Events are nested
• Today

98. Events are nested
• Today
• Got up

99. Events are nested
• Today
• Got up
• Made coffee

100. Events are nested
• Today
• Got up
• Made coffee
• Closed bugs

101. Events are nested
• Today
• Got up
• Made coffee
• Closed bugs
• Bug 1

102. Events are nested
• Today
• Got up
• Made coffee
• Closed bugs
• Bug 1
• Bug 2

103. Events are nested
• Today
• Got up
• Made coffee
• Closed bugs
• Bug 1
• Bug 2
• Had lunch

104. Events are scoped
• Today
• Got up
• Made coffee
• Closed bugs
• Bug 1
• Bug 2
• Had lunch
Inner items inherit the settings of outer items

105. for(int i = 0; i < 3; i++) {
[JucheLog revolt:@"i",[NSString
stringWithFormat:@"%d",i],^{
JUCHE(JINFO,@"My awesome
loop");
}];
}

106. for(int i = 0; i < 3; i++) {
[JucheLog revolt:@"i",[NSString stringWithFormat:@"%d",i],^{
JUCHE(JINFO,@"My awesome loop");
}];
}
| [INFO] 19:12:11 My awesome loop i=0 juchelog.py:142
| [INFO] 19:12:11 My awesome loop i=1 juchelog.py:142
| [INFO] 19:12:11 My awesome loop i=2 juchelog.py:142

107. for i in range(0,3):
with juche.revolution(i=i,eternal_president="kim-il-sun"):
juche.info("Outer loop!")
for j in range(0,2):
with juche.revolution(j=j):
juche.info("Inner loop!")
| [INFO] 19:12:11 Outer loop! i=0 eternal_president=kim-il-sun juchelog.py:147
| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150
| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150
| [INFO] 19:12:11 Outer loop! i=1 eternal_president=kim-il-sun juchelog.py:147
| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150
| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150
| [INFO] 19:12:11 Outer loop! i=2 eternal_president=kim-il-sun juchelog.py:147
| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150
| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150

108. | [INFO] 19:12:11 Outer loop! i=0 eternal_president=kim-il-sun juchelog.py:147
| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150
| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150
| [INFO] 19:12:11 Outer loop! i=1 eternal_president=kim-il-sun juchelog.py:147
| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150
| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150
| [INFO] 19:12:11 Outer loop! i=2 eternal_president=kim-il-sun juchelog.py:147
| | [INFO] 19:12:11 Inner loop! j=0 juchelog.py:150
| | [INFO] 19:12:11 Inner loop! j=1 juchelog.py:150
{"function": "__block_global_0", "who": "G88014V4XYK", "indent": "2",
"thread": "main", "eternal_president": "kim-il-sun", "i": "2", "app":
"com.dca.JucheLogTestMac", "j": "2", "version": "1", "file":
"JucheLogTests.m", "msg": "Inner loop!", "line": "44", "level":
"info"}

109. Call for contributors
http://github.com/drewcrawford/JucheLog

110. Call for contributors
Use the library
http://github.com/drewcrawford/JucheLog

111. Call for contributors
Use the library
See if it solves your problems
http://github.com/drewcrawford/JucheLog

112. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
http://github.com/drewcrawford/JucheLog

113. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
http://github.com/drewcrawford/JucheLog

114. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
Port it to (PHP, RoR, JS...)
http://github.com/drewcrawford/JucheLog

115. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
Port it to (PHP, RoR, JS...)
Write convenience functions
http://github.com/drewcrawford/JucheLog

116. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
Port it to (PHP, RoR, JS...)
Write convenience functions
“Log every Y times”
http://github.com/drewcrawford/JucheLog

117. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
Port it to (PHP, RoR, JS...)
Write convenience functions
“Log every Y times”
Log+UIAlertView
http://github.com/drewcrawford/JucheLog

118. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
Port it to (PHP, RoR, JS...)
Write convenience functions
“Log every Y times”
Log+UIAlertView
Log more types (NSError)
http://github.com/drewcrawford/JucheLog

119. Call for contributors
Use the library
See if it solves your problems
Critique the syntax
Critique the terminal output
Port it to (PHP, RoR, JS...)
Write convenience functions
“Log every Y times”
Log+UIAlertView
Log more types (NSError)
streamlined opt-out
http://github.com/drewcrawford/JucheLog

120. So how evil is this?

121. I am not a lawyer
And this is not legal advice

122. 17.1 Apps cannot transmit data about a user without
obtaining the user's prior permission and providing the
user with access to information about how and where
the data will be used

123. Your EULA says
Consent to Use of Data:You agree that Application
Provider may collect and use technical data and related
information, including but not limited to technical
information about Your device, system and application
software, and peripherals, that is gathered periodically
to facilitate the provision of software updates, product
support and other services to You (if any) related to the
Licensed Application. Application Provider may use this
information, as long as it is in a form that does not
personally identify You, to improve its products or to
provide services or technologies to You.

124. Don’t be evil

125. I use data

126. I use data
• To diagnose and fix bugs and crashes

127. I use data
• To diagnose and fix bugs and crashes
• To learn where users are getting stuck

128. I use data
• To diagnose and fix bugs and crashes
• To learn where users are getting stuck
• To learn what percentage of my customer
base experiences issues

129. I use data
• To diagnose and fix bugs and crashes
• To learn where users are getting stuck
• To learn what percentage of my customer
base experiences issues
• To plan new features and updates

130. I use data
• To diagnose and fix bugs and crashes
• To learn where users are getting stuck
• To learn what percentage of my customer
base experiences issues
• To plan new features and updates
• To A/B test and see what people like

131. I use data
• To diagnose and fix bugs and crashes
• To learn where users are getting stuck
• To learn what percentage of my customer
base experiences issues
• To plan new features and updates
• To A/B test and see what people like
• To understand my customers’ needs

132. I use data
• To diagnose and fix bugs and crashes
• To learn where users are getting stuck
• To learn what percentage of my customer
base experiences issues
• To plan new features and updates
• To A/B test and see what people like
• To understand my customers’ needs
• Always for the benefit of the users

133. Not to sell to someone else

134. (Don’t even log personal data)

135. So

136. Get a good signal

137. Turn down the noise

138. Try out Juche
http://github.com/drewcrawford/JucheLog

139. Look for common problems
and build common solutions

140. • Tiny iOS Developer
• Mix of contracts &
products
• Many other dev tools
like JucheLog

141. Drew Crawford
http://drewcrawfordapps.com
drew@drewcrawfordapps.com
http://sealedabstract.com <--you should be reading this