My keynote at NoSQL Now! on August 21st, 2013

1. Your Code is Wrong
Nathan Marz
@nathanmarz 1

2. Let’s start with an example

3. Storm’s “reportError” method

4. (Storm is a realtime computation
system, like Hadoop but for realtime)

5. Storm architecture

6. Storm architecture
Master node (similar to Hadoop JobTracker)

7. Storm architecture
Used for cluster coordination

8. Storm architecture
Run worker processes

9. Storm’s “reportError” method

10. Used to show errors in the Storm UI

11. Error info is stored in Zookeeper

12. What happens when a user deploys code like this?

13. Denial-of-service on Zookeeper
and cluster goes down

14. Robust!
Designed input space Actual input space

15. Your code is wrong

16. Your code is literally wrong

17. Your code is wrong

19. Why do you believe your
code is correct?

20. Your code
Dependency 1
Dependency 2
Dependency 3

21. Dependency 1
Dependency 4
Dependency 5

22. Dependency 4
Dependency 6
Dependency 9
Dependency 7
Dependency 8

23. Dependency 3,000,000
Hardware

24. Electronics

25. Chemistry

26. Atomic physics

27. Quantum mechanics

28. I think I can safely say
that nobody understands
quantum mechanics.
Richard Feynman

29. Your code is wrong

30. Your code
...

31. All the software you’ve
used has had bugs in it

32. Including the software
you’ve written

33. Your code is
sometimes correct

34. That’s good enough!

36. Treat code as nondeterministic

37. Embrace “your code is wrong”
to design better software

38. Robust!
Designed input space Actual input space

39. Robust!
Designed input space Actual input space

40. An example

41. Learning from Hadoop
Jobtracker
Job
Job
Job

42. Learning from Hadoop
Jobtracker
Job
Job
Job

43. Learning from Hadoop
Jobtracker
Job
Job
Job

44. Your code is wrong

45. So your processes will crash

46. Storm’s daemons are
process fault-tolerant

47. Storm
Nimbus
Topology
Topology
Topology

48. Storm
Nimbus
Topology
Topology
Topology

49. Storm
Nimbus
Topology
Topology
Topology

50. Storm
Nimbus
Topology
Topology
Topology

51. Storm
Nimbus
Topology
Topology
Topology

52. Robust!
Designed input space Actual input space

53. Robust!
Designed input space Actual input space

54. The impact of code being wrong

55. Robust!
Designed input space Actual input space
Failures!
Bad performance!
Security holes!
Irrelevant!

56. Design principle #1
Measuring and monitoring are the
foundation of solid engineering

57. Measuring:
Under what range of inputs
does my software function well?

58. Monitoring:
What’s the actual input space
of my software?

59. Measure & Monitor
Latency
Throughput
Stack traces
Buffer sizes
Memory usage
CPU usage
#threads spawned
...

60. How you monitor your software
is as important as its functionality

61. Design principle #2
Embrace immutability

62. Read/write
database
Application

63. MySQLApplication

64. MongoDBApplication

65. RiakApplication

66. CassandraApplication

67. HBaseApplication

68. Your code is wrong

69. So data will be corrupted

70. And you may not know why

71. Views
Immutable,
ever-growing
data
Application
Architecture based on immutability

72. Views
Immutable,
ever-growing
data
Application
Lambda architecture

73. Design principle #3
Minimize dependencies

74. The less that can go wrong,
the less that will go wrong

75. Example:
Storm’s usage of Zookeeper

76. Worker locations stored in Zookeeper

77. All workers must know locations of
other workers to send messages

78. Two ways to get location updates

79. 1. Poll Zookeeper
Worker Zookeeper

80. 2. Use Zookeeper “watch”
feature to get push notifications
Worker Zookeeper

81. Method 2 is faster but
relies on another feature

82. Storm uses both methods
Worker Zookeeper

83. If watch feature fails, locations
still propagate via polling

84. Eliminating dependence justified
by small amount of code required

85. Design principle #4
Explicitly respect
functional input ranges

86. Storm’s “reportError” method

87. Implement self-throttling to
avoid overloading other systems

88. Design principle #5
Embrace recomputation

89. “Your code is wrong” meanings
1. Design input space differs from actual input space
2. The logic of your code is wrong
3. Requirements are constantly changing

90. You must be able to change your
code to match shifting requirements

91. Example: blogging software

92. New requirement: search

93. Have to build a search index

95. Recomputation gives
you so much more

96. Views
Immutable,
ever-growing
data
Application

97. Building software no different
than any other engineering

98. The underlying challenges
are the same

101. What will break it?

102. What are limits of my
dependencies?

103. How can I add redundancy
to increase robustness?

104. Can I isolate failures?

105. Our raw materials are
ideas instead of matter

106. Thank you