Submit Search
Upload
Your Code is Wrong
•
29 likes
•
31,503 views
N
nathanmarz
Follow
My keynote at NoSQL Now! on August 21st, 2013
Read less
Read more
Technology
Report
Share
Report
Share
1 of 106
Download now
Download to read offline
Recommended
The Epistemology of Software Engineering
The Epistemology of Software Engineering
nathanmarz
Using Simplicity to Make Hard Big Data Problems Easy
Using Simplicity to Make Hard Big Data Problems Easy
nathanmarz
The inherent complexity of stream processing
The inherent complexity of stream processing
nathanmarz
Storm
Storm
nathanmarz
Storm: Distributed and fault tolerant realtime computation
Storm: Distributed and fault tolerant realtime computation
Ferran Galí Reniu
Introduction to Storm
Introduction to Storm
Eugene Dvorkin
Learning Stream Processing with Apache Storm
Learning Stream Processing with Apache Storm
Eugene Dvorkin
Storm presentation
Storm presentation
Shyam Raj
Recommended
The Epistemology of Software Engineering
The Epistemology of Software Engineering
nathanmarz
Using Simplicity to Make Hard Big Data Problems Easy
Using Simplicity to Make Hard Big Data Problems Easy
nathanmarz
The inherent complexity of stream processing
The inherent complexity of stream processing
nathanmarz
Storm
Storm
nathanmarz
Storm: Distributed and fault tolerant realtime computation
Storm: Distributed and fault tolerant realtime computation
Ferran Galí Reniu
Introduction to Storm
Introduction to Storm
Eugene Dvorkin
Learning Stream Processing with Apache Storm
Learning Stream Processing with Apache Storm
Eugene Dvorkin
Storm presentation
Storm presentation
Shyam Raj
Apache Storm Internals
Apache Storm Internals
Humoyun Ahmedov
Storm Real Time Computation
Storm Real Time Computation
Sonal Raj
Storm
Storm
Pouyan Rezazadeh
Multi-tenant Apache Storm as a service
Multi-tenant Apache Storm as a service
Robert Evans
Apache Storm
Apache Storm
masifqadri
Apache Storm based Real Time Analytics for Recommending Trending Topics and S...
Apache Storm based Real Time Analytics for Recommending Trending Topics and S...
Humoyun Ahmedov
Spark vs storm
Spark vs storm
Trong Ton
Resource Aware Scheduling in Apache Storm
Resource Aware Scheduling in Apache Storm
DataWorks Summit/Hadoop Summit
Analysis big data by use php with storm
Analysis big data by use php with storm
毅 吕
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Brian Brazil
Introduction to Apache Storm
Introduction to Apache Storm
Tiziano De Matteis
Introduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & Example
Dung Ngua
Apache Storm and twitter Streaming API integration
Apache Storm and twitter Streaming API integration
Uday Vakalapudi
Storm and Cassandra
Storm and Cassandra
T Jake Luciani
Storm: The Real-Time Layer - GlueCon 2012
Storm: The Real-Time Layer - GlueCon 2012
Dan Lynn
Apache Spark
Apache Spark
masifqadri
Jan 2012 HUG: Storm
Jan 2012 HUG: Storm
Yahoo Developer Network
Using Hystrix to Build Resilient Distributed Systems
Using Hystrix to Build Resilient Distributed Systems
Matt Jacobs
Storm-on-YARN: Convergence of Low-Latency and Big-Data
Storm-on-YARN: Convergence of Low-Latency and Big-Data
DataWorks Summit
Real-Time Analytics with Apache Storm
Real-Time Analytics with Apache Storm
Taewoo Kim
Clojure at BackType
Clojure at BackType
nathanmarz
Relational Databases are Evolving To Support New Data Capabilities
Relational Databases are Evolving To Support New Data Capabilities
EDB
More Related Content
What's hot
Apache Storm Internals
Apache Storm Internals
Humoyun Ahmedov
Storm Real Time Computation
Storm Real Time Computation
Sonal Raj
Storm
Storm
Pouyan Rezazadeh
Multi-tenant Apache Storm as a service
Multi-tenant Apache Storm as a service
Robert Evans
Apache Storm
Apache Storm
masifqadri
Apache Storm based Real Time Analytics for Recommending Trending Topics and S...
Apache Storm based Real Time Analytics for Recommending Trending Topics and S...
Humoyun Ahmedov
Spark vs storm
Spark vs storm
Trong Ton
Resource Aware Scheduling in Apache Storm
Resource Aware Scheduling in Apache Storm
DataWorks Summit/Hadoop Summit
Analysis big data by use php with storm
Analysis big data by use php with storm
毅 吕
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Brian Brazil
Introduction to Apache Storm
Introduction to Apache Storm
Tiziano De Matteis
Introduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & Example
Dung Ngua
Apache Storm and twitter Streaming API integration
Apache Storm and twitter Streaming API integration
Uday Vakalapudi
Storm and Cassandra
Storm and Cassandra
T Jake Luciani
Storm: The Real-Time Layer - GlueCon 2012
Storm: The Real-Time Layer - GlueCon 2012
Dan Lynn
Apache Spark
Apache Spark
masifqadri
Jan 2012 HUG: Storm
Jan 2012 HUG: Storm
Yahoo Developer Network
Using Hystrix to Build Resilient Distributed Systems
Using Hystrix to Build Resilient Distributed Systems
Matt Jacobs
Storm-on-YARN: Convergence of Low-Latency and Big-Data
Storm-on-YARN: Convergence of Low-Latency and Big-Data
DataWorks Summit
Real-Time Analytics with Apache Storm
Real-Time Analytics with Apache Storm
Taewoo Kim
What's hot
(20)
Apache Storm Internals
Apache Storm Internals
Storm Real Time Computation
Storm Real Time Computation
Storm
Storm
Multi-tenant Apache Storm as a service
Multi-tenant Apache Storm as a service
Apache Storm
Apache Storm
Apache Storm based Real Time Analytics for Recommending Trending Topics and S...
Apache Storm based Real Time Analytics for Recommending Trending Topics and S...
Spark vs storm
Spark vs storm
Resource Aware Scheduling in Apache Storm
Resource Aware Scheduling in Apache Storm
Analysis big data by use php with storm
Analysis big data by use php with storm
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Introduction to Apache Storm
Introduction to Apache Storm
Introduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & Example
Apache Storm and twitter Streaming API integration
Apache Storm and twitter Streaming API integration
Storm and Cassandra
Storm and Cassandra
Storm: The Real-Time Layer - GlueCon 2012
Storm: The Real-Time Layer - GlueCon 2012
Apache Spark
Apache Spark
Jan 2012 HUG: Storm
Jan 2012 HUG: Storm
Using Hystrix to Build Resilient Distributed Systems
Using Hystrix to Build Resilient Distributed Systems
Storm-on-YARN: Convergence of Low-Latency and Big-Data
Storm-on-YARN: Convergence of Low-Latency and Big-Data
Real-Time Analytics with Apache Storm
Real-Time Analytics with Apache Storm
Viewers also liked
Clojure at BackType
Clojure at BackType
nathanmarz
Relational Databases are Evolving To Support New Data Capabilities
Relational Databases are Evolving To Support New Data Capabilities
EDB
Scala Abide: A lint tool for Scala
Scala Abide: A lint tool for Scala
Iulian Dragos
Puppet at Google
Puppet at Google
Puppet
Why Spark?
Why Spark?
Álvaro Agea Herradón
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
In-Memory Computing Summit
The Need for Async @ ScalaWorld
The Need for Async @ ScalaWorld
Konrad Malawski
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
Helena Edelson
Purely Functional Data Structures in Scala
Purely Functional Data Structures in Scala
Vladimir Kostyukov
Monadic Java
Monadic Java
Mario Fusco
NewSQL overview, Feb 2015
NewSQL overview, Feb 2015
Ivan Glushkov
The Newest in Session Types
The Newest in Session Types
Roland Kuhn
Scala Days San Francisco
Scala Days San Francisco
Martin Odersky
Espresso: LinkedIn's Distributed Data Serving Platform (Paper)
Espresso: LinkedIn's Distributed Data Serving Platform (Paper)
Amy W. Tang
Functional Programming Patterns (BuildStuff '14)
Functional Programming Patterns (BuildStuff '14)
Scott Wlaschin
Concurrency: The Good, The Bad and The Ugly
Concurrency: The Good, The Bad and The Ugly
legendofklang
Viewers also liked
(16)
Clojure at BackType
Clojure at BackType
Relational Databases are Evolving To Support New Data Capabilities
Relational Databases are Evolving To Support New Data Capabilities
Scala Abide: A lint tool for Scala
Scala Abide: A lint tool for Scala
Puppet at Google
Puppet at Google
Why Spark?
Why Spark?
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
The Need for Async @ ScalaWorld
The Need for Async @ ScalaWorld
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
Purely Functional Data Structures in Scala
Purely Functional Data Structures in Scala
Monadic Java
Monadic Java
NewSQL overview, Feb 2015
NewSQL overview, Feb 2015
The Newest in Session Types
The Newest in Session Types
Scala Days San Francisco
Scala Days San Francisco
Espresso: LinkedIn's Distributed Data Serving Platform (Paper)
Espresso: LinkedIn's Distributed Data Serving Platform (Paper)
Functional Programming Patterns (BuildStuff '14)
Functional Programming Patterns (BuildStuff '14)
Concurrency: The Good, The Bad and The Ugly
Concurrency: The Good, The Bad and The Ugly
Similar to Your Code is Wrong
Security for AWS : Journey to Least Privilege (update)
Security for AWS : Journey to Least Privilege (update)
dhubbard858
Security for AWS: Journey to Least Privilege
Security for AWS: Journey to Least Privilege
Lacework
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Sylvain Kalache
Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
Brian Brazil
Hacking android apps by srini0x00
Hacking android apps by srini0x00
srini0x00
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Zhen Huang
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
Brian Brazil
Online java compiler with security editor
Online java compiler with security editor
IRJET Journal
How To Do Excel-Like Row Selection in jQuery DataTable?
How To Do Excel-Like Row Selection in jQuery DataTable?
Polyxer Systems
Internet census 2012
Internet census 2012
Giuliano Tavaroli
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
lior mazor
Procuring the Anomaly Packets and Accountability Detection in the Network
Procuring the Anomaly Packets and Accountability Detection in the Network
IOSR Journals
Deep Dive: AWS X-Ray London Summit 2017
Deep Dive: AWS X-Ray London Summit 2017
Randall Hunt
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Brian Brazil
Time Series Anomaly Detection with Azure and .NETT
Time Series Anomaly Detection with Azure and .NETT
Marco Parenzan
Node.js security tour
Node.js security tour
Giacomo De Liberali
Software Birthmark Based Theft/Similarity Comparisons of JavaScript Programs
Software Birthmark Based Theft/Similarity Comparisons of JavaScript Programs
Swati Patel
Interview with Dmitriy Vyukov - the author of Relacy Race Detector (RRD)
Interview with Dmitriy Vyukov - the author of Relacy Race Detector (RRD)
PVS-Studio
Secure programming with php
Secure programming with php
Mohmad Feroz
Information Management 2marks with answer
Information Management 2marks with answer
suchi2480
Similar to Your Code is Wrong
(20)
Security for AWS : Journey to Least Privilege (update)
Security for AWS : Journey to Least Privilege (update)
Security for AWS: Journey to Least Privilege
Security for AWS: Journey to Least Privilege
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
Hacking android apps by srini0x00
Hacking android apps by srini0x00
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
Online java compiler with security editor
Online java compiler with security editor
How To Do Excel-Like Row Selection in jQuery DataTable?
How To Do Excel-Like Row Selection in jQuery DataTable?
Internet census 2012
Internet census 2012
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
Procuring the Anomaly Packets and Accountability Detection in the Network
Procuring the Anomaly Packets and Accountability Detection in the Network
Deep Dive: AWS X-Ray London Summit 2017
Deep Dive: AWS X-Ray London Summit 2017
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Time Series Anomaly Detection with Azure and .NETT
Time Series Anomaly Detection with Azure and .NETT
Node.js security tour
Node.js security tour
Software Birthmark Based Theft/Similarity Comparisons of JavaScript Programs
Software Birthmark Based Theft/Similarity Comparisons of JavaScript Programs
Interview with Dmitriy Vyukov - the author of Relacy Race Detector (RRD)
Interview with Dmitriy Vyukov - the author of Relacy Race Detector (RRD)
Secure programming with php
Secure programming with php
Information Management 2marks with answer
Information Management 2marks with answer
More from nathanmarz
Demystifying Data Engineering
Demystifying Data Engineering
nathanmarz
Runaway complexity in Big Data... and a plan to stop it
Runaway complexity in Big Data... and a plan to stop it
nathanmarz
Storm: distributed and fault-tolerant realtime computation
Storm: distributed and fault-tolerant realtime computation
nathanmarz
ElephantDB
ElephantDB
nathanmarz
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
nathanmarz
The Secrets of Building Realtime Big Data Systems
The Secrets of Building Realtime Big Data Systems
nathanmarz
Cascalog workshop
Cascalog workshop
nathanmarz
Cascalog at Strange Loop
Cascalog at Strange Loop
nathanmarz
Cascalog at Hadoop Day
Cascalog at Hadoop Day
nathanmarz
Cascalog at May Bay Area Hadoop User Group
Cascalog at May Bay Area Hadoop User Group
nathanmarz
Cascalog
Cascalog
nathanmarz
Cascading
Cascading
nathanmarz
More from nathanmarz
(12)
Demystifying Data Engineering
Demystifying Data Engineering
Runaway complexity in Big Data... and a plan to stop it
Runaway complexity in Big Data... and a plan to stop it
Storm: distributed and fault-tolerant realtime computation
Storm: distributed and fault-tolerant realtime computation
ElephantDB
ElephantDB
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
The Secrets of Building Realtime Big Data Systems
The Secrets of Building Realtime Big Data Systems
Cascalog workshop
Cascalog workshop
Cascalog at Strange Loop
Cascalog at Strange Loop
Cascalog at Hadoop Day
Cascalog at Hadoop Day
Cascalog at May Bay Area Hadoop User Group
Cascalog at May Bay Area Hadoop User Group
Cascalog
Cascalog
Cascading
Cascading
Recently uploaded
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
apidays
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
debabhi2
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
Andrey Devyatkin
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
rafiqahmad00786416
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
apidays
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
Architecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Juan lago vázquez
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
DianaGray10
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
apidays
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
The Digital Insurer
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
sudhanshuwaghmare1
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Deepika Singh
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
The Digital Insurer
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
Recently uploaded
(20)
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
Architecting Cloud Native Applications
Architecting Cloud Native Applications
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Your Code is Wrong
1.
Your Code is
Wrong Nathan Marz @nathanmarz 1
2.
Let’s start with
an example
3.
Storm’s “reportError” method
4.
(Storm is a
realtime computation system, like Hadoop but for realtime)
5.
Storm architecture
6.
Storm architecture Master node
(similar to Hadoop JobTracker)
7.
Storm architecture Used for
cluster coordination
8.
Storm architecture Run worker
processes
9.
Storm’s “reportError” method
10.
Used to show
errors in the Storm UI
11.
Error info is
stored in Zookeeper
12.
What happens when
a user deploys code like this?
13.
Denial-of-service on Zookeeper and
cluster goes down
14.
Robust! Designed input space
Actual input space
15.
Your code is
wrong
16.
Your code is
literally wrong
17.
Your code is
wrong
18.
19.
Why do you
believe your code is correct?
20.
Your code Dependency 1 Dependency
2 Dependency 3
21.
Dependency 1 Dependency 4 Dependency
5
22.
Dependency 4 Dependency 6 Dependency
9 Dependency 7 Dependency 8
23.
Dependency 3,000,000 Hardware
24.
Electronics
25.
Chemistry
26.
Atomic physics
27.
Quantum mechanics
28.
I think I
can safely say that nobody understands quantum mechanics. Richard Feynman
29.
Your code is
wrong
30.
Your code ...
31.
All the software
you’ve used has had bugs in it
32.
Including the software you’ve
written
33.
Your code is sometimes
correct
34.
That’s good enough!
35.
36.
Treat code as
nondeterministic
37.
Embrace “your code
is wrong” to design better software
38.
Robust! Designed input space
Actual input space
39.
Robust! Designed input space
Actual input space
40.
An example
41.
Learning from Hadoop Jobtracker Job Job Job
42.
Learning from Hadoop Jobtracker Job Job Job
43.
Learning from Hadoop Jobtracker Job Job Job
44.
Your code is
wrong
45.
So your processes
will crash
46.
Storm’s daemons are process
fault-tolerant
47.
Storm Nimbus Topology Topology Topology
48.
Storm Nimbus Topology Topology Topology
49.
Storm Nimbus Topology Topology Topology
50.
Storm Nimbus Topology Topology Topology
51.
Storm Nimbus Topology Topology Topology
52.
Robust! Designed input space
Actual input space
53.
Robust! Designed input space
Actual input space
54.
The impact of
code being wrong
55.
Robust! Designed input space
Actual input space Failures! Bad performance! Security holes! Irrelevant!
56.
Design principle #1 Measuring
and monitoring are the foundation of solid engineering
57.
Measuring: Under what range
of inputs does my software function well?
58.
Monitoring: What’s the actual
input space of my software?
59.
Measure & Monitor Latency Throughput Stack
traces Buffer sizes Memory usage CPU usage #threads spawned ...
60.
How you monitor
your software is as important as its functionality
61.
Design principle #2 Embrace
immutability
62.
Read/write database Application
63.
MySQLApplication
64.
MongoDBApplication
65.
RiakApplication
66.
CassandraApplication
67.
HBaseApplication
68.
Your code is
wrong
69.
So data will
be corrupted
70.
And you may
not know why
71.
Views Immutable, ever-growing data Application Architecture based on
immutability
72.
Views Immutable, ever-growing data Application Lambda architecture
73.
Design principle #3 Minimize
dependencies
74.
The less that
can go wrong, the less that will go wrong
75.
Example: Storm’s usage of
Zookeeper
76.
Worker locations stored
in Zookeeper
77.
All workers must
know locations of other workers to send messages
78.
Two ways to
get location updates
79.
1. Poll Zookeeper Worker
Zookeeper
80.
2. Use Zookeeper
“watch” feature to get push notifications Worker Zookeeper
81.
Method 2 is
faster but relies on another feature
82.
Storm uses both
methods Worker Zookeeper
83.
If watch feature
fails, locations still propagate via polling
84.
Eliminating dependence justified by
small amount of code required
85.
Design principle #4 Explicitly
respect functional input ranges
86.
Storm’s “reportError” method
87.
Implement self-throttling to avoid
overloading other systems
88.
Design principle #5 Embrace
recomputation
89.
“Your code is
wrong” meanings 1. Design input space differs from actual input space 2. The logic of your code is wrong 3. Requirements are constantly changing
90.
You must be
able to change your code to match shifting requirements
91.
Example: blogging software
92.
New requirement: search
93.
Have to build
a search index
94.
95.
Recomputation gives you so
much more
96.
Views Immutable, ever-growing data Application
97.
Building software no
different than any other engineering
98.
The underlying challenges are
the same
99.
100.
101.
What will break
it?
102.
What are limits
of my dependencies?
103.
How can I
add redundancy to increase robustness?
104.
Can I isolate
failures?
105.
Our raw materials
are ideas instead of matter
106.
Thank you
Download now