A Look at the Network Searching for Truth in Distributed Applications c. scott andreas (cscotta) oscon 2012 - portland ore...
taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
THE NETWORK IS RELIABLEtaco.cat/oscon12
THE NETWORK IS SECURETHERE IS ONE ADMINISTRATORtaco.cat/oscon12
LATENCY IS ZERO      BANDWIDTH IS INFINITEtaco.cat/oscon12
THE NETWORK IS     HOMOGENOUS TOPOLOGY DOESN’T CHANGEtaco.cat/oscon12
TRANSPORT          COST IS ZEROtaco.cat/oscon12
where can i buy that?taco.cat/oscon12
[ another approach ]taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
NETWORK + YOUR APPS       ARE A GRAPHtaco.cat/oscon12
taco.cat/oscon12
YEAH BUT... WHAT DO WE DO WITH IT?taco.cat/oscon12
graphs change constantly     their edges can berepresented as a time seriestaco.cat/oscon12
GRAPH           TIME SERIES                   +taco.cat/oscon12
LEMMA 1: there exists no possible way for   applications to communicate      except via the networktaco.cat/oscon12
LEMMA 2:  applications are unable to fulfill their purpose without communic-ating and participating in a clustertaco.cat/o...
LEMMA 3: the network can be represented   as a time series and a graphtaco.cat/oscon12
LEMMA 4:   nearly all modes of failure in    distributed systems can beidentified and predicated this waytaco.cat/oscon12
WHAT CAN ONE OBSERVE?                     poor load balancingnetwork partitions                     bad deploys      GC pa...
taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
taco.cat/oscon12
TOOLS    – NProbe / NTop                 – R / RStudio    – CFlowd / flow-tools           – Esper    – TCPDump / TCPReplay...
FURTHER READING    – Network Flow Analysis                                                                      – Gigascop...
BONUS!                              a spec for thai chili salsa and reference implementation                              ...
BONUS!                                                                                U P                              a s...
A Look at the Network Searching for Truth in Distributed Applications oscon 2012 - portland oregontaco.cat/oscon12
Upcoming SlideShare
Loading in …5
×

A Look at the Network: Searching for Truth in Distributed Applications

2,757 views

Published on

A talk by C. Scott Andreas (@cscotta) of Boundary on "the network" and designing / deploying distributed applications.

This session offers a deep-dive into how application-level problems manifest at the network level. Some of these cases range from basic network partitions and node outages to sophisticated application-level changes such as garbage collections on managed runtimes, classes of bugs which evade conventional monitoring but constitute partial failures, changes in network activity based on database partitioning, load balancing, and sharding, and other warning signs that crop up at layer three long before wreaking havoc at layer seven as customer-visible failures begin to occur. Combining application-level metrics with network analytics is a powerful cocktail for identifying hot spots quickly, and connecting the dots out to the client closes the loop.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,757
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Look at the Network: Searching for Truth in Distributed Applications

  1. A Look at the Network Searching for Truth in Distributed Applications c. scott andreas (cscotta) oscon 2012 - portland oregontaco.cat/oscon12
  2. taco.cat/oscon12
  3. taco.cat/oscon12
  4. taco.cat/oscon12
  5. THE NETWORK IS RELIABLEtaco.cat/oscon12
  6. THE NETWORK IS SECURETHERE IS ONE ADMINISTRATORtaco.cat/oscon12
  7. LATENCY IS ZERO BANDWIDTH IS INFINITEtaco.cat/oscon12
  8. THE NETWORK IS HOMOGENOUS TOPOLOGY DOESN’T CHANGEtaco.cat/oscon12
  9. TRANSPORT COST IS ZEROtaco.cat/oscon12
  10. where can i buy that?taco.cat/oscon12
  11. [ another approach ]taco.cat/oscon12
  12. taco.cat/oscon12
  13. taco.cat/oscon12
  14. taco.cat/oscon12
  15. taco.cat/oscon12
  16. NETWORK + YOUR APPS ARE A GRAPHtaco.cat/oscon12
  17. taco.cat/oscon12
  18. YEAH BUT... WHAT DO WE DO WITH IT?taco.cat/oscon12
  19. graphs change constantly their edges can berepresented as a time seriestaco.cat/oscon12
  20. GRAPH TIME SERIES +taco.cat/oscon12
  21. LEMMA 1: there exists no possible way for applications to communicate except via the networktaco.cat/oscon12
  22. LEMMA 2: applications are unable to fulfill their purpose without communic-ating and participating in a clustertaco.cat/oscon12
  23. LEMMA 3: the network can be represented as a time series and a graphtaco.cat/oscon12
  24. LEMMA 4: nearly all modes of failure in distributed systems can beidentified and predicated this waytaco.cat/oscon12
  25. WHAT CAN ONE OBSERVE? poor load balancingnetwork partitions bad deploys GC pauses good deploys impaired nodes variance security breachestaco.cat/oscon12
  26. taco.cat/oscon12
  27. taco.cat/oscon12
  28. taco.cat/oscon12
  29. taco.cat/oscon12
  30. taco.cat/oscon12
  31. TOOLS – NProbe / NTop – R / RStudio – CFlowd / flow-tools – Esper – TCPDump / TCPReplay – Dynamic Time Warping (algo) – CollectD – Python / NumPy / SciPytaco.cat/oscon12
  32. FURTHER READING – Network Flow Analysis – Gigascope (No-Starch Press) (AT&T Research) – Eamonn Keogh – Dynamic Time Warping (“Atomic Wedgie”) (algo) – K-Snap – Studying Complex (“Efficient Aggr. for Graph Summ.”) – “Medians and Beyond” (Shrivastava) Adaptive Systems (Holland) – Exponential Smoothing: The State Space Approach (Hyndman) – HyperLogLog / Count-Min Sketchtaco.cat/oscon12
  33. BONUS! a spec for thai chili salsa and reference implementation – Half a thing of cilantro – 28oz peeled tomatoes – 6 cloves of garlic – Half a bag of hot thai chilis – 2 tablespoons of salt – Half a cucumber – 2 tablespoons of white vinegar – Handful of radishes – Juice from half a lime – One green bell pepper – Bit of parsley – A small red onion – Some lemongrasstaco.cat/oscon12
  34. BONUS! U P a spec for thai chili salsa and reference implementation L – L Half a thing of cilantro – 28oz peeled tomatoes A – 6 cloves of garlic – Half a bag of hot thai chilis – T 2 tablespoons of salt I – Half a cucumber – 2 tablespoons of white vinegar – X Handful of radishes I – Juice from half a lime – One green bell pepper – M Bit of parsley – A small red onion – Some lemongrasstaco.cat/oscon12
  35. A Look at the Network Searching for Truth in Distributed Applications oscon 2012 - portland oregontaco.cat/oscon12

×