Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Building Reactive Distributed Systems
For Streaming Big Data, Analytics & ML
Helena Edelson @helenaedelson Reactive Summ...
@helenaedelson #reactivesummit
Distributed Systems & Big Data Platform
Engineer, Event-Driven systems, Streaming
Analytics...
@helenaedelson #reactivesummit
If you’re here you’ve taken the red pill
@helenaedelson #reactivesummit
It’s Not Easy Being…
Globally Distributed
Eventually Consistent
Highly Available
While hand...
@helenaedelson #reactivesummit
Massive event spikes & bursty traffic
Fast producers / slow consumers
Network partitioning ...
@helenaedelson #reactivesummit
Beware the C word
Consistency?
6
Convergence?
@helenaedelson #reactivesummit 7
Complexity, or…
@helenaedelson #reactivesummit 8
http://www.slideshare.net/palvaro/ricon-keynote-outwards-from-the-middle-of-the-maze/42
T...
@helenaedelson #reactivesummit
“Everything fails, all the time”
Start with this premise
Build failure, chaos routing
and i...
@helenaedelson #reactivesummit 10
The matrix is everywhere…
It is a world where humans
vs machines need to be
directly inv...
@helenaedelson #reactivesummit 11
Log analysis tools like
Splunk perpetuate this
illusion…
@helenaedelson #reactivesummit 12
Self Healing, Intelligent Platforms
@helenaedelson #reactivesummit 13
Self Healing, Intelligent Platforms
Imagine a world
where engineers designed software
wh...
@helenaedelson #reactivesummit 14
Alerting and Failure Response
The New Boilerplate
@helenaedelson #reactivesummit
Building Systems That Learn
You have systems receiving, processing & collecting data. They ...
@helenaedelson #reactivesummit
Define Failures vs Error Fault
Tolerance Pathways & Routing
Failures - more network related,...
@helenaedelson #reactivesummit 17
I want to keep all my apps
healthy, handle errors with a
similar strategies, and failure...
@helenaedelson #reactivesummit 18
In A Nutshell
Event
Stream
Petabytes Multi-DC
Aware Mgmt
Kernel
Clustered
Apps
Clustered...
@helenaedelson #reactivesummit 19
In A Nutshell
Multi-DC
Aware Mgmt
Kernel
Clustered
Apps
Clustered
Apps
Clustered
AppsClu...
@helenaedelson #reactivesummit
Multi-DC
Aware Mgmt
Kernel
20
X-DC-Aware Kernel
Clustered
Apps
Clustered
Apps
Clustered
App...
@helenaedelson #reactivesummit 21
I want to
route failure context
(when a node is alive
enough to talk)
to my.failures.fu ...
@helenaedelson #reactivesummit 22
Akka Cluster it. The cluster knows
the node is down. Many options/
mechanisms to send/pu...
@helenaedelson #reactivesummit 23
I think I’ll start with a
base cluster-aware Akka Extension for
my platform.
Any non-JVM...
@helenaedelson #reactivesummit 24
Then I can bake my fault tolerance
strategy & route awareness in every
app & node.
And m...
@helenaedelson #reactivesummit 25
And all logic and flows for graceful handling of
node lifecycle are baked in to the fram...
@helenaedelson #reactivesummit
Akka Cluster
Node-aware cluster membership
Gossip protocol
Status of all member nodes in th...
@helenaedelson #reactivesummit 27
Akka Cluster has tunable Gossip.I
can partition segments of my node ring
by role, and ro...
@helenaedelson #reactivesummit 28
Akka Cluster has
AdaptiveLoadBalancing logic.
I can route events away from nodes in trou...
@helenaedelson #reactivesummit 29
Dynamically Route by Node Health
Cluster Metrics API & AdaptiveLBR
Leader
Let’s pummel m...
@helenaedelson #reactivesummit 30
Dynamically Route by Node Health
Cluster Metrics API & AdaptiveLBR
Oh look,
slow consume...
@helenaedelson #reactivesummit 31
Dynamically Route by Node Health
Cluster Metrics API & AdaptiveLBR
Leader
The
Cluster le...
@helenaedelson #reactivesummit
Reactive - Proactive
32
@helenaedelson #reactivesummit
Proactive - Parallelized
Akka Cluster on nodes
Cluster LB routers
Node router Actors
Kafka ...
@helenaedelson #reactivesummit 34
@helenaedelson #reactivesummit 35
I drank the cool aid,
put on the costume,
now what…
@helenaedelson #reactivesummit 36
Embrace Uncertainty
@helenaedelson #reactivesummit
So many opportunities for
failure in this statement
Eventually Consistent Across DCs
37
@helenaedelson #reactivesummit 38
there is no now…
Embrace Asynchrony & Location Transparency
…or where
@helenaedelson #reactivesummit
There Is No Now
39
US-East-1
MirrorMaker
EU-west-1
ZK
ZK
Akka
micro
services
Akka
micro
ser...
@helenaedelson #reactivesummit 40
US-East-1
MirrorMaker
EU-west-1
ZK
ZK
Akka
micro
services
Akka
micro
services
Akka
micro...
@helenaedelson #reactivesummit
Replaying Data from Time (T)
General time - We updated some compute
algorithms which run on...
@helenaedelson #reactivesummit
Cassandra Bakes Clustered
Ordering Into Data Model
CREATE TABLE timeseries_t (
id text,
yea...
@helenaedelson #reactivesummit 43
US-East-1
MirrorMaker
EU-west-1
ZK
ZK
Compute Clusters
Event Stream
Build Apps Not Clust...
@helenaedelson #reactivesummit
Kafka Streams Key Features
Secure stream processing using Kafka security
Elastic and highly...
@helenaedelson #reactivesummit
Kafka Streams Key Features
Interactive queries
Time Model & Windowing
Supports late-arrivin...
@helenaedelson #reactivesummit
val builder = new KStreamBuilder()

// Consume from topic x in the stream, from any or one ...
@helenaedelson #reactivesummit 47
Kafka Streams: KTable
val locations: KTable[UserId, Location] = builder.table(“user-loca...
@helenaedelson #reactivesummit
val streams = new KafkaStreams(builder, props)
streams.start()
// Called when a stream thre...
@helenaedelson #reactivesummit 49
Integrating Intelligence
For Fast Data and Feedback Loops
@helenaedelson #reactivesummit
Translation Layer
50
US-East-1
MirrorMaker
EU-west-1
PubSub
Apps
PubSub
Apps
PubSub
Apps
Co...
@helenaedelson #reactivesummit
Translation Layer
51
US-East-1
MirrorMaker
EU-west-1
PubSub
Apps
PubSub
Apps
PubSub
Apps
Co...
@helenaedelson #reactivesummit
Translation Layer
52
US-East-1
MirrorMaker
EU-west-1
PubSub
Apps
PubSub
Apps
PubSub
Apps
Co...
@helenaedelson #reactivesummit 53
Your Akka Cluster got in my
Reactive Distributed Analytics
Database!
Walking with a jar ...
@helenaedelson #reactivesummit
FiloDB: Reactive
Scala & SBT
Spark (also written in scala)
Akka actors & Akka Cluster for C...
@helenaedelson #reactivesummit
import filodb.spark._
KafkaUtils.createDirectStream[..](..)
.map(_._2)
.map(FuEvent(_))
.fo...
@helenaedelson #reactivesummit
Read: FiloDB & Spark SQL
56
import filodb.spark._
val df1 = sqlContext.read.format(“filodb....
@helenaedelson #reactivesummit 57
Determine
Train the
model
Train on historic data
Score
Based on Objectives
Proactive
Res...
@helenaedelson #reactivesummit
What do I want to predict & what can I learn from the data?
Which attributes in datasets ar...
@helenaedelson #reactivesummit 59
I need fast ML feedback loops,
what tools should I pick?
Depends…
@helenaedelson #reactivesummit
Read from stored training data
into MLLib
60
import org.apache.spark.mllib.stat.Statistics
...
@helenaedelson #reactivesummit
Read from the stream to MLLib
& store for later work in FiloDB
61
val stream = KafkaUtils.c...
@helenaedelson #reactivesummit 62
Parting Thoughts
@helenaedelson #reactivesummit
Don’t take when someone says X
doesn’t work at face value.
It may not have been the right c...
@helenaedelson #reactivesummit
Modular
does one thing
Decoupled
knows one thing
Reusable & Extendable
when not final or
pr...
@helenaedelson #reactivesummit 65
Did you just duplicate codez?
@helenaedelson #reactivesummit 66
Versus duplicating, with disparate
strategies, all over your teams & orgs
Build A Reacti...
@helenaedelson #reactivesummit
Byzantine Fault Tolerance?
67
Looks
like I'll miss
standup
@helenaedelson #reactivesummit
twitter.com/helenaedelson
slideshare.net/helenaedelson
github.com/helena
Thanks!
Upcoming SlideShare
Loading in …5
×

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning

22,182 views

Published on

Building Self Healing, Intelligent Platforms, systems that learn, multi-datacenter, removing human intervention with ML. Reactive Summit 2016 @helenaedelson

Published in: Technology

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning

  1. 1. 1 Building Reactive Distributed Systems For Streaming Big Data, Analytics & ML Helena Edelson @helenaedelson Reactive Summit 2016
  2. 2. @helenaedelson #reactivesummit Distributed Systems & Big Data Platform Engineer, Event-Driven systems, Streaming Analytics, Machine Learning, Scala Committer: FiloDB, Spark Cassandra Connector, Kafka Connect Cassandra Contributor: Akka (Akka Cluster), prev: Spring Integration Speaker: Reactive Summit, Kafka Summit, Spark Summit, Strata, QCon, Scala Days, Philly ETE 2 I play with big data twitter.com/helenaedelson slideshare.net/helenaedelson linkedin.com/in/helenaedelson github.com/helena and think about FT
  3. 3. @helenaedelson #reactivesummit If you’re here you’ve taken the red pill
  4. 4. @helenaedelson #reactivesummit It’s Not Easy Being… Globally Distributed Eventually Consistent Highly Available While handling TBs of data Self Healing Fast 4
  5. 5. @helenaedelson #reactivesummit Massive event spikes & bursty traffic Fast producers / slow consumers Network partitioning & out of sync systems DC down Failure before commit DDOS'ing yourself - no backpressure Lack of graceful lifecycle handling, i.e. data loss when auto-scaling down 5
  6. 6. @helenaedelson #reactivesummit Beware the C word Consistency? 6 Convergence?
  7. 7. @helenaedelson #reactivesummit 7 Complexity, or…
  8. 8. @helenaedelson #reactivesummit 8 http://www.slideshare.net/palvaro/ricon-keynote-outwards-from-the-middle-of-the-maze/42 The Real Topic ^Other
  9. 9. @helenaedelson #reactivesummit “Everything fails, all the time” Start with this premise Build failure, chaos routing and intelligence into your platform infrastructure Be fault proactive vs just reactive 9 - Werner Vogels
  10. 10. @helenaedelson #reactivesummit 10 The matrix is everywhere… It is a world where humans vs machines need to be directly involved in health of systems, respond to alerts…
  11. 11. @helenaedelson #reactivesummit 11 Log analysis tools like Splunk perpetuate this illusion…
  12. 12. @helenaedelson #reactivesummit 12 Self Healing, Intelligent Platforms
  13. 13. @helenaedelson #reactivesummit 13 Self Healing, Intelligent Platforms Imagine a world where engineers designed software while the machines did triage, failure analysis and took action to correct failures and learned from it…
  14. 14. @helenaedelson #reactivesummit 14 Alerting and Failure Response The New Boilerplate
  15. 15. @helenaedelson #reactivesummit Building Systems That Learn You have systems receiving, processing & collecting data. They can provide insight into trends, patterns, and when they start failing, your monitoring triggers alerts to humans. Alternative 1. Automatically route data only to nodes that can process it. 2. Automatically route failure contexts to event streams with ML algorithms to learn: What is normal? What is an anomaly? When these anomalies occur, what are they indicative of? 3. Automate proactive actions when those anomalies bubble up to a fault-aware layer. 15
  16. 16. @helenaedelson #reactivesummit Define Failures vs Error Fault Tolerance Pathways & Routing Failures - more network related, connectivity, … Errors - application errors, user input from REST requests, config load-time errors… Escape Loops and Feedback system Built-in failure detection
  17. 17. @helenaedelson #reactivesummit 17 I want to keep all my apps healthy, handle errors with a similar strategies, and failures with a similar strategy… …and share that data across datacenters with my multi-dc aware kernel to build smarter, automated systems.
  18. 18. @helenaedelson #reactivesummit 18 In A Nutshell Event Stream Petabytes Multi-DC Aware Mgmt Kernel Clustered Apps Clustered Apps Clustered AppsClustered Apps Clustered Apps Clustered Apps Clustered Apps Clustered Apps
  19. 19. @helenaedelson #reactivesummit 19 In A Nutshell Multi-DC Aware Mgmt Kernel Clustered Apps Clustered Apps Clustered AppsClustered Apps Clustered Apps Clustered Apps Clustered Apps Clustered Apps Danger Sauce Event Stream Petabytes
  20. 20. @helenaedelson #reactivesummit Multi-DC Aware Mgmt Kernel 20 X-DC-Aware Kernel Clustered Apps Clustered Apps Clustered AppsClustered Apps Clustered Apps Clustered Apps Clustered Apps Clustered Apps Event Stream Petabytes Danger Sauce
  21. 21. @helenaedelson #reactivesummit 21 I want to route failure context (when a node is alive enough to talk) to my.failures.fu Kafka topic for my X-DC kernel to process through my ML streams to learn about usage, traffic, memory patterns and anomalies, to anticipate failures, intercept and correct Without Human Intervention
  22. 22. @helenaedelson #reactivesummit 22 Akka Cluster it. The cluster knows the node is down. Many options/ mechanisms to send/publish failure metadata to learning streams. If it’s a network failure and my node can’t report to the kernel? Bring it
  23. 23. @helenaedelson #reactivesummit 23 I think I’ll start with a base cluster-aware Akka Extension for my platform. Any non-JVM apps can share data with these via Kafka, ZeroMQ, Cassandra… Sew easy. Might sneak it in without a PR.
  24. 24. @helenaedelson #reactivesummit 24 Then I can bake my fault tolerance strategy & route awareness in every app & node. And make it know when to use Kafka and when to use Akka for communication.
  25. 25. @helenaedelson #reactivesummit 25 And all logic and flows for graceful handling of node lifecycle are baked in to the framework too. Dude, that’s tight. Graceful Lifecycle
  26. 26. @helenaedelson #reactivesummit Akka Cluster Node-aware cluster membership Gossip protocol Status of all member nodes in the ring Tunable, automatic failure detection No single point of failure or bottleneck A level of composability above Actors & Actor Hierarchy 26
  27. 27. @helenaedelson #reactivesummit 27 Akka Cluster has tunable Gossip.I can partition segments of my node ring by role, and route data by role or broadcast. Tunable Gossip
  28. 28. @helenaedelson #reactivesummit 28 Akka Cluster has AdaptiveLoadBalancing logic. I can route events away from nodes in trouble and directed to healthier nodes. Many options, highly configurable, good granularity. AdaptiveLoadBalancing* Router Cluster Metrics API
  29. 29. @helenaedelson #reactivesummit 29 Dynamically Route by Node Health Cluster Metrics API & AdaptiveLBR Leader Let’s pummel my cluster with critical data !
  30. 30. @helenaedelson #reactivesummit 30 Dynamically Route by Node Health Cluster Metrics API & AdaptiveLBR Oh look, slow consumers or something in GC thrash… Leader Uses EWMA algo to weight decay of health data
  31. 31. @helenaedelson #reactivesummit 31 Dynamically Route by Node Health Cluster Metrics API & AdaptiveLBR Leader The Cluster leader ‘knew’ the red node was nearing capacity & routed my traffic to healthy nodes.
  32. 32. @helenaedelson #reactivesummit Reactive - Proactive 32
  33. 33. @helenaedelson #reactivesummit Proactive - Parallelized Akka Cluster on nodes Cluster LB routers Node router Actors Kafka replication & routing across datacenters 33
  34. 34. @helenaedelson #reactivesummit 34
  35. 35. @helenaedelson #reactivesummit 35 I drank the cool aid, put on the costume, now what…
  36. 36. @helenaedelson #reactivesummit 36 Embrace Uncertainty
  37. 37. @helenaedelson #reactivesummit So many opportunities for failure in this statement Eventually Consistent Across DCs 37
  38. 38. @helenaedelson #reactivesummit 38 there is no now… Embrace Asynchrony & Location Transparency …or where
  39. 39. @helenaedelson #reactivesummit There Is No Now 39 US-East-1 MirrorMaker EU-west-1 ZK ZK Akka micro services Akka micro services Akka micro services FiloDB Cassandra Akka micro services Akka micro services Akka micro services Spark Compute Clusters Analytics/Timeseries/ML Storage Storage Clusters S3/ Cold Raw Event Stream Event Stream DC-1 DC-2 Akka Cluster-ed apps, partitioned by Role=service Raw Event Replay Flink Backpressure
  40. 40. @helenaedelson #reactivesummit 40 US-East-1 MirrorMaker EU-west-1 ZK ZK Akka micro services Akka micro services Akka micro services FiloDB Cassandra Akka micro services Akka micro services Akka micro services Spark Compute Clusters Analytics/Timeseries/ML Storage Storage Clusters S3/ Cold Raw Event Stream Raw Event Replay Event Stream But I Can Replay Future[Then] DC-1 DC-2 Akka Cluster-ed apps, partitioned by Role=service Backpressure
  41. 41. @helenaedelson #reactivesummit Replaying Data from Time (T) General time - We updated some compute algorithms which run on raw and aggregated data. I need to replay some datasets against the updated algos Specific time - A node crashed or there was a network partitioning event and we lost data from T1 to T2. I need to replay from T1 to T2 without having to de-dupe data 41
  42. 42. @helenaedelson #reactivesummit Cassandra Bakes Clustered Ordering Into Data Model CREATE TABLE timeseries_t ( id text, year int, month int, day int, hour int, PRIMARY KEY ((wsid), year, month, day, hour) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); 42 Now the read path queries automatically return sorted by most recent. No sorts to do or ordering in your code. Can also easily bucket in cold storage for faster reads.
  43. 43. @helenaedelson #reactivesummit 43 US-East-1 MirrorMaker EU-west-1 ZK ZK Compute Clusters Event Stream Build Apps Not Clusters? App clusters vs Infra clusters Actors & Kafka Streams Actors & Kafka Streams Actors & Kafka Streams Event Stream Event Stream
  44. 44. @helenaedelson #reactivesummit Kafka Streams Key Features Secure stream processing using Kafka security Elastic and highly scalable Fault-tolerant Stateful and stateless computations 44
  45. 45. @helenaedelson #reactivesummit Kafka Streams Key Features Interactive queries Time Model & Windowing Supports late-arriving and out-of-order data Millisecond processing latency At-least-once processing guarantees exactly-once is in progress :) 45
  46. 46. @helenaedelson #reactivesummit val builder = new KStreamBuilder()
 // Consume from topic x in the stream, from any or one DC val kstream: KStream[K,V] = builder.stream(des, des, “fu.input.topic") // Do some analytics computations // Publish to subscribers in all DCs in the stream kstream.to(“serviceA.questionB.aggregC.topic”, …) // Start the stream val streams = new KafkaStreams(builder, props) streams.start() // Do some more work then close the stream streams.close() Kafka Streams: KStream 46
  47. 47. @helenaedelson #reactivesummit 47 Kafka Streams: KTable val locations: KTable[UserId, Location] = builder.table(“user-locations-topic”) val userPrefs: KTable[UserId, Prefs] = builder.table(“user-preferences-topic”) // Join detailed info from 2 streams as events stream in val userProfiles: KTable[UserId, UserProfile] =
 locations.join(userPrefs, (loc, prefs) -> new UserProfile(loc, prefs)) // Compute statistics val usersPerRegion: KTable[UserId, Long] = userProfiles .filter((userId, profile) -> profile.age < 30) .groupBy((userId, profile) -> profile.location) .count()
  48. 48. @helenaedelson #reactivesummit val streams = new KafkaStreams(builder, props) streams.start() // Called when a stream thread abruptly terminates // due to an uncaught exception. streams.setUncaughtExceptionHandler(myErrorHandler) Kafka Streams DSL Basics 48
  49. 49. @helenaedelson #reactivesummit 49 Integrating Intelligence For Fast Data and Feedback Loops
  50. 50. @helenaedelson #reactivesummit Translation Layer 50 US-East-1 MirrorMaker EU-west-1 PubSub Apps PubSub Apps PubSub Apps Compute Clusters Event Stream DC-1 Raw Data Cassandra Spark SQL Spark MLLib Spark Streaming Kafka
  51. 51. @helenaedelson #reactivesummit Translation Layer 51 US-East-1 MirrorMaker EU-west-1 PubSub Apps PubSub Apps PubSub Apps Compute Clusters Event Stream DC-1 Raw Data Cassandra To Columnar Format Reads chunks, translates to spark rows Spark MLLib Spark SQL Spark Streaming Kafka FiloDB
  52. 52. @helenaedelson #reactivesummit Translation Layer 52 US-East-1 MirrorMaker EU-west-1 PubSub Apps PubSub Apps PubSub Apps Compute Clusters Event Stream DC-1 Raw Data Cassandra To Columnar Format Reads chunks, translates to spark rows Spark MLLib Spark SQL Spark Streaming Kafka FiloDB Faster Ad Hoc Querying Faster ML Feedback Instead of slowing things down, it makes it faster.
  53. 53. @helenaedelson #reactivesummit 53 Your Akka Cluster got in my Reactive Distributed Analytics Database! Walking with a jar of PB as one does…
  54. 54. @helenaedelson #reactivesummit FiloDB: Reactive Scala & SBT Spark (also written in scala) Akka actors & Akka Cluster for Coordinators Futures for IO Typesafe Config 54
  55. 55. @helenaedelson #reactivesummit import filodb.spark._ KafkaUtils.createDirectStream[..](..) .map(_._2) .map(FuEvent(_)) .foreachRDD { rdd => sql.insertIntoFilo(rdd.toDF, “fu_events”, database = Some(“org1Keyspace”))} 55 FiloDB API: Append only, to existing dataset Write: Kafka, Spark Streaming, Spark SQL & FiloDB
  56. 56. @helenaedelson #reactivesummit Read: FiloDB & Spark SQL 56 import filodb.spark._ val df1 = sqlContext.read.format(“filodb.spark”) .option(“dataset”, “aggrTable”) .load() val df2 = sqlContext.filoDataset(“aggrTable”, database = Some(“ks2”)) More typesafe than the dataframe read (df1)
  57. 57. @helenaedelson #reactivesummit 57 Determine Train the model Train on historic data Score Based on Objectives Proactive Responses React in real time Machine Learning
  58. 58. @helenaedelson #reactivesummit What do I want to predict & what can I learn from the data? Which attributes in datasets are valuable predictors? Which algorithms will best uncover valuable patterns? 58 Optimization & Predictive Models
  59. 59. @helenaedelson #reactivesummit 59 I need fast ML feedback loops, what tools should I pick? Depends…
  60. 60. @helenaedelson #reactivesummit Read from stored training data into MLLib 60 import org.apache.spark.mllib.stat.Statistics import filodb.spark._ val df = sqlContext.filoDataset(“table”, database = Some(“ks”)) val seriesX = df.select(“NumX”).map(_.getInt(0).toDouble) val seriesY = df.select(“NumY”).map(_.getInt(0).toDouble) val correlation: Double = Statistics.corr(seriesX, seriesY, “pearson”) Calculate correlation between multiple series of data
  61. 61. @helenaedelson #reactivesummit Read from the stream to MLLib & store for later work in FiloDB 61 val stream = KafkaUtils.createDirectStream[..](..) .map(transformFunc) .map(LabeledPoint.parse) val trainingData = sqlContext.filoDataset(“trainings_fu") val model = new StreamingLinearRegressionWithSGD() .setInitialWeights(Vectors.dense(weights)) .setStepSize(0.2).setNumIterations(25) .trainOn(trainingData) model .predictOnValues(stream.map(lp => (lp.label, lp.features))) .insertIntoFilo(“predictions_fu")
  62. 62. @helenaedelson #reactivesummit 62 Parting Thoughts
  63. 63. @helenaedelson #reactivesummit Don’t take when someone says X doesn’t work at face value. It may not have been the right choice for their use case They may not have configured/ deployed/wrote code for it properly for their load All technologies are optimized for a set of things vs every thing Invest in R&D 63 Cultivate Skepticism
  64. 64. @helenaedelson #reactivesummit Modular does one thing Decoupled knows one thing Reusable & Extendable when not final or private for a reason 64 Keep Everything Separate
  65. 65. @helenaedelson #reactivesummit 65 Did you just duplicate codez?
  66. 66. @helenaedelson #reactivesummit 66 Versus duplicating, with disparate strategies, all over your teams & orgs Build A Reactive / Proactive Failure & Chaos Intelligence System for platform infrastructure
  67. 67. @helenaedelson #reactivesummit Byzantine Fault Tolerance? 67 Looks like I'll miss standup
  68. 68. @helenaedelson #reactivesummit twitter.com/helenaedelson slideshare.net/helenaedelson github.com/helena Thanks!

×