EmpireNode 2014 talk about the route towards cloud automation. How do we work towards a self-healing cloud

1. THE ROUTE TOWARDS
CLOUD AUTOMATION

2. SIMPLY EVERY ENGINEER’S
DESIRE TO USE THE FORCE
THE ROUTE TOWARDS
CLOUD AUTOMATION

3. SO LIKE ...
... CLOUD ... ... AUTOMATION

4. And why you should care Who am I?
Automate the Cloud at
On the internet as..
@jcrugzz

5. and why this matters What is Nodejitsu?
We execute
untrusted user code
in the cloud using a system of..
Microservices

6. Words that I like Vocabulary
Determinism
Computers act consistently
Non-determinism
Computers begin to be affected by
Quantum properties and begin to act
inconsistently.

7. C LOUD AUTOMATION FOR
MICROSERVICES

8. From one we have many Microservices and SOA
Monolithic Web Service Collection of
Microservices

9. From one we have many Microservices and SOA
MICROSERVICES
AND SOA TURN
HARD COUPLING
INTO LOOSE(R)
Monolithic Web Service Collection of
Microservices
COUPLING

10. O NE DOES NOT SIMPLY
S C ALE ... ANYTHING

11. Patterns for scaling service architectures Microservices and SOA
Monolithic Web Service
Our only choice is to replicate the
WHOLE SERVICE AGAIN ... ... AND AGAIN ... ... AND AGAIN

12. Patterns for scaling service architectures Microservices and SOA
Microservices and SOA
` When you have them separated.. You can
scale ... EACH ... ... ONE ... ... INDIVIDUALLY

13. YO DAWG I HEARD YOU LIKE MICROSERVICES
SO I PUT A GEOENCODING SERVICE IN A QUEUE SERVICE IN A
FACEBOOK SERVICE IN A TWITTER SERVICE SO YOU CAN USE
MICROSERVICES WHILE YOU USE MICROSERVICES

14. YO DAWG I HEARD YOU LIKE MICROSERVICES
DON’T GET
CARRIED AWAY
SO I PUT A GEOENCODING SERVICE IN A QUEUE SERVICE IN A
FACEBOOK SERVICE IN A TWITTER SERVICE SO YOU CAN USE
MICROSERVICES WHILE YOU USE MICROSERVICES

15. Unix philosophy for software services Microservices and SOA
Unix Philosophy
Do one thing and one thing well Modularity and reusability Streams! SIMPLICTY!
"We should have some ways of connecting programs like
garden hose--screw in another segment when it becomes
necessary to massage data in another way. This is the way of
IO also.”
!
—Doug McIlroy. October 11, 1964

16. Unix philosophy for software services Microservices and SOA
Unix Philosophy
Do one thing and one thing well Modularity and reusability Streams! SIMPLICTY!
"We should have some ways of connecting programs like
garden hose--screw in another segment when it becomes
necessary to massage data in another way. This is the way of
IO also.”
!
—Doug McIlroy. October 11, 1964

17. It’s kind of like “small modules” Microservices and SOA
Node and Microservices
They go hand in hand Products not projects - Martin Fowler Think MODULE === Microservice API Simplicity

18. Problems and illusions Microservices and SOA
Illusions?
Silver Bullet Better Availability Easier Scaling Eliminates SPOF
`
Problems
You now have N number of SPOF More moving parts Increased surface area for failure Overall increase in non-determinism

19. Problems and illusions Microservices and SOA
Illusions?
Silver Bullet Better Availability Easier Scaling Eliminates SPOF
BUT ITS ALL OK!
!
Everything has `
tradeoffs
Problems
You now have !
N number of SPOF More moving parts Increased surface area for failure Overall increase in non-determinism
Nothing is Perfect

20. SAY MICROSERVICES
ONE MORE TIME!

21. SAY MICROSERVICES
BUT
SERIOUSLY...
ONE MORE TIME!

22. Lets Make things Complicated
Because I said so

23. How many services do we have?! DevOps Mania
When things go wrong
` What happens when services go down..
ONE ... ... BY ... ... ONE

25. YES. YES IT IS

26. CALM DOWN
WE HAVE MONITORING

27. Get all the data Monitoring: Collect Metrics
What do we collect?
ALL THE METRICS!
Logs
Memory
Requests
Apps
Metrics!
Cluster

28. W h at if we were generating data
and not using it at all?

29. What are we looking at Monitoring. Next steps
Assess the data
Metrics!
Cluster
Strategies
Store the data
Graphs are your friend
Apps
Find trends and bridge correlations
Data Store

31. “As intelligence goes up,
happiness goes down. See, I made
a graph. I make lots of graphs.”
- Lisa Simpson

32. “As intelligence goes up,
happiness goes down. See, I made
a graph. I make lots of graphs.”
- Lisa Simpson

33. Act Now!
“As intelligence goes up,
happiness goes down. See, I made
a graph. I make lots of graphs.”
- Lisa Simpson

34. Do something! Monitoring: Now what?
Metrics!
Cluster
Take Action
Actions to take
Fix something
Scale up/down
Apps
Remove server from pool
Data Store
“Action” Services

35. BRACE YOURSELVES
IT’S STORY TIME

36. Lets fix things SSH Micro-service
Metrics!
Cluster
An action
It is all in the simplicity
Identified problem
Crafted a solution
Put together the pieces to fix it
SSH Service
Load Balancer
Dead App
Fix it

37. FIX IT! FIX IT! FIX IT!

38. Welcome to distributed systems What’s the problem?
Metrics!
Cluster
Not distributed
Simplicity can lack flexibility
No built in fault tolerance to service
Uses a local queue that doesn’t scale
But this is still all fixable
SSH Service
Load Balancer
Dead App
Fix it

39. Distributed Systems
How do they work?

40. Use a distributed approach How do we fix this?
Metrics!
Cluster
Make distributed
Scale with a particular approach
SSH Service
Load Balancer
Dead App
Fix it

41. Use a distributed approach How do we fix this?
Metrics!
Cluster
Make distributed
Scale with a particular approach
Use a proven distributed protocol (raft)
Use a working module or implement it
Read the white paper and test
SSH Service
Load Balancer
Dead App
Fix it

42. Leader Election Raft: Distrubuted Consensus
What is Raft?
A consensus Algorithm
Log replication
Leaders, Followers, Candidates
Leader election and failover

43. Reading white papers

44. Papers Distributed Systems
Reading?
Yes we should read them
Yes they are hard
Reading once is just the beginning
Consider it like stretching

45. OMG
B UT IM OVERWHELMED

46. Take a deep breath Distributed Systems
RELAX
SIMPLICITY
Start small
Importance of modules
Everything in stages

47. Demo Architecture

48. github.com/jcrugzz/empirenode-demo
DOGE FOR DEMO

49. THE END
Jarrett Cruger - @jcrugzz