Cloud foundry, DevOps, Monitoring, Yaas.io, Visualization, Challenge, Cloud scale

1. 2015 ROBERT BIGOS
Cloud capacity planning &
monitoring challenge
Robert Bigos
bigosr@gmail.com
+48 665-168-240
hCp://www.slideshare.net/RobertBigos
hCps://pl.linkedin.com/in/robertbigos
@bigosr
Blog: bigosr.com

2. 2015 ROBERT BIGOS
2
CloudApps
DevOps culture

3. 2015 ROBERT BIGOS
Requirements…
Func2onal requirement: A want make sex with B

4. 2015 ROBERT BIGOS
Requirements
Func2onal requirement: A want make sex with B
Non-Func2onal: one per day, maybe two, not to short,
not to long, safe but not to safe, 24x7 with 99.9%
monthly availability ,
Decisions: price ? cost ? candles? … , how long ?

5. © 2015 ROBERT BIGOS
Architecture
set of decisions affec-ng
func-onality
…
documented part of communica-on
between stakeholder helping
understand/balance: Func-onal
and Non-Func-onal requirements
…

6. 2015 ROBERT BIGOS
Conflict by design
Dev
Ops
Owner
“…Divide et impera…”

7. 2015 ROBERT BIGOS
SoluPon ?
DevOps DevOpsDevOps

8. 2015 ROBERT BIGOS
Conway's Law
Team A
TEST
STAGE
kingdom
/silo
team B-
X
PROD
“Organiza2ons which design systems are constrained to produce
designs which are copies of the communica2on structures of these
organiza2ons.”

9. 2015 ROBERT BIGOS
2
WHY ?

10. 2015 ROBERT BIGOS
11
Why is so important?
Capacity = fuel, Performance = speed and al2tude
Capacity and performance management helps understand
how quickly and safely you can transport your customers to planned des2na2ons
Source: http://s134.photobucket.com/user/charlesfrith/media/disaster.gif.html

11. 2015 ROBERT BIGOS
Queueing
From operaPonal perspecPve Cloud system use “the
same” formulas to manage capacity of Queue in
Town office
Source: Kanal von FerdinandLutz "Stay in queue" youtube.com
Cloud it is all about cloud scale … everything is
interconnected and instrumented …

12. 2015 ROBERT BIGOS
Monitoring faces
… don’t forget about Conway’s Law …
Logs Metrics AlertsTrendsTresholdsEvents

13. 2015 ROBERT BIGOS
2
Backup plan ?
try:
CtrZ
CmdZ
• blue/green deployment gives possibiliPes not availability
• snapshot is not a backup

14. 2015 ROBERT BIGOS
2
There is no magic buCon
Source: http://make-everything-ok.com/

15. 2015 ROBERT BIGOS
You will fail for sure !
Source: presenter studies for top enterprises in Poland.
Source: http://www.skybrary.aero/index.php/James_Reason_HF_Model Source: Józef Tischner "The Highlander's History of Philosophy"
"the truth,
the whole truth
and
the bullshit truth!
Postmortem is part of DevOps culture.
Recovery has to be part of design.

16. 2015 ROBERT BIGOS
WHAT ?

17. 2015 ROBERT BIGOS
IOT /monitoring/ BigData
Monitoring use tools to collect/process/visualize logs and metrics for
beCer understanding your system to close development and
operaPonal feedback loop. It is not kinder-garden to try nice tools.
Today everything is instrumented and interconnected , be sure you
collect the right data in the right scale to be able get informaPon from
it. Your laptop can generate BigData volumes …
VOLUME VARIETY
VELOCITY

18. 2015 ROBERT BIGOS
32
Source: Łukasz Piskorz IBM SWG Lab
Death or lost signal?

19. 2015 ROBERT BIGOS
32
Computer vs human scale
5 mins = 5*60/10^-9/(60*60*24*365) = 9512 years
few objects, few variables,
no dependency, no relaPons…
Peeping through the
keyhole

20. 2015 ROBERT BIGOS
2
HOW ?

21. 2015 ROBERT BIGOS
Tools
…. …you need tools but tool is not a soluKon…

22. 2015 ROBERT BIGOS
32
Typical “Pme-centric” dashboard
Source: Demo site dashboard grafana.org

23. 2015 ROBERT BIGOS
32
“Threshold violaPon” troubleshooPng
Source: Łukasz Piskorz IBM SWG Lab

24. 2015 ROBERT BIGOS
2
Big
picture ?
try understand , go deeper

25. 2015 ROBERT BIGOS
2
Know unknowns and unknown unknowns
“…Reports that say that something hasn't happened are always
interes2ng to me, because as we know, there are known knowns;
there are things we know we know. We also know there are known
unknowns; that is to say we know there are some things we do not
know. But there are also unknown unknowns -- the ones we don't
know we don't know…”
Donald Rumsfeld, February 12th, 2004 DOD News Briefing
Source: http://www.defense.gov/transcripts/transcript.aspx?transcriptid=2636

26. 2015 ROBERT BIGOS
VisualisaPon RealityGames
hhp://www.wearerealitygames.com/

27. 2015 ROBERT BIGOS
VisualisaPon Cloud Foundry

28. 2015 ROBERT BIGOS
2
Summary

29. 2015 ROBERT BIGOS
2
Lessons learned ?
• suggested approach
• WHY
• WHAT
• HOW
• most popular
• HOW
• HOW
• HOW
…. try understand big picture , go deeper , focus on details

30. 2015 ROBERT BIGOS
2
Lessons learned ?
• There is No Single Version of the Truth … be open for
communicaPon
• There is NO perfect tools … we can build beCer teams and
beCer communicate to close feedback loop
• Cloud scale design , operaPonal design not just easy to
consume…
• There is no GO@FASTER opPon in the Cloud

31. 2015 ROBERT BIGOS
2
Lessons learned ?
• Keep standards: date ISO_8601 example
• Keep what you need … not just everything
• Keep slim: json is nice but …
• Keep informaPon not just numbers/hashtags
• AutomaPzaPon requires standardisaPon
• Microservices architecture assume : team = silo/kingdom.
• Don’t use wrong paCerns!
• If you don't understand your system , your micro service will
not work in scale.
Avoid: designed by programmers for programmers, keep balance

32. 2015 ROBERT BIGOS
never deploy at
Friday
!what ever

33. © 2015 ROBERT BIGOS
If you need more…
There are no infinite resources. There are no perfect resources. Monitoring is
part of a capacity management process which is only a part of operaPons
management. Heading to simplicity of usage we accept complexity of the
soluPon and somePmes blindness hoping that when the Pme comes magic
words will solve all the problems. Hope always dies last and silence ayer
failure gives us a lesson in humility: we have to learn how to interpret
monitoring data as this is engineering not magic. Process is more than just
tools and people and management is inspiraPon and determinaPon to
achieve at least goals that were defined. More failures, less Pme to learn -
this is part of :
”You build it, you run it”
See more :
hCp://www.slideshare.net/RobertBigos
hCps://pl.linkedin.com/in/robertbigos
Blog: bigosr.com