1.
William Yeh
DevOps Summit 2016 (2016-07-06)
Monitoring
Monitoring: a Process Perspective

2.
CRT (Current Reality Tree)
#6
#10
( )
best practice
#3 #5
( ) ( )( )
AND
AND
AND
AND
#1#2
#4
#7
#8
#9
#11
Murphy exists
AND
AND
×
AND
DevOps
AND
AND
AND
AND
AND
AND
AND
AND
Op
AND
http://www.slideshare.net/williamyeh/devops-63711710

3.
#3
#1#2
#7
#8
#9
#11
Murphy exists
AND
AND
AND
AND
AND
AND

4.
#1#2
#4
#8
#9
Murphy exists
AND
AND
×
AND
AND
AND

5.
#7
#9 #1
#2
…
#11
…
#8 #4

6.
Risk management
• Threats
• avoid
• transfer
• mitigate
7
• Opportunities
• exploit
• enhance
• share
👍👎
http://www.slideshare.net/williamyeh/whoscall-realtime-monitoring

7.
William Yeh
DevOps Summit 2016 (2016-07-06)
Monitoring
Monitoring: a Process Perspective

8.
Process Monitoring
Monitoring

9.
Process
?
?
?
?
Monitoring
Monitoring

10.
?
?
Process
Monitoring

11.
#5
Part 2

12.
Efrat Goldratt-Ashlag

13.
Efrat Goldratt-Ashlag

14.
What to change
To What to change
How to cause the change

15.
CRT (Current Reality Tree)

16.

17.
DevOps

18.
DevOps
leverage
✘
TOC
CCPM
FRT (Future Reality Tree)

19.
DevOps
leverage
✘
TOC
CCPM
FRT (Future Reality Tree)

20.
TOC
CCPM

21.
Stephen R. Covey

22.

23.
What get measured, get done.
Peter Drucker

24.
Policy
What get measured, get done.

25.
Policy

26.
Policy

27.
Policy
Buy-in
Policy

28.
What to change
To What to change
How to cause the change

29.
Adrian Cockcroft

30.
CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
StatsD
BigQuery

31.
CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
StatsD
BigQuery

32.
CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
BigQuery

33.
CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
BigQuery

34.

35.
http://school.soft-arch.net/blog/125009/change-viewpoint-on-lord-of-rings
Lean Change Canvas

36.
Lean Change Canvas
Commitment Wins/Benefits
Urgency
Target State
Success Criteria
Vision
Communication
Action
Change Recipients
FYI: http://kojenchieh.pixnet.net/blog/post/442550432-firstthing_of_agile_promotion
FYI: http://leankit.com/blog/2015/02/lean-change-method/
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
TO DO LIST
details
Augmented

37.
Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1

38.
What to change
To What to change
How to cause the change

39.
Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
Flow
Tech
Monitoring

40.
Buy-in
Flow
Buy-in
Policy

41.
Flow
TOC Lean Thinking
CCPM

42.
TOC

43.
Lean Thinking
Value
Value stream
Flow
Pull
Perfection
http://school.soft-arch.net/blog/115652/devops-a-lean-perspective

44.
“The Three Ways”
Create fast flow of work from Dev into IT Ops.
Shorten and amplify feedback loops.
Create a culture that simultaneously fosters 2
things:
1. continual experimentation, learning from
failure.
2. repetition and practice is the prerequisite
to mastery.
Create fast flow of work
from Dev into IT Ops.
Shorten and amplify
feedback loops.

45.
CCPM
Critical Chain
Project Management

46.
Flow
TOC Lean Thinking
CCPM

47.
VPC
CloudFront ELB API servers DB
Simplified version

48.
CloudFront ELB API servers DB
ELB API servers DB
Microservices
Simplified version

49.
Flow

50.
Flow

51.
Flow

52.
Flow

53.
Overview

54.
Incoming
requests

55.
API
servers

56.
DB servers

57.
DB servers
API
servers
Incoming
requests
Overview
Flow

58.
Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
Flow

59.
TOC
Flow
TOC

60.
Flow
Buy-in
Policy
Tech
Flow

61.
Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
Tech

62.
Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
of server technologies

63.
Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
Runtime dependency
william Ansible

64.
Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
Scalability
Overhead

65.
Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
Node/system metrics exporter
AWS CloudWatch exporter
Blackbox exporter
Collectd exporter
Consul exporter
Graphite exporter
HAProxy exporter
InfluxDB exporter
JMX exporter
Memcached exporter
Mesos task exporter
MySQL server exporter
SNMP exporter
StatsD exporter
cAdvisor
Doorman
Etcd
Kubernetes-Mesos
Kubernetes
RobustIRC
SkyDNS
Weave Flux
Aerospike exporter
Apache exporter
BIG-IP exporter
BIND exporter
Ceph exporter
CouchDB exporter
Django exporter
Google's mtail log data extractor
Heka dashboard exporter
Heka exporter
IoT Edison exporter
Jenkins exporter
knxd exporter
Meteor JS web framework exporter
Minecraft exporter module
Mirth Connect exporter
MongoDB exporter
Munin exporter
New Relic exporter
Nginx metric library
NSQ exporter
OpenWeatherMap exporter
Passenger exporter
PgBouncer exporter
PostgreSQL exporter
PowerDNS exporter
RabbitMQ exporter
RabbitMQ Management Plugin exporter
Rancher exporter
Redis exporter
RethinkDB exporter
rTorrent exporter
scollector exporter
SMTP/Maildir MDA blackbox prober
Speedtest.net exporter
SQL query result set metrics exporter
Ubiquiti UniFi exporter
Varnish exporter
Zookeeper exporter

66.
CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
StatsD
BigQuery

67.
ELB API servers MongoDB
Cloud Manager
CloudWatch

68.
Prometheus vs Graphite/StatsD

69.
abs()
absent()
bottomk()
ceil()
changes()
clamp_max()
clamp_min()
count_scalar()
delta()
deriv()
drop_common_labels()
exp()
floor()
histogram_quantile()
holt_winters()
increase()
irate()
label_replace()
ln()
log2()
log10()
predict_linear()
rate()
resets()
round()
scalar()
sort()
sort_desc()
sqrt()
time()
topk()
vector()
<aggregation>_over_time()

70.
node_cpu
time
number

71.
node_cpu
time
number
{mode="idle"}
mode
node_cpu {mode="irq"}
node_cpu {instance="10.0.37.12"}
{service="web"}
{zone="ap-northest-1a"}

72.
sum(
irate(
node_netstat_TcpExt_TCPTimeWaitOverflow[1m]
)
) by (ec2tag_Service)
counter
gauge
aggregate
TCP Timeout
node_netstat_TcpExt_TCPTimeWaitOverflow[1m]
irate(
node_netstat_TcpExt_TCPTimeWaitOverflow[1m]
)
grouping

73.
gauge
aggregate
Memory Used
1 - node_memory_MemFree/node_memory_MemTotal
grouping
avg(
1 - node_memory_MemFree/node_memory_MemTotal
) by (ec2tag_Service)

74.
avg by (ec2tag_Service) (
irate(
node_cpu{job="node", mode="idle"}[1m]
)
)
counter
gauge
aggregate
CPU Utilization
100 - (
* 100)

75.
avg(
request_time_summary
) by (ec2tag_Service, quantile)
summary
aggregate
Latency
grouping
Customized metrics
with Fluentd plugin for Prometheus

76.
Conclusion

77.
#7
#9 #1
#2
…
#11
…
#8 #4

78.
Policy
Buy-in
Flow
Tech

79.
Policy
Buy-in
Flow
Tech
???
Issue tracking

80.
William Yeh
DevOps Summit 2016 (2016-07-06)
Monitoring
Monitoring: a Process Perspective