This presentation was given at TopConf Tallinn in May 2015. Title: Driving the DevOps feedback loop in the Enterprise Description (brief overview for marketing purposes, max. length 400 characters-about 65 words): DevOps requires feedback loops backed by consistent data, within enterprises many layers of tools make visibility a challenge. This session outlines modernizing monitoring and providing the right data collection capabilities to support agile application operations. This session will outline the use of open source and commercial software solutions which fit startups to the enterprise. Abstract (Longer, more detailed description (3-6 paragraphs, bullet points welcome) of your presentation to help the program committee understand what you will cover. Please keep in mind that if your proposal is chosen, this abstract will also appear on the website to help conference attendees decide if it's right for them.): Enterprises do not have the benefit of starting from scratch, they must implement DevOps in an existing environment, often managing heritage investments along with fast moving projects. The monitoring tools unfortunately fall into the same category. These complex, dated, and costly monitoring tools don’t meet today’s needs in providing the visibility required for agile development that leverage continuous delivery and DevOps. Not only are the tools an issue, but the visibility and approaches are also problematic. The use of open source tools has been the de facto approach for DevOps, but this approach comes with it’s own set of challenges in terms of managing a complex stack of tools with varying quality of support/community. The move to a metrics-driven feedback loop enables teams to act fast at any stage of the product lifecycle. We will outline common technology stacks and approaches using graphite, statsd, and collectd along with nagios, munin, and other system monitoring tools. We will also outline commercial solutions, and how these tools tend to fall into silos as well. Monitoring must be reformed with new platforms and technologies, fitting specific design patterns which meet the needs of DevOps teams, these will be outlined and explained from a requirements perspective, providing a roadmap for how to apply these learnings inside your company. Join this session to take back an evolved approach to unified monitoring and how you can get your company to be metrics driven Attendees of this session will walk away with a clear understanding of: Understanding today’s tooling limitations Open source tool sprawl issues Fundamental patterns of monitoring How these will be solved in the future

1. DevOps Monitoring
Feedback Loops in Enterprise Environments
May 12th, 2015
Jonah Kowall, VP Market Development and Insights

2. Agenda
1. What is Changing?
2. Why do we need to monitor?
3. How do we monitor?
4. What are the best practices in monitoring?
5. Why does monitoring suck?
6. How to create business context in monitoring?
Copyright © 2015 AppDynamics. All rights reserved. 2

3. Applications are Transforming
Conventional Enterprise Cloud "Native" Pattern
Adapted From Cloud Architecture Tutorial by Adrian Cockcroft (Netflix)
Central SQL Database
Sticky In-memory Session
Chatty Protocols
Tangled Service Interfaces
Polled Information
Fat Complex Objects
Components as Jar Files
Distributed Key/Value NoSQL
Latency Tolerant Protocols
Event-driven
Lightweight Serializable Objects
Components as Services
Layered Service Interfaces
Shared Memcached/Redis Session
Java, .NET JavaScript, Python, Ruby, node.js
Copyright © 2015 AppDynamics. All rights reserved. 3

4. Generic Feedback Loop
Copyright © 2015 AppDynamics. All rights reserved. 4
Change
or
Correct
MeasureAnalyze

5. User Feedback Loop
Copyright © 2015 AppDynamics. All rights reserved. 5

6. DevOps Feedback Loop
Copyright © 2015 AppDynamics. All rights reserved. 6
Develop
Test
DeployMonitor
Analyze

7. Measurement: Push vs Pull
Both are essential and scalable
• Push
• Easier to manage since new instances begin sending data
• Real-time streaming of metrics/data
• Monitoring system can have stale or otherwise
disconnected data
• Must have centralized configuration management
• Pull
• Centralized management of polls or requests for data
• Must build specific infrastructure to scale polling
Copyright © 2015 AppDynamics. All rights reserved. 7

8. Measurement: Interrogation
Request a metric
Relies upon another device or manufacturer
Often an API
Copyright © 2015 AppDynamics. All rights reserved. 8

9. Measurement: Observation
Inspect transaction/conversation
Agent – APM
Device - Network Capture (NPM)
Network
Application
Packet and Flow
Transaction
Code Instrumentation
OR
Generate, Gather, and Analyze/Parse Logs
Copyright © 2015 AppDynamics. All rights reserved. 9

10. How Is This Done
Ops - Out of the box instrumentation
Infrastructure
Application Components
Transactions
Developers
Custom Instrumentation
Metrics, Logs
Copyright © 2015 AppDynamics. All rights reserved. 10

11. Overhead of Monitoring
Often ignored, even logs have an overhead, not just APM tools
Overhead impacts end user experience
Most do not measure end user experience, must measure with Real
User Monitoring
Can verify impact of monitoring based on load testing or real users
Open Source RUM : Boomerang
Commercial tools : AppDynamics or other APM products
Copyright © 2015 AppDynamics. All rights reserved. 11

12. Not Just About the Application: Must Understand the End User
Know Your Fans!
Copyright © 2015 AppDynamics. All rights reserved. 12

13. We Have Data Now What?
Alerting
Calculated rate of change
Never use a threshold
Anomaly detection improving
Analytics
Mostly reporting today, needs to change
with Machine Learning
Copyright © 2015 AppDynamics. All rights reserved. 13

14. Never Store Rates or Calculated Values
Copyright © 2015 AppDynamics. All rights reserved. 14

15. Too Many Graphs, Too Much Time Wasted
Typical NOC, inefficient.
Lots of screens and data.
Too many email alerts.
Alert on what matters for
end-user experience, otherwise
handle component or redundant
outages without notification.
Very primitive, cobbled together, custom built solutions:
• Nagios, Zabbix, or others doing alerting.
• Graphite dashboards.
• StatsD custom metrics.
• collectd service/system metrics.
• Elasticsearch, Logstash and Kibana (ELK) for logs.
Copyright © 2015 AppDynamics. All rights reserved. 15

16. New Web-scale Process Requirements:
Deployment and Monitoring Are Now Linked
Deployment
Monitoring
Continuous Delivery
Source: http://www.flickr.com/photos/yandle/4337747398
Copyright © 2015 AppDynamics. All rights reserved. 16

17. Do It Yourself: Heavy Commitment and Integration
Graphitestatsd
collectd
Graphsky
Descartes
Tasseo
Giraffe
Graphene
Orion
Copyright © 2015 AppDynamics. All rights reserved. 17

18. Why does Monitoring Still Suck?
Common advanced stack is completely component based:
• statsD + collectd -> Graphite (plus other visualizers)
• Nagios or Zabbix
• ELK (ElasticSearch, Logstash, Kibana)
Lack of Context or Relationships
• No topology awareness
• No transactional visibility
• No end user metrics unless you code your own
• No event suppression or management
Copyright © 2015 AppDynamics. All rights reserved. 18

19. What Should I Monitor?
Server CPU, Memory, Network?
Capacity? Utilization? Throughput?
Throughput is a rate, don’t measure that 
If your business is selling server CPU, Memory, and network, yes, but most are not
Copyright © 2015 AppDynamics. All rights reserved. 19

20. Up Level the Conversation
Capture business transactions!
How? (APM or Custom Instrumentation)
Assume you are a retail bank, you don’t just monitor the amount of money being
deposited?
Monitor if your customers can deposit money and are depositing money
Is this a rate?
Not if you store it as each transaction and analyze/display it as a rate.
Copyright © 2015 AppDynamics. All rights reserved. 20

21. Context is King: Unified Monitoring
Copyright © 2015 AppDynamics. All rights reserved. 21
Application
PERFORMANCE
Infrastructure
CAPACITY%
End User
EXPERIENCE
Business
REVENUE
Mobile
CRASH
Machine data
LOGS
Code
DIAGNOSTICS
Database
PERFORMANCE
Real user
MONITORING

22. Buy it Already Integrated
Analytics
• Visualization
• Insight into data (ex: root cause, SLA violations)
Language Support
• Java shop?
• You will have more if you don’t already today
Application Stack Support
• app server
• Databases
• Data stores
• Cloud services
Deployment Flexibility
• On premises maybe today
• SaaS possibly in the future
Copyright © 2015 AppDynamics. All rights reserved. 22

23. It's Ultimately About Understanding Your Customers
"If you're not looking at your data (in
its rawest possible form), then you
don't understand your business and
you almost certainly don't understand
your customers"
— John Rauser (Amazon)
Copyright © 2015 AppDynamics. All rights reserved. 23

24. Thank You