1.
From 0 to DevOps in 80 days
Lessons learnt from shifting an on-prem to a cloud culture
Bernd Greifeneder, CTO
http://dynatrace.com/trial

2.
From the DevOps Webinar with Gene & Mark
Mark Tomlinson
Performance Sherpa
@mark_on_task
Andi Grabner
Performance Advocate
@grabnerandi
Gene Kim, CTO
Researcher and Author
@RealGeneKim

3.
High Performers Are …
200x 2,555x
more frequent deployments faster lead times than their peers
Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report
More Agile
3x 24x
lower change failure rate faster Mean Time to Recover
More Reliable

4.
@RealGeneKim
24 “Features in a Box” Ship the whole box!
Very late feedback 
@grabnerandi

5.
@RealGeneKim
Continuous Innovation and Optimization
1 “Feature at a Time”
Optimize Before DeployImmediate Customer Feedback
@grabnerandi

6.
2011: APM about to be disrupted!
• Migrate from On-Prem to VM, Cloud, Containers and PaaS
• Architectures include micro-services, on-demand scaling, self-healing
• ”Cloud Natives“ demand SaaS based solutions
• Bridging the gap between ”New Stack“ and “Enterprise Stack“
• Digital Transformers demand Analytics for Biz, Dev, Ops & Sec
• Many new players on the market
@grabnerandi

7.
From 0 to DevOps in 80 days
Lessons learnt from shifting an on-prem to a cloud culture
Bernd Greifeneder, CTO
http://dynatrace.com/trial

8.
2 major releases/year
customers deploy & operate on-prem
26 major releases/year
170 prod deployments/day
self-service online sales
SaaS
&
Managed
On-Prem
2011 2016

9.
ManualCICDFcontinuous
integration
continuous delivery & deployment
and feedback

10.
NOC lessons learntContinuous Integration faster than classic Ops
Automation and APIs
One delivery stack across the pipeline

11.
DevOps
Continuous Delivery
is a culture that emphasizes
the collaboration of various teams
involved in software delivery
is an approach about building, testing
and releasing software
reliable, faster and more frequent

12.
Utmost goal: minimize cycle time
timefeature cycle time
minimize Users

13.
This is where you
create value!
feature cycle time time
minimize Users
Utmost goal: minimize cycle time

14.
Deploy faster!!

15.
Fail
faster??

16.
Right focus?!

17.
The “War Room”
Facebook – December 2012

18.
ruxit theme 2014.05.15Production facts – Oct 2016
450
AWS EC2 instances
>2 years
value to customers in production
99.998%cluster availability since June’15 NO
24/7 OPs Team
170
deployments per working day
2 weeks
release cycle

19.
20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Shift-Left Quality
Quality/Performance matters in Dev/Staging as well!
Make Dev/CSA/PM dependent from Quality in trunk!

20.
Sure glad the hole
isn’t at our end.

21.
Confidential, Dynatrace, LLC
http://dynatrace.github.io/ufo

22.
ruxit theme 2014.05.15
s

23.
ruxit theme 2014.05.15
Make also architects responsible for production

24.
26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
iPhone 6
failing early
improves
quality
late feedback
sucks

25.
27 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Shift-Left Quality
Quality/Performance matters in Dev/Staging as well!
Make Dev/CSA/PM dependent from Quality in trunk!
DevOps = start thinking like an Ops before Commit

26.
28 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Shift-Left Quality
Quality/Performance matters in Dev/Staging as well!
Make Dev/CSA/PM dependent from Quality in trunk!
DevOps = start thinking like an Ops before Commit
Shift-Right Metrics
enable DEVs defining quality metrics
make DEVs to the primary consumers of their metrics

27.
29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Developer will never do that!
Operator’s job

28.
DevOpsNoOps
From 0 to
&
Continuous Delivery
in 80 days

29.
ruxit theme 2014.05.15
call our DevOps
NoOps

30.
ruxit theme 2014.05.15Automation high availability global
Global AWS Infrastructure

31.
ruxit theme 2014.05.15
The rise of the chaos monkeys

32.
ruxit theme 2014.05.15

33.
35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Daily deployments from trunk the whole DEV team depends on

34.
36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
believe in the mission impossible
6months
major/minor release
+ intermediate fix-packs
+ weeks to months
rollout delay
sprint releases (continuous-delivery)
1h : code to production

35.
New “multimodal” dynamics
Apps of
innovation
Apps of records
Apps of Engagement
Apps of
Engagement
Explore new revenue models
Look ahead, see beyond present. Non-IT led.
New hyper-scale, hyper-dynamic apps.
Public cloud and open source bias
New cloud platforms and micro-services stacks.
Decisions made in hours/days, no POC.
DevOps Assumed
Expect release cycles measured in hours.
Tight teaming between biz, dev, ops.

36.
ruxit theme 2014.05.15

37.
Thank you!
Food for thoughts:
• In which of the following stages are you and what's next?
• Classic siloed, Continuous Integration, DevOps?
• Is technology or process/culture the hurdle?
• Do you have the right monitoring strategy?
http://dynatrace.com/trial

38.
Confidential, Dynatrace LLC
Performance management for the digital customer age

39.
DevOps Tooling @

40.
Dev Stage Daily
Continuous Delivery & Feedback
Production Stage
Bi-Weekly & On Demand
Acceptance Stage Bi-Daily
Deploy
Develop &
Fixing
Acceptance &
Performance
& Load Tests
& Monitor
Deploy
Fixing
Acceptance &
Performance
& Load Tests
& Monitor
Deploy
Hotfixing
Monitor
Release
Every 2 weeks version is pushed to next stage = 2 weeks release cycle
Unit+Integration
testing & Build
Unit+Integration
testing & Build
Unit+Integration
testing & Build
SaaS
Week 1 Week 2 Week 3 Week 4

41.
Continuous Delivery & Feedback
Dev Stage Daily
Production Stage
Monthly & On Demand
Acceptance Stage Bi-Daily
Week 1 Week 2 Release
Every 4 weeks version is pushed to next stage = 4 weeks release cycle
Managed - On Premises
Deploy
Develop &
Fixing
Acceptance &
Performance
& Load Tests
& Monitor
Deploy
Fixing
Acceptance &
Performance
& Load Tests
& Monitor
Deploy
Hotfixing
MonitorUnit+Integration
testing & Build
Unit+Integration
testing & Build
Unit+Integration
testing & Build
Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

42.
CDF Tooling Chain - Dev Stage
Acceptance Tests
Selenium/Appium Automated Functional Tests
on UI Level
Automated E2E Tests on UI Level
Browser Compatibility Testing
(Chrome, IE, FF, Edge, Safari)
24/7 Deployment Checks
Jira Manual Regression Testing
AWS, VMware, VirtualBox Test Environments
Develop
Gradle + Artifactory build/deployment automation
SVN + Git Version Control
Eclipse / IntelliJ IDEA
Quickbuild + Jenkins Continuous Integration
Unit & Integration Tests (fully automated)
Unit Tests +Integration Tests
Quickbuild, Eclipse/IntelliJ IDEA for local
execution before commit
Memory/CPU Overhead tests
Dynatrace AppMon
Code Quality and Coverage Checks
Sonar
Memory leak detection
Bullseye, Valgrind
Virus Scan Kaspersky
Open Source License compliance checks
Blackduck
Facts/Numbers
28.000 Unit Tests + 3.000 integration test executions / hour
Deployment (fully automated)
Quickbuild Automated daily deployment of Trunk Builds
using Ansible and Puppet or calling
Dynatrace Ruxit CloudControl, that uses
AWS Cloud Formation
(whole infrastructure as code!)
Selenium/Appium
Automated Customer-Like Deployment
Deployment Checks on UI Level
Performance & Load Tests
Cluster Workload Simulator
Eclipse Mem Analyzer (MAT) + Eclipse Thread Dump
Analyzer
Java Flightrecorder
Monitoring
Dynatrace Real-User, Service and
Infrastructure monitoring
WebChecks, AWS Monitoring, Log Analytics
& Monitoring
OpsGenie and common HipChat Room to
escalate detected Problems directly to
Development
Security 24/7 OSSEC Host intrusion monitoring
Deploy
Develop &
Fixing
Unit+Integration
testing & Build
Acceptance &
Performance
& Load Tests
& Monitor

43.
CDF Tooling Chain - Acceptance Stage
Deploy
Fixing
Unit+Integration
testing & Build
Acceptance &
Performance
& Load Tests
& Monitor
Facts/Numbers (Dev + Acceptance Stage)
~700 Automated UI Tests
~60 hours UI Test execution per Build  ~20
parallel running test sets executed on ~30
execution machines up to ~5 hours per test set
~15 different OS (Windows, Linux)
Unit & Integration Tests (fully automated)
Unit Tests +Integration Tests
Quickbuild, Eclipse/IntelliJ IDEA for local
execution before commit
Virus Scan Kaspersky
Deployment (fully automated)
Quickbuild Automated deployment of Sprint Builds
using Ansible and Puppet or calling
Dynatrace Ruxit CloudControl, that uses
AWS Cloud Formation
(whole infrastructure as code!)
Selenium/Appium
Automated Customer-Like Deployment
Deployment Checks on UI Level
Acceptance Tests
Selenium/Appium Automated Functional Tests
on UI Level
Automated E2E Tests on UI Level
Browser Compatibility Testing
(Chrome, IE, FF, Edge, Safari)
24/7 Deployment Checks
Jira Manual Regression Testing
AWS, VMware, VirtualBox Test Environments
Fixing
Gradle + Artifactory build/deployment automation
SVN + Git Version Control
Eclipse / IntelliJ IDEA
Quickbuild + Jenkins Continuous Integration
Performance & Load Tests
Cluster Workload Simulator
Eclipse Mem Analyzer (MAT) + Thread Dump Analyzer
Java Flightrecorder
Monitoring
Dynatrace Real-User, Service and
Infrastructure monitoring
WebChecks, AWS Monitoring, Log Analytics
& Monitoring
OpsGenie and common HipChat Room to escalate
detected Problems directly to Development
Security 24/7 OSSEC Host intrusion monitoring

44.
CDF Tooling Chain - Production Stage
Deploy
Hotfixing
Unit+Integration
testing & Build
Monitor
Fixing
Gradle + Artifactory build/deployment automation
SVN + Git Version Control
Eclipse / IntelliJ IDEA
Quickbuild + Jenkins Continuous Integration
Monitoring
Dynatrace Real-User, Service and
Infrastructure monitoring
WebChecks, AWS Monitoring, Log Analytics
& Monitoring
OpsGenie and common HipChat Room to escalate
detected Problems directly to Development
Security 24/7 OSSEC Host intrusion monitoring
Monthly and on demand Vulnerability Scans
(KPMG Linz)
Closed Bug Bounty Program at HackerOne
Unit & Integration Tests (fully automated)
Unit Tests +Integration Tests
Quickbuild, Eclipse/IntelliJ IDEA for local
execution before commit
Virus Scan Kaspersky
Deployment (fully automated)
Quickbuild Automated deployment of Sprint Builds
using Ansible and Puppet or calling
Dynatrace Ruxit CloudControl, that uses
AWS Cloud Formation
(whole infrastructure as code!)
Selenium/Appium
Automated Customer-Like Deployment
Deployment Checks on UI Level