Confidential, Dynatrace, LLC
How to explain DevOps (to somebody non tech)
How it works at Facebook, Paypal …
Andreas Grabner, DevOps Activist @ Dynatrace
@grabnerandi
#1 How I explain DevOps #2 What I learned so far
#3 What our customers do #4 Trends & Final Thoughts
#1 How I explain DevOps …
… to somebody non-technical like …
Ship the whole box!
Quality Control
Back to customer
24 “Features in a Box”
Very late feedback 
F r u s t r a t i o n !
1 “Feature at a Time”
Optimize Before DeployImmediate Customer Feedback
Continuous user-driven innovation
2major releases/year
customers deploy & operate on-prem
26 feature releases/year
500 prod deployments/day
self-service online sales
SaaS & Managed
2011 2017
sprint releases (continuous-delivery)
1h: Code -> Prod6months
major/minor release
THANK YOU
#2 What I learned so far …
Image from https://fanart.tv/movie/241/natural-born-killers/
99.999
Updates Slow & Error Prone
Leads to Fewer Updates & More Processes
More on this from DevOpsDays Salt Lake by John Esser: https://youtu.be/YPImHSjvLJI?list=PLS0-KIJQsgRaZLXGBkN7Ma2aUZ3YORuH4
Image from https://fanart.tv/movie/241/natural-born-killers/
Source: https://www.cbinsights.com/blog/increasingly-crowded-unicorn-club/
Source: https://www.cbinsights.com/blog/disrupting-banking-fintech-startups/
… like Software, Organizations should
be refactored regularly!
To achieve “DevOps Fairy Tale Status” 
Many that try typically realize that …
“In the last week there were
67 deploys
of 496 changes
by 18 people”
Flickr DevBlog, December 17th 2008
@ Amazon: “You build it, you run it”
– Dr. Werner Vogels, CTO - 2006
Features
GorankaBjedov
Canary Releases
Onboarding @ FB
Push Karma
6-8 Week Boot Camp
Release 1st, Optimize 2nd
Success Criteria
Fix-It Ticket
Features
Canary Releases
Push Karma
15.000Engineers
4.000Projects
1Source Code Tree
5.500Commits / Day
75Mio Test Cases / Day
“Automated Tests transformed FEAR into BOREDOM”
– Eran Messeri, Google
DevOps @
Dev operates until proven stable in production
Then hands the project to Ops Team!
1 Monolithic App
6 Weeks to DevOps’ify App
Goal: Cloud Continuous Delivery
Dev take Ops Ownership
#3 What our customers do
Dynatrace enables DevOps: Monitoring as Pipeline
Dev Perf/Test Ops Biz
Faster Innovation with Quality Gates
Faster Acting on Feedback
Unit Perf
Cont. Perf
New Deploy
New Capability
CI CD Remove/Promote
Triage/Optimize
Update Tests
Innovate/Design
$$$
Lower Costs
Happy Users
~6000
AppMon Personal Licenses in 2016
364 Share Your PurePaths in 18 Months
2.5s (48%) spent in ASP.NET
Code Execution -> Mainly NHibernate
Rest of the time spent
executing these queries
Aiglon.ch: N+1 Query Problem 1812!SQL Executions on two
different DB Connections
DevOps Reflected in Product
#1: Click Here!
#2: Look Here!
LAUNCHING Share Your Problem Pattern
Confidential, Dynatrace, LLC
Shift-Left at
50 350 90 749
Regressions
Detected
Confidential, Dynatrace, LLC
Shift-Left at
50 350 90 749
43x RT 0 OOM’s Sync / Wait
Cycle Time
Avg. Bugs per Test
10.5 Days
Oct 2015 Apr 2016
30 Days
2.785.79
SPEED of Delivery
QUALITY of Delivery
~3x
faster
50%
less bugs
Mark Tomlinson
“Performance Sherpa”
Host of www.perfbytes.com
“If you can impact the
[performance of the] code
before it is written, then and
only then, can you consider
your work to be performance
engineering. Otherwise, you
are just doing testing.”
- Jim Duggan, Gartner
Continuous Performance as Self-Service
Continuous Performance @ Dynatrace
“Performance Signature”
for Build Nov 16
“Performance Signature”
for Build Nov 17
“Gold Standard” Reference from Production Test Results from latest Build / Release
Black: Overall application load and peak volume Percentile breakdown of fast, warning, slow txs
VS.
Visual Release Validation
Pablo Alvarez Messina
Architecture & Innovation
IT Manager
Carlos Soneyra
CEO and Co-Founder
DevOps(Infra) @
The “Bad App” List
What’s Next?
“The List”
What’s Next?
“The List”
What’s Next?
Total Number of Users
per User Experience
Conversion Rate
BizOps @ Dynatrace Marketing
New Features + Day # 1 of Mkt Push
Overall increase of Users!
Jump in Conversion Rate!
BizOps @ Dynatrace Marketing
Users keep growing
Increase # of “tolerating” users!
Lower Conversion as Day #1
Day #2 of Marketing Campaign
BizOps @ Dynatrace Marketing
Drop in Conversion Rate
Spikes in FRUSTRATED Users!
Hotfix Deployment was rolled out
BizOps @ Dynatrace Marketing
User Experience Back to Normal
Jump in Conversion Rate!
Fix of the Hotfix was rolled out
BizOps @ Dynatrace Marketing
Dynatrace enables DevOps: Monitoring as Pipeline
Dev Perf/Test Ops Biz
Faster Innovation with Quality Gates
Faster Acting on Feedback
Unit Perf
Cont. Perf
New Deploy
New Capability
CI CD Remove/Promote
Triage/Optimize
Update Tests
Innovate/Design
$$$
Lower Costs
Happy Users
#4 Trends & Final Thoughts
ChatOps: AUTOMATE “Tap on Shoulder Tasks”
“Extend the trial period for user XYZ” -> Marketo
“Who is the sales rep for this account?” -> Salesforce
“How many people use Dynatrace 7.0?” -> eServices
“Run that Build Job and Deploy in Staging!” -> Jenkins
“Turn on that Feature for Tenant ABCDEF” -> Dynatrace
“What’s the Failure Rate on easyTravel?” -> Dynatrace
ChatOps: Automate DevOps via Chat
Coding Bots!
Deploy Bots!
DevOps == Devs on Call?
Rotating Shifts
(Trade Shifts with others)
Incentives
(pay internet at home, parties …)
EVERYONE is on Call
Including CTO
Shadowing Ops
Learning by doing!
?
“You Build It, You Run It!”, Amazon
11 min - Mean Time Between Interruption
25 min - Mean Time To Deep Work
Study “Brain, Interrupted”: http://nyti.ms/1fdkVUT
Foreground and Background Teams
also applies for “Devs On Call”
Andi: “Hey Karolina!! Quick Q!:
Who could help me with technical questions on AWS!!??”
Karolina: “Well … I think I can help you. What exactly do you need?”
2 Weeks Later in Boston – Face 2 Face
Karolina: “Good we finally meet face 2 face.
I really thought you were an a…..”
Andi: “??? WHAT ???”
7% verbal
55%
body language
38%
tone of voice
Why I choose email/chat?
What I loose?
I am in control!
I can edit!
I can end the conversation!
I can do it in a “boring” meeting!
Spontaneity
Connectedness / Relatedness
Unexpected problems
Unexpected solutions
Pick up the Phone or Video Chat vs Email
Managers: Lead by Example, Make time for Talk!
Distributed Teams: bring them together from time 2 time
Design for conversations: Standups, Offsites,
Lunchroom, Open Workspace
‘18
Dynatrace Innovation Day
DevOps is about … … creating more value …
… making money faster …
… by also making your life
easier!
Confidential, Dynatrace, LLC
How to explain DevOps (to somebody non tech)
How it works at Facebook, Paypal …
Andreas Grabner, DevOps Activist @ Dynatrace
@grabnerandi

How to explain DevOps to your mom

Editor's Notes

  • #2 Get your Dynatrace SaaS Trial @ http://bit.ly/dtsaastrial
  • #4 My parents 
  • #5 Here is an analogy for Waterfall as if you were explaining it to a family member or friend who was not technical. Remember back in the day we used to have a camera and would take it to an occasions. If we did not use all the film, a month later, we would take more pictures, and finally send the pictures to be developed. Then the pictures would be developed and come back to us and we would not be happy with a picture we were waiting for, because something was not right, it was not a good picture. And this felt frustrating. The analogy to Waterfall: Putting many features into a single release Ship it to some other entity who does quality control Final product comes back very late -> hard to remember which features / photos we created, and then often we realize its not what we wanted Frustrating
  • #6 Here is an analogy of the new way of delivering software – continuous, in small batches with changes made immediately to bring better results for the user and the experience. Using my girlfriend on a trip and a photo in continuous delivery: The picture is received one at a time Quality control and optimization is in her own hands (thanks to software that is “part of the delivery chain” (the photo application software on her iPhone) She controls what to push into production -> post it on Instagram / Facebook She wants to make her users (friends & family) happy – she is hoping for LIKES! If she gets dislikes she can remove an image. If she gets comments she can take another picture and deploy it within seconds -> that is Continuous User Driven Innovation She has the tools and the power to deliver, and she can quickly take in feedback and redeploy.
  • #7 This is where I then start talking about our own transformation … - BUT NOT TODAY!
  • #8 TODAY I want to start with a big THANK YOU
  • #9 to a lot of people out there that actually allow me to do my job in a much easier way by sharing stories with me
  • #10 We also shared our story on stage at PERFORM – in case you haven’t seen it
  • #11 Which brought me personally and many others of our labs to speak at different events …
  • #12 Just last week I made it to DevOps Days Toronto and we got some phenomenal feedback from the audience
  • #13 And you can see that people are clearly impressed
  • #14 Another big THANK YOU goes to Helmut. I know – you in Linz don’t think the UFO is all that spectacular any longer. But believe me – it is a BIG HIG around the world
  • #15 Andy built his own version of the UFO – doesn’t like like a UFO any longer – but he basically uses it to visualize his the status of his different projects
  • #16 Looks pretty sweet when everything is green 
  • #17 And the folks in Toronto also liked it  - and came up with some new ideas
  • #18 ALL OF THIS – our own transformation story as well as the UFO which visualizes our Cultural Transformation - opened up a lot of conversations with customers and prospects that are all about to go through the same transformation – and they TRUST US because we have gone through the same transformation
  • #20 DevOps has been and still is a HOT TOPIC around the globe! Image from http://roflol.in/earth-then-now-nasa/
  • #21 Especially now within so called “Enterprise” companies
  • #24 The reason why DevOps is such a hot topic for the Enterprise is because of the Natural Born DevOps companies. It all started with these so called “Web Scale IT” companies
  • #25 We also often refer to these companies as Unicorns. But it is not just the Ubers of the world ….
  • #26 We have an explosion of Unicorns in the last couple of years ….
  • #27 And more are yet to come and disrupt traditional industries
  • #28 Banking is one example where a bank that used to cover many different areas – some more profitable – some less profitable – are seeing that these unicorns are eating away their “golden nuggets”
  • #29 This is why we see a lot of traditional companies trying to rethink on who they are – even traditional “enterprise” companies
  • #30 Some are trying it by just mimicking others …
  • #31 The real approach though is that once in a while ….
  • #32 The lessons learned from DevOps started with several stories we heard – for instance ….
  • #33 When do you think this quote was made public?
  • #34 Amazon was also famous for their quote on putting developers in charge by “You Build it, You Run it!”
  • #35 I was in the fortunate situation to meet Goranka Bjedov, Performance Engineer at Facebook – she shared a lot of stories with me while I met her at WOPR this year. If you want to learn more I recommend listening in to our podcasts with her
  • #36 You get assigned to a project. You will not be able to pick the product or technology that you are most familiar with but something that is totally new for you. With that the learning curve is much better. You will Fix well known bugs and work on support tickets. This allows you to get familiar with the process at Facebook and you also get to know the teams.
  • #37 Move Fast and Break Early! Stop working on things that don’t matter Quality of Pipeline takes away the fear of code commit
  • #38 A/B Testing with New Zealand! - https://dzone.com/articles/pushing-twice-daily-our
  • #39 Developers have a Push Karma which can go down if they push something bad! Every dev starts with 4 stars - https://www.facebook.com/notes/facebook-engineering/release-engineering-and-push-karma-chuck-rossi/10150660826788920/
  • #40 Google Web Server Team used to be the “dumping ground of bad code” – until they invested in automated testing
  • #41 DevOps: Dev operates new projects until proven stable in production. Then hands it over to Ops!
  • #44 Here is why our customers choose Dynatrace as a “DevOps Enabler”
  • #45 The AppMon Personal License has been a key enabler of success in spreading the word about dynatrace to new accounts but also “land and expand” in existing accounts While these logos are by far not complete – they represent a couple of companies I recently worked with that use the trial internally Two I want to point out particularly because I like the two stories #1: Sentry: EVERY developer in their onboarding training gets Dynatrace AppMon training including a Personal License. Code changes have to be “Dynatrace Validated” #2: MasterCard: they have been a customer in the past – but – the trial opened up a huge opportunity with MasterCard in China which would have not been possible without the Trial -> tell story!
  • #46 We also had a lot of people sending in their PurePaths through the Personal License Program – here is my most favorite problem pattern that we detect in about 70% of the PurePaths we get sent in
  • #47 Great to see that a lot of these auto-detection features made it into the product – which means that more people can benefit from this “Artificial Intelligence”. This used to be a competitive advantage for a while but in the meantime our competition also offers some basic auto-detection of problem patterns.
  • #59 Once you know how your app is exercised in PRD, you need to be able to feed back to QA and dev the proper distribution of PERFORMANCE they should be shooting for…. The left is a picture of one of our high volume apps during peak (*explain the performance buckets here and how they work*). You can feed them (QA DEV) this picture as an expectation as to how the app should perform under large load. You can use this a barometer and judge QA load tests against. As you can see on the right – there is a much higher mount of yellow transactions making up Maybe mention that there’s a 3rd bucketizing and tease te next webinar for UEM
  • #66 The next slides show a scenario that happened in our organization. This dashboard is used by our marketing and business teams to see how well frequented our website is (total numbers in top chart), how user experience plays out (top chart with green/yellow/red) and how many people sign up for our free trial offering (conversion rate)
  • #67 May 1st was a push of a new release and a marketing campaign started that promoted these features and tried to get people to sign up Seems everything was working as expected
  • #68 Day 2 started good but we also saw that slower web site performance (due to the heavy load) was impacting our end user experience and also conversion rate
  • #69 The Dev Team provided a hotfix to make the sign up for faster #1: It got deployed around noon #2: Fix had negative impact as it broke the whole website due to a javascript problem on certain browsers #3: problem was immediately visible to both business (drop in conversion) and dev (they looked at the reported JavaScript problems and user experience)
  • #70 Due to the fast feedback from Production the Dev Team immediately fixed that regression – bringing the system back to where they wanted it to be in the first place
  • #71 Here is why our customers choose Dynatrace as a “DevOps Enabler”
  • #79 I travel a lot and I learned my lessons the hard way when interacting with people from different cultures and nobody told me how to correctly act or react …. But – communication is not only a challenge when we deal with totally different ppl
  • #80 Here is the situation I ran into with Karolina Why am I telling you about this? Because I think communication is one of the key cornerstones of good collaboration which is a key part of DevOps
  • #81 These stats explain it to me on why face2face communication is key to successful collaboration
  • #83 Some advice