Tyranny of the SLA

THE TYRANNY
OF THE SLA
JIM KIMBALL
J. PAUL REED
VELOCITY NEW YORK, 2015
99.999%
uptime

OUR
SERVICE LEVEL AGREEMENT
WITH YOU
@jpaulreed@jimkimball #velocityconf

THIS PRESENTATION WILL KEEP YOU
AWAKE 99.5% OF THE TIME.

THE MTTL WILL BE 4.2 MINUTES.

YOU AGREE THAT, AS A VALUED
CUSTOMER, ALL TWEETS REGARDING THIS
PRESENTATION MUST BE POSITIVE IN
NATURE

IN THE CASE OF DEGRADED
PERFORMANCE OF THE PRESENTATION,
OUR FINANCIAL LIABILITY IS LIMITED TO
THE ASSESSED VALUE OF THIS SESSION.*

* O’Reilly’s valuation pending**

** Value likely to be 1/100th of a cent
* O’Reilly’s valuation pending**

IN CASE OF EXCESSIVE CELL PHONE
UTILIZATION, YOUR CONFERENCE
ACCESS WILL BE REVOKED.

J. PAUL REED
• @JPAULREED ON
• HOST OF THE
@SHIPSHOWPODCAST
• 15+ YEARS IN BUILD/RELEASE
ENGINEERING
• WORK WITH ALL SORTS OF ORGS
ON “THE DEVOPS™”
• VISITING SCIENTIST/CHIEF
DELIVERY OFFICER AT PRAXISFLOW

JIM KIMBALL
• CTO, HEDGESERV
• 25 YEARS IN THE FINANCIAL
SOFTWARE INDUSTRY
• @JIMKIMBALL ON
• TOC ICO JONAH
• THOUGHTS ON LEADING SOFTWARE
ORGANIZATIONS AT
SHARINGLUNCH.TUMBLR.COM

CAUTION: EXPERIMENT AHEAD

PROBLEMS WITH
TODAY’S SLAs

WE
HAVE
SLA
FATIGUE

Availability Year Quarter Month
90.0% 36.5 days 9 days 72 hours
99.0% 3.65 days 4.5 days 36 hours
99.5% 1.83 days 11.7 hours 3.6 hours
99.9% 8.76 hours 2.19 hours 43.8 mins
99.99% 52.6 mins 13.1 mins 4.38 mins
99.999% 5.26 mins 77.7 secs 25.9 secs
55.5555555% 162.2 days 40 days 13.3 days
Remember All Those Nines?

Remember All Those Nines?
Availability Year Quarter Month
90.0% 36.5 days 9 days 72 hours
99.0% 3.65 days 4.5 days 36 hours
99.5% 1.83 days 11.7 hours 3.6 hours
99.9% 8.76 hours 2.19 hours 43.8 mins
99.99% 52.6 mins 13.1 mins 4.38 mins
99.999% 5.26 mins 77.7 secs 25.9 secs
55.5555555% 162.2 days 40 days 13.3 days

Deﬁnitions Are Hard
What is an “outage?”

Uptime vs. Availability

Maintenance windows?

“Acts of God”

Often Opaque

SLAs
Can
Promote
Silos

Every conceivable thing has
been taken into consideration.
That’s why we have what we
call defense in depth.
Now that means backup
systems to backup systems to
backup systems. …
Even with a faulty relay,
even with a stuck valve,
that system works.

But we didn’t uncover
[the core], did we?
We stopped it in time for one
simple reason, and I told you
that: the system works.
Dammit, the system works.
That’s not the problem.

FOR ABOUT A CENTURY, THEN,
DETERMINISM WAS ASSUMED TO EXIST
AND TO BE THE FIRST REQUIREMENT TO BE
ABLE TO EXERT PRECISE CONTROL OVER
THE WORLD. THIS HAS COME TO DOMINATE
OUR ATTITUDES TOWARD CONTROL.
TODAY, DETERMINISM IS KNOWN TO BE
FUNDAMENTALLY FALSE, AND YET THE
ILLUSION OF DETERMINISM IS STILL
CLUNG ONTO WITH FERVOR IN OUR HUMAN
WORLD OF … COMPUTERS AND
INFORMATION SYSTEMS.
MARK BURGESS
IN SEARCH OF CERTAINTY

EXTERNAL SLAs

Amazon Web Services
First, which SLA?

Amazon Web Services
First, which SLA?

But not for account suspensions and terminations

Ditto maintenance (as deﬁned!)

Also ignore “failures of individual instances or volumes not
attributable to Region Unavailability”

Expect no more than a 10% credit. (Maybe 30%.)

(Doesn’t apply to one-time charges… aka “reserved instances.”)

SLAs ARE SAFEGUARDS YOU PUT INTO
BROKEN RELATIONSHIPS.
— Roy Rappaport, Netﬂix

Pagerduty
Dig for the SLA

Basic plan: “best eﬀort”

Standard plan: “5 minutes”

Enterprise plan: Insurance ($3 million!)

But really: they focus on reliability and resilience

[REDACTED]

“100% SLA availability? Really?!”

“I’ve forwarded your question to our
attorney and he’s suggested that we
remove the reference to 100%.
So we’ll do that ASAP.”

“I’ve forwarded your question to our
attorney and he’s suggested that we
remove the reference to 100%.
So we’ll do that ASAP.”
(They did.)

Why SLAs In The First Place?

THE SLA ELEPHANT IN THE DATACENTER@jpaulreed@jimkimball #velocityconf

INTERNAL SLAs

Cautionary
Uses of
Internal SLAs

Language Matters
Service Level Agreements
versus Service Level
Commitments

Service Level Agreements
as “relationship
agreements?”

NOT ALL SLAs ARE BAD
(OR CREATED EQUAL)

Making Prioritization Contextual

Fostering the Right Conversations

OTHER CONSIDERATIONS
FOR IMPROVEMENT

Brené Brown
“One of the ways we deal
with it is: we numb.”

“We make everything
that’s uncertain, certain.”

“We perfect. And, more
dangerously, we perfect
our kids.”

Whether it’s a bailout, an oil
spill, a recall: we pretend like
what we’re doing doesn’t have
a huge impact on other people.
I would say to companies: this
isn’t our ﬁrst rodeo, people.
We just need you to be
authentic and real and say:
We’re sorry. We’ll ﬁx it.

A MODEST PROPOSAL

Remember The Manifesto!

A LONG-TERM RELATIONSHIP
BETWEEN PURCHASER AND SUPPLIER
IS NECESSARY FOR BEST ECONOMY.
…
MORE IMPORTANT THAN PRICE IN THE
JAPANESE WAY OF DOING BUSINESS
IS CONTINUAL IMPROVEMENT OF
QUALITY, WHICH CAN BE ACHIEVED
ONLY ON A LONG-TERM RELATIONSHIP
OF LOYALTY AND TRUST.
W. EDWARDS DEMING
OUT OF THE CRISIS

Obvious
ComplicatedComplex
Chaotic

Obvious
ComplicatedComplex
Chaotic
Governing Constraints
Good Practice
Rigid Constraints
Best Practice
Lack of Constraints
Novel Practice
Enabling Constraints
Emergent Practice

Obvious
ComplicatedComplex
Chaotic
Good Practice
Rigid Constraints
Best Practice
Lack of Constraints
Novel Practice
Emergent Practice
MTTR, etc.

Obvious
ComplicatedComplex
Chaotic
Good Practice
Rigid Constraints
Best Practice
Lack of Constraints
Novel Practice
Emergent Practice
MTTR, etc.
Crew
Formation

Obvious
ComplicatedComplex
Chaotic
Good Practice
Rigid Constraints
Best Practice
Lack of Constraints
Novel Practice
Emergent Practice
MTTR, etc.
Crew
Formation
Formalized
Response

Obvious
ComplicatedComplex
Chaotic
Good Practice
Rigid Constraints
Best Practice
Lack of Constraints
Novel Practice
Emergent Practice
Disorder
MTTR, etc.
Crew
Formation
Formalized
Response

Obvious
ComplicatedComplex
Chaotic
Good Practice
Rigid Constraints
Best Practice
Lack of Constraints
Novel Practice
Emergent Practice
MTTR, etc.
Crew
Formation
Formalized
Response
MTTD, etc.
Disorder

A Minimum Viable SLA
Covers all complexity domains

Involves the business through to the customer

Prompts good behavior among teams…

… and within the organization

Facilitates organizational / team learning

Lightweight as possible

THE SLA, HISTORICALLY,
INCENTIVIZES THE WRONG
BEHAVIORS

MODERN SLAs MUST BE
COMPLEXITY INFORMED

BE AWARE OF THE SLAs
YOU ARE A PART OF…
AND THAT ARE A PART OF
YOUR SYSTEM

J. Paul Reed
preed@release-approaches.com
@jpaulreed
http://jpaulreed.com
Jim Kimball
jkimball@hedgeserv.com
@jimkimball
http://eng.hedgeserv.com
Anonymous Feedback
http://sayat.me/jimkimball
http://sayat.me/jpaulreed

(Pssst… Yes, we’re hiring!)
eng.hedgeserv.com

DevOps
inPractice
J. Paul Reed

Photo Credits
Slide 1: http://jcb.lunaimaging.com/luna/servlet/detail/
JCBMAPS~2~2~313~100035:The-Colossus-of-the-North-or-The-St

Slide 12: http://jimhillmedia.com/editor_in_chief1/b/jim_hill/archive/2009/04/27/
wanna-make-a-muppet-happy-go-vote-for-beaker-s-ode-to-joy.aspx

Slide 15: Generated via imgﬂip.com

Slide 21: https://commons.wikimedia.org/wiki/
File:Pigneau_de_Behaine_Annamite_Latin_Dictionary.jpg

Slide 22: http://www.beckoncall.info/why-do-my-inside-windows-get-so-dirty/

Slide 23: http://commons.wikimedia.org/wiki/File:Wooden_silo.JPG

Photo Credits, Continued
Slide 24: https://commons.wikimedia.org/wiki/File:Nuclear_Missile_Silo_
%287332367192%29.jpg

Slide 25: http://www.cio.com/article/2883770/cloud-computing/the-death-of-the-sla.html

Slide 28: http://markburgess.org/bio.html

Slide 43: http://res.freestockphotos.biz/pictures/10/10006-an-elephant-in-the-wild-pv.jpg

Slide 46: http://www.oldhouseonline.com/patch-plaster-walls/

Slide 47: http://seuss.wikia.com/wiki/The_Zax

Slide 48: http://www.gapingvoidart.com/

Photo Credits, Continued
Slide 50: https://www.youtube.com/watch?v=c4QWgfHOvSM

Slide 51: Courtesy Jabe Bloom / Will Evans

Slide 53: https://www.flickr.com/photos/stignygaard/450640129, https://
www.flickr.com/photos/midendian/358942212/

Slide 54, 55: https://www.youtube.com/watch?v=iCvmsMzlF7o

Slide 56: http://www.moviepostersetc.com/MoviePostersEtc/salvador-dali-poster-les-
montres-molles-clock-explosion-36-x-24--ff8080813ae48649013aebe4facd03b6-
p.html

Slide 57, 58: https://www.flickr.com/photos/62981668@N06/6012392939/

Tyranny of the SLA

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Tyranny of the SLA

Similar to Tyranny of the SLA (20)

More from J. Paul Reed

More from J. Paul Reed (6)

Recently uploaded

Recently uploaded (9)

Tyranny of the SLA