Forum Speed IT, Warsaw 19/11/2015
Jerzy Gulczyński, Dreamlab.pl
DevOps
By Dreamlab
IS DEVOPS LIKE TEENAGE SEX…?
www.inzynierem.pl
Everyone
talks about it
No one really knows
how to do it
Everyone thinks
Everyone else is doing it
So everyone claims they are doing it.
REALITY BASED ON PUPPET LABS 2014
REPORT
„Our analysis makes it clear that DevOps teams are
a growing trend.”
16% of respondents were part
of a DevOps department
https://puppetlabs.com/2014-devops-report
57% are implementing, or have
already implemented DevOps
practices
WHAT IS DEVOPS
ALL IN A SELF-ORGANIZED TEAM AND ONE CYCLE
AGILE
EXTENSION
CULTURE OF CONTINUOUS IMPROVEMENT
DEVOPS CALMS
• Focus on People
• Embrace Change & ExperimentationCULTURE
• „Continuous Delivery”
• „Infrastructure as Code”Automation
• Focus on producing value for end user
• Small batch sizesLean
• Measure everything
• Show the improvementMeasurement
• Open information sharing
• Collaboration & CommunicationSharing
WHY #1
OF
BUSSINES
Puppets Labs findings:
OF BUSINESS
OF
BUSSINES
OF BUSINESS
„High-performing IT organizations
• experience 60 times fewer failures
• and recover from failure [MTTR] 168
times faster than their lower-
performing peers
• They also deploy 30 times more
frequently
with 200 times shorter lead times”
https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf
WHY #1
WHY #2
JOB SATISFACTION
HAPPY COWS MAKE BETTER CHEESE
IT WAS YEAR 2006
To organise, make it „professional”
and the results predictible
ROAD TO MATURITY
NETWORK
1stLINE
UNIX
DBA
FRONTEND
PM’s
MIDDLEWARE
DATABASE
DEVELOPMENT OPERATIONS
UPTIME
NETWORK
1stLINE
UNIX
DBA
FRONTEND
PM’s
MIDDLEWARE
DATABASE
DEVELOPMENT OPERATIONS
TIMECOST
SCOPE
ROAD TO MATURITY
LESSONS LEARNED
CREATIVITY ?
FORGET
JOB SATISFACTION ?
BURN OUTI CAN RUN FASTER
SPEED LIMIT 60 KM/h ?
ROAD TO AGILE
In 2011 we went into AGILE
And into DEVOPS in 2012
ROAD TO AGILE
Development Operations
2nd line1st lineFrondend DatabasesMiddleware
NET UNIX DBA
SILO STRUCTURE, PROJECT ORIENTED
Project Management Office
Quality Assurance Team
Development Operations
2nd line1st line
NET UNIX DBA
Project Management Office
Quality Assurance Team
STEP #1 PRODUCT ORIENTED (NOT PROJECT]
Product 1 Product...Product 2
SG VOD
Product 2
Zumi …
Development Operations
2nd line1st lineProduct 1 Product...Product 2 Product 2
BETTER, BUT…
I want a
change
I want
stability
Who is
responsible
Accountability++ but limited due to lack of competences
Number of changes++ but still controlled and slowed down by ITIL
24 support in a product oriented teams getting hurt
Inconsistent architecture due to separate product teams
Fuzzy responsibility from business point of view
STEP #2 PLATFORMS
PaaS
Product
s
Queue
CMS
OCDN
Search
Big Data Video
RDBS
Cache
NO SQL
Object St.
…
…
Vod.pl
Onet.pl
Wiadomosci.onet.pl
….pl / …rs / …hu / …cz
Architecture Board
Continuous Delivery Team
Master of Service Team
Products Platforms
Cloud
PaaS
Services
VideoSG VOD
…
STEP #3 DEVOPS TEAMS
Bigdata …
Project Management Office
Quality Assurance Team [provides QA standards and tools]
Solution Product Owners
Email
Zumi
Social CMS
Monitoring as a service Team
PRODUCT TEAM (NOT PROJECT)
They Make It,
They Break It,
They Fix It.
WHEN PRODUCT IS TOO BIG - DIVIDE IT VERTICALLY
Video Platform
Team
VOD Service
Team
Agile & DevOps
CULTURE OF LEARNING
Purpose
Mastery
Autonomy
Trust
Self-organization
HIGH ALIGNMENT AND AUTONOMY
http://blog.crisp.se/author/henrikkniberg
CLEAR GOALS AND WIDER PERSPECTIVE
Company
goals
Quarterly goals
for Teams
DevOps
Demo days
Quarterly
Loop
Demo Days
SHARE YOUR KNOWLEDGE, BRAG, CELEBRATE
ENGAGE THE PEOPLE AND LET THEM CREATE COMPANY CULTURE
HackDay
ENGAGE THE PEOPLE
BarCamp
BLAMELESS POSTMORTEMS
WHAT IF I TOLD YOU
IT WAS A BUG IN A THIRD-PARTY SYSTEM
• Do not blame for failures
• Be focused on conclusions
and on actions after
• Be transparent to Business
TEAM HEALTH CHECK & IMPROVEMENT
BOARDS
MEASURE EVERYTHING
• Log everything
• Provide logs and events open
for all devops
we collect 130k
logs, events per
second
1 mln metrics
in Graphite
Online console, Graphite, Hadoop
Monitoring is intended for administrators only.
People who don't know how to create it and
thereby who are not able to interpret it are kindly
REQUESTED NOT to run alarms and NOT to disturb
administrators.
MONITORING THE PAST
Monitoring Center and MaaS Team
REACTIVE MONITORING
PROACTIVE MONITORING
Monitoring TVs in Teams’ Rooms
MONITORING FEEDBACK LOOP
MaaS Team [24/7/365]
Product/TeamReactive
Proactive
Monitoring points (buttons)
Troubleshooters/HowTo’s
Escalation Path
Trainings
First aid based on Howto's
Fake alarms, lack of
alarms
DEPLOY OFTEN
10+ deployments / day
50-60 deployments / day
Every 3 minutes (in average 200 daily)
Every 11.6 seconds
Deployments rate
CONTINUOUS DEPLOYMENT
Automated and boring process
Development
Unit Tests,
Lint Score
Code
Review
Integration
Tests
Functional
Tests
Deployment
Monitoring
AB TESTING
WHEN
DEPLOYING
AB TESTING WHEN DEPLOYING
Deploy on 10% of traffic, check, increase
PET
Pets
Cattle
INFRASTRUCTURE AS CODE
orchestrator
PaaS
orchestrator
IaaS
servers
VMapplications
TO BE INFRASTRUCTURE INDEPENDENT
Amazon
EC2
Amazon S3
Onet DataCenter Credit Card
Amazon ELB
Business
continuity plan
4h a day
1% of traffic
OWN PRIVATE VS PUBLIC
IS IT STILL POSSIBLE TO CHASE
PUBLIC CLOUD SERVICES ?
UNFORTUNATELLY NOT,
NO LONGER
Amazon AWS with over 1 million active customers
S3 data transfer has increased by 120%.Y2Y
Databases services increased by 127% Y2Y
EC2 usage increased by 95% Y2Y
PRIVATE CLOUDS ARE NO LONGER ABLE TO COMPETE WITH PUBLIC
„Private cloud will not keep up with the public cloud
dynamics of innovations, breadth and depth of offering.”
„80% of private cloud fails today as no one wants to use it:
developers prefer public cloud”
„Cloud is a software platform game not a hardware
commodity game.”
„Yes, you can decide to use IaaS in most commoditized
mode, build your orchestration and middleware layers, and
now you are locked into your own in-house technology and
you will maintain it for the rest of your career.”
Gartner:
Capital One: "We want to build great applications for our
customers, not to invest in costly & complex infrastructure
[...] we believe it enables us to operate more securely in a
public cloud that we can even could in our own datacenters”
General Electric „The Cloud has gone from probable to
inevitable. We intend to move over 60% of our global
workload into AWS. For us this is no longer an experiment,
it’s no longer a test, it’s not something we talk being
probable, its inevitable.”
ENTERPRISES ARE REALLY USING AMAZON
BIMODAL IT ?
DON’T JUMP INTO DEVOPS IN A WATERFALL
WAY
LINKS
One piece flow: https://www.youtube.com/watch?v=Dr67i5SdXiM
Puppet Labs Reports:
https://puppetlabs.com/2014-devops-report
https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf
Microservices: http://martinfowler.com/articles/microservices.html
Small Batch: http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-
improve-flow/
BOOKS
DevOps at DreamLab

DevOps at DreamLab

  • 1.
    Forum Speed IT,Warsaw 19/11/2015 Jerzy Gulczyński, Dreamlab.pl DevOps By Dreamlab
  • 4.
    IS DEVOPS LIKETEENAGE SEX…? www.inzynierem.pl Everyone talks about it No one really knows how to do it Everyone thinks Everyone else is doing it So everyone claims they are doing it.
  • 5.
    REALITY BASED ONPUPPET LABS 2014 REPORT „Our analysis makes it clear that DevOps teams are a growing trend.” 16% of respondents were part of a DevOps department https://puppetlabs.com/2014-devops-report 57% are implementing, or have already implemented DevOps practices
  • 6.
    WHAT IS DEVOPS ALLIN A SELF-ORGANIZED TEAM AND ONE CYCLE AGILE EXTENSION CULTURE OF CONTINUOUS IMPROVEMENT
  • 7.
    DEVOPS CALMS • Focuson People • Embrace Change & ExperimentationCULTURE • „Continuous Delivery” • „Infrastructure as Code”Automation • Focus on producing value for end user • Small batch sizesLean • Measure everything • Show the improvementMeasurement • Open information sharing • Collaboration & CommunicationSharing
  • 8.
    WHY #1 OF BUSSINES Puppets Labsfindings: OF BUSINESS
  • 9.
    OF BUSSINES OF BUSINESS „High-performing ITorganizations • experience 60 times fewer failures • and recover from failure [MTTR] 168 times faster than their lower- performing peers • They also deploy 30 times more frequently with 200 times shorter lead times” https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf WHY #1
  • 10.
    WHY #2 JOB SATISFACTION HAPPYCOWS MAKE BETTER CHEESE
  • 11.
    IT WAS YEAR2006 To organise, make it „professional” and the results predictible
  • 12.
  • 13.
  • 14.
    LESSONS LEARNED CREATIVITY ? FORGET JOBSATISFACTION ? BURN OUTI CAN RUN FASTER SPEED LIMIT 60 KM/h ?
  • 15.
    ROAD TO AGILE In2011 we went into AGILE
  • 16.
    And into DEVOPSin 2012 ROAD TO AGILE
  • 17.
    Development Operations 2nd line1stlineFrondend DatabasesMiddleware NET UNIX DBA SILO STRUCTURE, PROJECT ORIENTED Project Management Office Quality Assurance Team
  • 18.
    Development Operations 2nd line1stline NET UNIX DBA Project Management Office Quality Assurance Team STEP #1 PRODUCT ORIENTED (NOT PROJECT] Product 1 Product...Product 2 SG VOD Product 2 Zumi …
  • 19.
    Development Operations 2nd line1stlineProduct 1 Product...Product 2 Product 2 BETTER, BUT… I want a change I want stability Who is responsible Accountability++ but limited due to lack of competences Number of changes++ but still controlled and slowed down by ITIL 24 support in a product oriented teams getting hurt Inconsistent architecture due to separate product teams Fuzzy responsibility from business point of view
  • 20.
    STEP #2 PLATFORMS PaaS Product s Queue CMS OCDN Search BigData Video RDBS Cache NO SQL Object St. … … Vod.pl Onet.pl Wiadomosci.onet.pl ….pl / …rs / …hu / …cz
  • 21.
    Architecture Board Continuous DeliveryTeam Master of Service Team Products Platforms Cloud PaaS Services VideoSG VOD … STEP #3 DEVOPS TEAMS Bigdata … Project Management Office Quality Assurance Team [provides QA standards and tools] Solution Product Owners Email Zumi Social CMS Monitoring as a service Team
  • 22.
    PRODUCT TEAM (NOTPROJECT) They Make It, They Break It, They Fix It.
  • 23.
    WHEN PRODUCT ISTOO BIG - DIVIDE IT VERTICALLY Video Platform Team VOD Service Team
  • 24.
  • 25.
  • 26.
    HIGH ALIGNMENT ANDAUTONOMY http://blog.crisp.se/author/henrikkniberg
  • 27.
    CLEAR GOALS ANDWIDER PERSPECTIVE Company goals Quarterly goals for Teams DevOps Demo days Quarterly Loop
  • 28.
    Demo Days SHARE YOURKNOWLEDGE, BRAG, CELEBRATE
  • 29.
    ENGAGE THE PEOPLEAND LET THEM CREATE COMPANY CULTURE HackDay
  • 30.
  • 31.
    BLAMELESS POSTMORTEMS WHAT IFI TOLD YOU IT WAS A BUG IN A THIRD-PARTY SYSTEM • Do not blame for failures • Be focused on conclusions and on actions after • Be transparent to Business
  • 32.
    TEAM HEALTH CHECK& IMPROVEMENT BOARDS
  • 33.
    MEASURE EVERYTHING • Logeverything • Provide logs and events open for all devops we collect 130k logs, events per second 1 mln metrics in Graphite Online console, Graphite, Hadoop
  • 34.
    Monitoring is intendedfor administrators only. People who don't know how to create it and thereby who are not able to interpret it are kindly REQUESTED NOT to run alarms and NOT to disturb administrators. MONITORING THE PAST
  • 35.
    Monitoring Center andMaaS Team REACTIVE MONITORING
  • 36.
  • 37.
    MONITORING FEEDBACK LOOP MaaSTeam [24/7/365] Product/TeamReactive Proactive Monitoring points (buttons) Troubleshooters/HowTo’s Escalation Path Trainings First aid based on Howto's Fake alarms, lack of alarms
  • 38.
    DEPLOY OFTEN 10+ deployments/ day 50-60 deployments / day Every 3 minutes (in average 200 daily) Every 11.6 seconds Deployments rate
  • 39.
    CONTINUOUS DEPLOYMENT Automated andboring process Development Unit Tests, Lint Score Code Review Integration Tests Functional Tests Deployment Monitoring
  • 40.
    AB TESTING WHEN DEPLOYING AB TESTINGWHEN DEPLOYING Deploy on 10% of traffic, check, increase
  • 41.
  • 42.
  • 43.
  • 44.
    TO BE INFRASTRUCTUREINDEPENDENT Amazon EC2 Amazon S3 Onet DataCenter Credit Card Amazon ELB Business continuity plan 4h a day 1% of traffic
  • 45.
    OWN PRIVATE VSPUBLIC IS IT STILL POSSIBLE TO CHASE PUBLIC CLOUD SERVICES ? UNFORTUNATELLY NOT, NO LONGER Amazon AWS with over 1 million active customers S3 data transfer has increased by 120%.Y2Y Databases services increased by 127% Y2Y EC2 usage increased by 95% Y2Y
  • 46.
    PRIVATE CLOUDS ARENO LONGER ABLE TO COMPETE WITH PUBLIC „Private cloud will not keep up with the public cloud dynamics of innovations, breadth and depth of offering.” „80% of private cloud fails today as no one wants to use it: developers prefer public cloud” „Cloud is a software platform game not a hardware commodity game.” „Yes, you can decide to use IaaS in most commoditized mode, build your orchestration and middleware layers, and now you are locked into your own in-house technology and you will maintain it for the rest of your career.” Gartner:
  • 47.
    Capital One: "Wewant to build great applications for our customers, not to invest in costly & complex infrastructure [...] we believe it enables us to operate more securely in a public cloud that we can even could in our own datacenters” General Electric „The Cloud has gone from probable to inevitable. We intend to move over 60% of our global workload into AWS. For us this is no longer an experiment, it’s no longer a test, it’s not something we talk being probable, its inevitable.” ENTERPRISES ARE REALLY USING AMAZON
  • 48.
  • 49.
    DON’T JUMP INTODEVOPS IN A WATERFALL WAY
  • 50.
    LINKS One piece flow:https://www.youtube.com/watch?v=Dr67i5SdXiM Puppet Labs Reports: https://puppetlabs.com/2014-devops-report https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf Microservices: http://martinfowler.com/articles/microservices.html Small Batch: http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches- improve-flow/
  • 51.