Lviv DevOps Conference 2019 ОЛЕКСАНДР ВІЛЬЧИНСЬКИЙ «DevOps culture» Телеграм канал: https://t.me/GoDevOpsEvent Фейсбук сторінці: https://www.facebook.com/godevopsevent/ Сайт: https://devopsconf.org/

1. DEVOPS PRINCIPLES
How to decrease employee retention rate to 10%,
raise employee satisfaction rate to 85%,
get rid of “managers”,
stop worrying about and other things implicitly
influenced by the DevOps culture

2. 5 years in DevOps
20 years in IT
IT addict
OLEKSANDR VILCHYNSKYY

3. SYMPHONY
SOLUTIONS
10+
YEARS OF
EXPERIENCE
6
LOCATIONS
LVIV
DELIVERY OFFICE
IN UKRAINE
35
INTERNATIONAL
CLIENTS

4. DEVOPS IS A
CULTURE,
NOT A ROLE

5. - SRE (Site reliability engineer)
- CloudOps
- SysOps
- Cloud migration engineer
- DataOps
- TestOps
- ArchOps
- WinOps
- HumanOps
- DevOps evangelist
- *Ops
- Someone who knows Kubernetes, Terraform, CI/CD, etc...
INFLUENCED AND COMMONLY MISPLACED BY
FOLLOWING ROLES:

6. • DevOps Pillars of success
• Reduce organizational silos
• Accept failure as normal
• Implement gradual changes
• Leverage tooling and
automation
• Measure everything
• SRE realization of them
• SRE shares ownership with
developers to create shared
responsibility; SREs use the same
tools that developers use, and vice
versa
• SREs embrace risk; SRE quantifies
failure and availability in a
prescriptive manner using Service
Level Indicators (SLIs) and Service
Level Objectives (SLOs); SRE
mandates blameless post mortems
• SRE encourages developers and
product owners to move quickly by
reducing the cost of failure
• SREs have a charter to automate
menial tasks (called "toil") away
• SRE defines prescriptive ways to
measure values; SRE fundamentally
believes that systems operation is a
software problem
As for me, SRE is the closest role
influenced and inherited from
DevOps cultural movement

7. SRE IS “WHAT HAPPENS WHEN A
SOFTWARE ENGINEER IS TASKED
WITH WHAT USED TO BE CALLED
OPERATIONS”
Picture credit: https://giphy.com

8. OK, I’VE GOT THAT
OPERATIONS SHOULD
BE DEVELOPED LIKE A
SOFTWARE, BUT ...

9. WHAT IS
OPERATIONS?

10. THE PEOPLE AND MANAGEMENT
PROCESSES ASSOCIATED WITH IT
SERVICE MANAGEMENT TO DELIVER
THE RIGHT SET OF SERVICES AT THE
RIGHT QUALITY AND AT COMPETITIVE
COSTS FOR CUSTOMERS
IT operations by Gartner

11. APPLICATION-
HAPPY
OPERATIONS:
- CI/CD
- application and
infrastructure
monitoring and
logging
- configuration
management
- Infrastructure as
Code
- environments
parity
- Test automation
- Orchestration
- Containerization
TEAM-HAPPY OPERATIONS:
- team communication (single point of
information, transparency and full insights
from Product owners)
- agile automations (meetings minimization)
- shared responsibilities and silos removal
(everybody is an Engineer, QA, SysOps, SRE
and Product owner)
- new employee bootstrap processes (good if
less than half a day, with local environment
and IDE configuration)
- self-service for getting all required accesses
(no need to ask someone and wait for long
approvals)
- self-updated application documentation
(Self-documented code, high-level
documentation for project in Readme.md)
- possibility for proactiveness (Hackatons,
Bug Squashing Fridays and Refactoring
Fridays)
- blameless mistakes, unless they happen
twice (mistake should be fixed and short
post-mortem should be written)
- good developed and written: teams, roles,
escalation policies, conventions and
development processes
Let’s assume that operations is
everything needed for application
to function and be developed.

12. EVOPS PRINCIPLES
AUTOMATION
MEASUREMENT
IMPROVEMENT
CULTURE
AGILE
PARITY

13. CULTURE
IS THE
KEY
DevOps spend
their energy on
Code, not
Culture

14. CULTURE IS IN CREATING
THE ATMOSPHERE OF
SAFE PLAYGROUND
MISTAKES ARE OK, AS LONG AS THEY DO NOT GET REPEATED

15. DAY-1 MENTALITY AND WHERE
ENGINEERS MAY/SHOULD
CONTRIBUTE BUSINESS-WISE

16. BUSINESS
PLANNING
INFRASTRUCTURE
REQUIREMENTS
OPERATIONS
DEVELOPMENT
USE
BUSINESS
OPERATIONS

17. Portfolio backlog
• Demands
• Portfolio backlog
• Project portfolio
Enterprise architecture
• Solution
• Architecture
Content/ document
• Content
• Information
Source
code
• Source code
• Commits/
merge
requirements
CICD pipeline
• Build
pipeline
• Release
pipeline
Code quality
• Peer reviews
• Static code analysis
Build
• Build dependencies
• Build package
Test management
• Unit test
• Performance
• Functional
Deployment
• Deployment
workflows
Monitoring
• Infrastructure
• Application
Security monitoring
• Logs
• Vulnerabilities
Application performance
• Availability
• Performance
Log analytics
• Log monitor
• Log analytics
AIOPs
• Events,
metrics
• Telemetry,
traces
ITSM system
• Incident
• Problem
• Change
Self service portal
• Self help
• Self service
CMDB
Discovery
Service models
Dashboards and
reports
Metrics/ reports
Product/ team
backlog
• Stories, features,
sprint plans
Artifact repository
• Built artifacts
• Packages
• Conversations
Notification and
collaboration
ILLUSTRATION OF END-TO-END DEVOPS/ IT4IT TOOL CHAIN

18. DEVOPS SUCCESS
INDICATORS

19. THE SPEED WITH WHICH
YOUR SYSTEM TURNS
BUSINESS REQUIREMENT
INTO A WORKING CODE

20. IT performance High IT
performers
Medium IT
performers
Low IT
performers
Deployment frequency
(how often your organization deploys
code)
1h/ 1d 1w/ 1m 1w/ 1m
Lead time for changes
(how long does it take from code
commit to code successfully running
in production)
1d/ 1 w 1w/ 1m 1m/ 6m
Change failure rate
(what % of changes either result in
degraded service or subsequently
require remediation)
0%- 15% 0%- 15% 46%- 60%
Mean time to recover
(how long does it take to restore
services when a service incident
outage, impairment) occurs)
≤1d ≤1d 1w- 1 m
DEVIATIONS IN IT PERFORMANCE
*Data credit: DevOps Stage 2018

21. HOW TO DISTINCT DEVOPS
CULTURE

22. EX. PRODUCT RELEASE
OPERATIONS
ARE NOT
Picture credit: https://giphy.com

23. EX. PRODUCT RELEASE
OPERATIONS
ARE
Picture credit: https://giphy.com

24. NOT A DEVOPS CULTURE
• Separate DevOps team from
others
• Headphones and/or muted
channels of communication
• Single responsibility (as
engineer that is responsible only
for code). Think only for yourself
• Toil (tedious, boring job), which
is always solved manually day
by day
• Tons of half-obsolete or missing
documentation
• Idea promotion through short
summaries (like, let’s update our
framework, because it will be
cool)
• Neglecting other’s ideas
• Blaming others
• Processual time wasting
• “DevOps” quantity grows linear
on the project with the services
quantity and project growth -
O(n)
DEVOPS CULTURE
• Embed operations skills into
delivery teams to reduce friction
and deliver better outcomes
• Fast response and pro-activeness
to the problems
• Close collaboration (as engineer
writing testing and release notes in
ticket); Think for team
• Automation of toil job (even if it
management job). Everything can
be expressed as code.
• Think twice before writing
documentation and how to support
it in the future.
• Writing press releases and FAQ for
your ideas.
• All good written ideas should be
considered.
• Writing post-mortems and
discussing them together.
• LEAN
• “DevOps” quantity grows
logarithmically O(log(n))
DEVOPS CULTURE
INDICATORS

25. TALENT RETENTION

26. Tensions or
misunderstandings
in the team
Lack of direction, ex.
the person has
delivered something,
and lacks clarity
what next
Poor recognition
within and/or outside
the team
Poor understanding
of operational
processes
WHY PEOPLE
ARE LEAVING

27. Opportunity of
choice
Opportunity to have
full visibility of
processes, BUT
without an obligation
to track everything
Offsite projects, as a
personal challenge
(may grow into full
time business
initiatives with time)
Single source of
information: up to
date, and precise
WHAT KEEPS
PEOPLE

28. “MANAGERS” APPEAR AT
PLACES WHERE
AUTOMATION IS MISSING

29. • Project managers
• Release managers
• Scrum masters
• DevOps evangelists
• Git Managers, Test Managers …
• Even engineers if their work can be automated
WHAT “MANAGERS”?

30. HOW DO YOU KNOW, IF YOU NEED A
“MANAGER” OR NOT
Manager does
monotonous,
admin, repetitive
job? Replace by a
chat bot
Manager involved in
value adding tasks?
Keep them

31. DEVOPS’S MISSION
– IN LIQUIDATION
OF TOIL

32. WHY DO WE NEED ALL OF THIS?
BECAUSE, IT'S IMPOSSIBLE TO
IMAGINE A FUTURE 10 YEARS FROM
NOW WHERE A CUSTOMER COMES UP
AND SAYS, 'JEFF I LOVE AMAZON; I
JUST WISH THE PRICES WERE A
LITTLE HIGHER,' [OR] 'I LOVE
AMAZON; I JUST WISH YOU'D DELIVER
A LITTLE MORE SLOWLY.'
IMPOSSIBLE.
(C) JEFF BEZOS

33. • Make work visible;
• Limit work in progress;
• Reduce batch size;
• Reduce handoffs;
• Continuously work with
constraints;
• Eliminate hardships &
waste in value stream.
• Enable learning culture;
• Institutionalize daily
improvements;
• Transform local discoveries
into global improvements;
• Inject resilience patterns in
daily work;
• Leaders as role models for
learning culture.
• See problems as they
occur;
• Swarm and solve to
share knowledge;
• Keep pushing quality
closer to source;
• Enable optimizing for
downstream work
centers.
• Make security part of everyone's job;
• Integrate preventative controls into the
shared source code repository;
• Integrate security with deployment
pipeline;
• Integrate security with telemetry;
• Protect the deployment pipeline;
• Integrate deployment activities with
change approval process;
• Reduce reliance on separation of duty.
POST
MORTEMS
FLOW FEEDBACK
IMPROVEMENT INTEGRATION
*Data credit: The DevOps Handbook, by Gene Kim, John Willis, Jez Humble

34. THANK YOU