DEVOPS
START WALKING IN THE SAME DIRECTION
WHO I AM?
Demis Rizzotto
System Engineer,DevOps Linux
Engineer @Exxoss
Currently consultant @Lampiris
THE PROBLEM OF
DOWNWARD SPIRAL
DIAMETRICALLY OPPOSED GOALS AND INCENTIVES
DEV
▸ Respond to the rapidly
changing competitive
landscape
OPS
▸ Provide stable, reliable, and
secure service to the
customer
THE CORE, CHRONIC CONFLICT
WHEN ORGANIZATIONAL MEASUREMENTS AND INCENTIVES ACROSS DIFFERENT SILOS PREVENT THE
ACHIEVEMENT OF GLOBAL, ORGANIZATIONAL GOALS
FRUSTRATION
▸Unnecessary process steps
▸Unnecessary rework
▸Unnecessary features
▸Waiting for someone else to act on an action or task
▸Waiting for an approval
▸Waiting for an environment
▸Creating a ticket for a manual task
▸Reporting to management by manually updating
spreadsheets
▸Manual status reporting
▸…
RESULTS
▸ Fragile application are prone to failure
▸ Ever increasing backlog and technical debt
▸ Urgent security rework and remediation
▸ More urgent projects into the queue
▸ Big deployments very difficult to diagnose when fail
DEVOPS
More features
quicker
Stability
DEVOPS
DEV OPS
FACILITATE AND AUTOMATE DEPLOYMENT
HELP DEV TEAM BECOME AUTONOMUS
FACILITATE DIAGNOSTIC
GET DEVS INTERESTED IN « PROD STUFF »
GIVE ACCESS TO LOGS
GIVE ACCESS TO MONITORING
OPTIMIZE LOGS
DEFINE METRICS
QUALITY TESTS
MONITOR YOUR APPS
DEV IN DEVOPS
DEV IN DEVOPS
IMPROVEMENTS
▸ The ability to accurately track dependencies
▸ Services that are resilient and degrade gracefully
▸ The ability to archive data to manage the size of the production
data set
▸ The ability to easily search and understand log messages across
services
▸ The ability to trace requests from users through multiple services
▸ Simple, centralized runtime configuration using feature flags, …
DEV IN DEVOPS
▸ Make Dev responsible for their own services
▸ Integrate Dev into the incident management processes
▸ Have Dev cross-train Ops
▸ Embed Ops knowledge and capabilities into Dev
▸ Design for IT Operations
EMBED IT OPERATIONS INTO DEVELOPMENT
OPS IN DEVOPS
OPS IN DEVOPS
IMPROVEMENTS
▸ Self service
▸ Infrastructure as code
▸ Standardize infrastructure
▸ Make all infrastructure data visible
▸ Make application info data visible
▸ Modify the incident resolution process and blameless post-mortems
▸ Monitor the health of the deployment pipelines
OPS IN DEVOPS
▸ Single “repository of truth” containing both the code and
environments
▸ Create the one-step Dev, Test and Production environment build
process
▸ Extend the deployment pipeline processes into production
EXTEND DEVELOPMENT INTO IT OPERATIONS
WE ARE ALL DEVOPS
WE ARE ALL DEVOPS
▸ Stop think . Do
▸ Share responsability
▸ It’s Never Someone Else’s Problem
▸ Dedicated Designated Ops in every team
▸ Different type of devops, but devops is possible for all type
and size of companies
▸ Iterate!
▸ Identify bottleneck and remediate
▸ Define small target and share results
DEVOPS IS FOR ALL
DEVOPS IS FOR ALL
DEVOPS IS FOR ALL
▸ AUTONOMUS TEAMS
▸ Trust but verify
▸ Local autonomy, but global consistency
▸ SELF SERVICE
▸ How ? API, web portal, infra as code , AWS accounts, …
▸ AUTOMATE
▸ If you are tired to do it manually. Automate it.
▸ REDUCE NON-VALUE ADDED
▸ If isn't your business it's maybe better don't do it your self
KEY OF SUCCESS : REDUCE FRUSTRATION
INFRASTRUCTURE
ON DEMAND ( IAAS )
EASIER TO REBUILD THAN TO REPAIR
INFRASTUCTURE
INFRASTRUCTURE
IMMUTABLE INFRASTRUCTURE
▸ Automate the setup and deployment for every part and
every layer of your infrastructure.
▸ Never change any part of your system once it is deployed.
If you need to change it, deploy a new system.
AVOID HISTORY
INFRASTRUCTURE
INFRASTRUCTURE AS CODE
Versionning and code review
Easier track changes
Bootstrap your infrastructure from scratch
Automate
CI/CD
TESTING
TESTING
CONTINUOUS TESTING
▸ Automated tests
▸ code quality
▸ Unit tests
▸ Stress tests
▸ Don’t forget to test infrastructure (load banacer, recovery
etc.)
▸ Extend you build pipeline
SECURITY
SECURITY
INFORMATION SECURITY AS EVERYONE’S JOB, EVERY DAY
▸ Updates
▸ Use secrets management
▸ Integrate security into our CI/CD pipeline
▸ Check code libraries and their recommended configurations
▸ Dependency scanning
▸ Static analysis
▸ Dynamic analysis (ex: OWASP Zed Attack Proxy + Selenium)
SECURITY
SECURITY METRICS MONITORING
▸ Successful and unsuccessful user logins
▸ User password resets
▸ User email address resets
▸ Database syntax error
▸ User credit card changes
DEPLOYMENTS
DEPLOYMENTS
DEPLOYMENTS ARE ROUTINE AND PREDICTABLE
No stress . It’s a routine.
On business day when everyone is
already in the office and without our
customers even noticing
DEPLOYMENTS
HOW
▸ Work in small batches and delivering quickly and
frequently
▸ Blue green deployment
▸ Feature toggle
▸ Canary testing
▸ Dark launch
▸ Automated deployments
MONITORING
ALERTING
▸ Mesure service health not only server health (Health pattern)
▸ Reducing Alert Fatigue
▸ Cut alerts that aren’t actionable
▸ Adjust thresholds
▸ Consolidate related alerts
▸ Give alerts relevant names & descriptions
▸ Make sure the right people are getting alerts
▸ Multiple notification rules and choose right channel
▸ Use incident management tools(Pagerduty, Victoryops,..)
MONITORING
IF IT MOVES, TRACK IT
ANOMALY DETECTIONTRENDS
APM
End user monitoring
Application Performance Management
DELIVERY PIPELINE METRICS - TEAM LEVEL DASHBOARD
LOGS
LOGS
▸ Set your log severity right
▸ Remember you will not be the only one reading these logs
▸ Track your communication with other systems
▸ Log exceptions correctly
▸ Use an ID to track your events
▸ Normalize yours logs
▸ Separate and Centralize your Log Data
LOGS
DOCUMENT, MEASURE,
COMMUNICATE AND
COLLABORATE
DOCUMENT
WRITING EFFECTIVE DOCUMENTATION
▸ Treat documentation like a requirement
▸ Put the Information in the Most Appropriate Place
▸ *Keep your documentation close to the code
▸ Keep documentation just simple enough, but not too
simple
▸ Avoid creating large volumes of documentation of questionable
detail which become obsolete shortly after they are written
MEASURE
MEASURE PROGRESS
▸ KPIs
▸ Deployment frequency
▸ Deployment speed
▸ Failure rate
▸ Time from code to prod
▸ Share goals
▸ Increase the visibility of work
COMMUNICATE
▸ Find better ways to communicante
▸ Promote day to day direct communication
▸ Coffee machine meetings
▸ Chat (Slack, HipChat, Team,…)
▸ Use video (Slack, Skype, Hangout,..)
▸ Quality over quantity
▸ Post Mortem meetings
▸ Bootcamps,…
IN DEVOPS CULTURE, COMMUNICATION AND COLLABORATION ARE KEY
COLLABORATE
TIPS FOR BETTER COLLABORATION
▸ Change the seating and rotate people through development
▸ Improve the environment for collaboration (Breakout areas,
whiteboards, nice meeting rooms, a decent coffee machine)
▸ Cross domain workshops for analyze your process
▸ Block time for process improvement work (lean)
▸ Run lunch and learn / brown bag sessions
▸ Put a persistent chat room in place
Demis Rizzotto
demis.r@me.com
@DemisRiz
THANKS YOU!
QUESTIONS?

Devops (start walking in the same direction) by ops

  • 1.
    DEVOPS START WALKING INTHE SAME DIRECTION
  • 2.
    WHO I AM? DemisRizzotto System Engineer,DevOps Linux Engineer @Exxoss Currently consultant @Lampiris
  • 3.
  • 4.
    DIAMETRICALLY OPPOSED GOALSAND INCENTIVES DEV ▸ Respond to the rapidly changing competitive landscape OPS ▸ Provide stable, reliable, and secure service to the customer THE CORE, CHRONIC CONFLICT WHEN ORGANIZATIONAL MEASUREMENTS AND INCENTIVES ACROSS DIFFERENT SILOS PREVENT THE ACHIEVEMENT OF GLOBAL, ORGANIZATIONAL GOALS
  • 5.
    FRUSTRATION ▸Unnecessary process steps ▸Unnecessaryrework ▸Unnecessary features ▸Waiting for someone else to act on an action or task ▸Waiting for an approval ▸Waiting for an environment ▸Creating a ticket for a manual task ▸Reporting to management by manually updating spreadsheets ▸Manual status reporting ▸…
  • 6.
    RESULTS ▸ Fragile applicationare prone to failure ▸ Ever increasing backlog and technical debt ▸ Urgent security rework and remediation ▸ More urgent projects into the queue ▸ Big deployments very difficult to diagnose when fail
  • 7.
  • 8.
    DEVOPS DEV OPS FACILITATE ANDAUTOMATE DEPLOYMENT HELP DEV TEAM BECOME AUTONOMUS FACILITATE DIAGNOSTIC GET DEVS INTERESTED IN « PROD STUFF » GIVE ACCESS TO LOGS GIVE ACCESS TO MONITORING OPTIMIZE LOGS DEFINE METRICS QUALITY TESTS MONITOR YOUR APPS
  • 9.
  • 10.
    DEV IN DEVOPS IMPROVEMENTS ▸The ability to accurately track dependencies ▸ Services that are resilient and degrade gracefully ▸ The ability to archive data to manage the size of the production data set ▸ The ability to easily search and understand log messages across services ▸ The ability to trace requests from users through multiple services ▸ Simple, centralized runtime configuration using feature flags, …
  • 11.
    DEV IN DEVOPS ▸Make Dev responsible for their own services ▸ Integrate Dev into the incident management processes ▸ Have Dev cross-train Ops ▸ Embed Ops knowledge and capabilities into Dev ▸ Design for IT Operations EMBED IT OPERATIONS INTO DEVELOPMENT
  • 12.
  • 13.
    OPS IN DEVOPS IMPROVEMENTS ▸Self service ▸ Infrastructure as code ▸ Standardize infrastructure ▸ Make all infrastructure data visible ▸ Make application info data visible ▸ Modify the incident resolution process and blameless post-mortems ▸ Monitor the health of the deployment pipelines
  • 14.
    OPS IN DEVOPS ▸Single “repository of truth” containing both the code and environments ▸ Create the one-step Dev, Test and Production environment build process ▸ Extend the deployment pipeline processes into production EXTEND DEVELOPMENT INTO IT OPERATIONS
  • 15.
    WE ARE ALLDEVOPS WE ARE ALL DEVOPS ▸ Stop think . Do ▸ Share responsability ▸ It’s Never Someone Else’s Problem ▸ Dedicated Designated Ops in every team
  • 16.
    ▸ Different typeof devops, but devops is possible for all type and size of companies ▸ Iterate! ▸ Identify bottleneck and remediate ▸ Define small target and share results DEVOPS IS FOR ALL DEVOPS IS FOR ALL
  • 17.
    DEVOPS IS FORALL ▸ AUTONOMUS TEAMS ▸ Trust but verify ▸ Local autonomy, but global consistency ▸ SELF SERVICE ▸ How ? API, web portal, infra as code , AWS accounts, … ▸ AUTOMATE ▸ If you are tired to do it manually. Automate it. ▸ REDUCE NON-VALUE ADDED ▸ If isn't your business it's maybe better don't do it your self KEY OF SUCCESS : REDUCE FRUSTRATION
  • 18.
  • 19.
    ON DEMAND (IAAS ) EASIER TO REBUILD THAN TO REPAIR INFRASTUCTURE
  • 20.
    INFRASTRUCTURE IMMUTABLE INFRASTRUCTURE ▸ Automatethe setup and deployment for every part and every layer of your infrastructure. ▸ Never change any part of your system once it is deployed. If you need to change it, deploy a new system. AVOID HISTORY
  • 21.
    INFRASTRUCTURE INFRASTRUCTURE AS CODE Versionningand code review Easier track changes Bootstrap your infrastructure from scratch Automate
  • 22.
  • 23.
  • 24.
    TESTING CONTINUOUS TESTING ▸ Automatedtests ▸ code quality ▸ Unit tests ▸ Stress tests ▸ Don’t forget to test infrastructure (load banacer, recovery etc.) ▸ Extend you build pipeline
  • 25.
  • 26.
    SECURITY INFORMATION SECURITY ASEVERYONE’S JOB, EVERY DAY ▸ Updates ▸ Use secrets management ▸ Integrate security into our CI/CD pipeline ▸ Check code libraries and their recommended configurations ▸ Dependency scanning ▸ Static analysis ▸ Dynamic analysis (ex: OWASP Zed Attack Proxy + Selenium)
  • 27.
    SECURITY SECURITY METRICS MONITORING ▸Successful and unsuccessful user logins ▸ User password resets ▸ User email address resets ▸ Database syntax error ▸ User credit card changes
  • 28.
  • 29.
    DEPLOYMENTS DEPLOYMENTS ARE ROUTINEAND PREDICTABLE No stress . It’s a routine. On business day when everyone is already in the office and without our customers even noticing
  • 30.
    DEPLOYMENTS HOW ▸ Work insmall batches and delivering quickly and frequently ▸ Blue green deployment ▸ Feature toggle ▸ Canary testing ▸ Dark launch ▸ Automated deployments
  • 31.
  • 32.
    ALERTING ▸ Mesure servicehealth not only server health (Health pattern) ▸ Reducing Alert Fatigue ▸ Cut alerts that aren’t actionable ▸ Adjust thresholds ▸ Consolidate related alerts ▸ Give alerts relevant names & descriptions ▸ Make sure the right people are getting alerts ▸ Multiple notification rules and choose right channel ▸ Use incident management tools(Pagerduty, Victoryops,..)
  • 33.
    MONITORING IF IT MOVES,TRACK IT ANOMALY DETECTIONTRENDS
  • 34.
  • 35.
    DELIVERY PIPELINE METRICS- TEAM LEVEL DASHBOARD
  • 36.
  • 37.
    LOGS ▸ Set yourlog severity right ▸ Remember you will not be the only one reading these logs ▸ Track your communication with other systems ▸ Log exceptions correctly ▸ Use an ID to track your events ▸ Normalize yours logs ▸ Separate and Centralize your Log Data
  • 38.
  • 39.
  • 40.
    DOCUMENT WRITING EFFECTIVE DOCUMENTATION ▸Treat documentation like a requirement ▸ Put the Information in the Most Appropriate Place ▸ *Keep your documentation close to the code ▸ Keep documentation just simple enough, but not too simple ▸ Avoid creating large volumes of documentation of questionable detail which become obsolete shortly after they are written
  • 41.
    MEASURE MEASURE PROGRESS ▸ KPIs ▸Deployment frequency ▸ Deployment speed ▸ Failure rate ▸ Time from code to prod ▸ Share goals ▸ Increase the visibility of work
  • 42.
    COMMUNICATE ▸ Find betterways to communicante ▸ Promote day to day direct communication ▸ Coffee machine meetings ▸ Chat (Slack, HipChat, Team,…) ▸ Use video (Slack, Skype, Hangout,..) ▸ Quality over quantity ▸ Post Mortem meetings ▸ Bootcamps,… IN DEVOPS CULTURE, COMMUNICATION AND COLLABORATION ARE KEY
  • 43.
    COLLABORATE TIPS FOR BETTERCOLLABORATION ▸ Change the seating and rotate people through development ▸ Improve the environment for collaboration (Breakout areas, whiteboards, nice meeting rooms, a decent coffee machine) ▸ Cross domain workshops for analyze your process ▸ Block time for process improvement work (lean) ▸ Run lunch and learn / brown bag sessions ▸ Put a persistent chat room in place
  • 44.

Editor's Notes

  • #4 Spirale NEGATIVE
  • #5 DEV: changes into production as quickly as possible OPS: stable, reliable, and secure, making it difficult or even impossible for anyone to introduce production changes that could jeopardize (Problème comptable et objectifs différents entre dev et ops ) 
  • #6 même en tant que ops j’en peut plus d’attendre la création d’un environment
  • #9 DevOps ne signifie pas q’un dev peut se passer des ops.Chauqun son metirer ; Collaborer Exemple de spécialisation .
  • #11 Quick fix ok , mais il faut ensuite il faut améliorer
  • #14 serveur = commodité (EX : prise de courant éléc) STOP Reboot
  • #15 SINGLE REPOSITORY : application code , dependencies (e.g., libraries, static content, etc.), infrastructure, deploy, Any script used to create database schemas, application reference data, etc, All the environment creation tools and artifacts described in the previous step (e.g., VMware or AMI images, Puppet or Chef recipes,
  • #17 What problems are we trying to solve?
  • #18 REDUCE NON-VALUE ADDED : Refaire le Nieme service d’authentification, ou de gestion de fichiers, gestion des logs, IAAS , …
  • #20 ON DEMAND EASIER TO REBUILD THAN TO REPAIR
  • #23 Pipeline Important (aussi pour les Ops)
  • #27 Mise à jour système ( Infra as code) , java, librairies ,… Open Web Application Security Project
  • #28 Bcp d’infos dans les logs pour avoir une vue « sécurité »
  • #30 Instead of starting deployments at midnight on Friday and spending all weekend working to complete them, deployments occur throughout the business day when everyone is already in the office and without our customers even noticing
  • #31 Automatiser! Ansible, kubernetes, simple shell script, git,… Exemple Facebook (dark launch)
  • #33 Notifier uniquement si nécessaire Moins de notifications plus d’actions
  • #34 Analyser vos métriques
  • #36 Si vous voulez aller plus loin et mieux comprendre les interactions
  • #37 Vous devez aussi monitorer votre CI/CD
  • #38 Ops: partages les logs!!! On perd juste du temps avec la boucle infernale des logs « protégées »
  • #39 Utilises plus les logs! tenez compte que vous n’êtes pas les seuls à les lire! Mais c’est difficile d’examiner des fichiers de lois pour les humains donc…
  • #40 From TEXT to visual data! Vous pouvez aller plus loin que des fichiers texte
  • #42 La documentation c’est important SWAGGER API
  • #43 Mesurez pour mieux comprendre ce qui change
  • #44 Promote