SlideShare a Scribd company logo
1 of 21
Download to read offline
DevOps Risk Mitigation
www.immobilienscout24.de
Berlin | 23.07.2014 | Schlomo Schapiro
Systems Architect, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
Test Driven Infrastructure
@schlomoschapiro
Expensive
Fix
Cheap
Fix
Costs Of Finding
Bugs In Production
Plan
Design
Budget
Develop Test
RUN
DEV
Plan
Design
Budget
Develop Test RUNOPS
TIME
Plan
Design
Budget
Develop Test
Plan
Design
Budget
Develop Test RUN
Proxy
config
Database
borked
„Buy Now“
button broken
MTA drop
all mail
Load Balancer
Configuration
Everything costs
only 0 €
Login possible
without password
NFS not
available
DB Replication
stopped
No ads
shown
Broken
CSS / JS
Tomcat won't
start/stop
Service user
not defined
sudoers
invalid
DevOps: Respect & Learning
DEV learn
from OPS to
think about:
●
Resources (CPU, RAM, Disk)
●
Services (Start, Stop, Status)
●
Dependencies (Start DB before App)
●
Logfiles (Rotate, Remove)
●
Disk Space
●
Monitoring and Alarming
●
...
OPS learn
from DEV to
think about:
●
Incremental Improvement
●
Infrastructure as Code
●
Version Control System
●
Coding (OO, Functions, Libraries …)
●
Code Quality
●
Unit & Integration Tests
●
Test Automation
●
...
Untested
=
Broken
Unit Tests
Test the smallest possible components in an
artifical environment.
System Tests
Test the entire application in a real(istic)
environment together with other
applications.
Part of build process
Syntax checks
 Scripts
 Config Files
 Data Files
Unit tests for functions/libs
Run program with test data
 Check result
 Check program behaviour
with wrong/broken test data
Also run on Developer desktop
Quick feedback (~ seconds)
Install on test server
Run tests from outside
 HTTP calls
 Send emails
 Try to login
Run tests from inside
 Remote Exec (rsh, ssh …)
http://go.schapiro.org/rshpitfall
 Service Start, Stop & Status
 Modify server to create
good & bad test scenarios
 Reboot
Unit Tests System Tests
Unit
Tests
%prep
%setup -q
%install
install … %{buildroot}/…
install … %{buildroot}/…
%files
%defattr(-,root,root,-)
/...
BuildRequires: sudo
%build
set -e
visudo -c -f sudoers
%install
install -m 0440 sudoers -D 
%{buildroot}/etc/suoders.d/%{name}
%files%defattr(-,root,root,-)
/etc/suoders.d/%{name}
BuildRequires: PyYAML, pylint
%build
set -e
# syntax checks
bash -n my_script.sh
# Should be valid python code
pylint -E yum-repo-propagate
# should be valid YAML file
python -c "↩
import yaml↩
yaml.safe_load(open('config.yaml'))↩
"
System
Tests
http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Create RPM
●
Source
Code DEV
YUM
Repo
Upload
PRO
YUM
Repo
Propagate
RPM
Test Server
Deploy
and
Run
TestJob
yum
Prod Server
Deploy
to PROD
yum
Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Create RPM
●
Source
Code DEV
YUM
Repo
Upload
PRO
YUM
Repo
Propagate
RPM
Test Server
Deploy
and
Run
TestJob
yum
Prod Server
Deploy
to PROD
yum
SAN mount service
Test via rsh
Mock SAN devices
with losetup
service start, stop
mounts/umounts
Error handling
HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all function groups!
Test Server
GET http://external.com/
X-Forwarded-For: 10.11.12.01
502 Bad Gateway✔
GET http://external.com/
X-Forwarded-For: 10.34.56.01
403 Forbidden
ERR_ACCESS_DENIED by proxy server
✘
✘
VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and install Linux OS
Scrape VM screen via OCR
http://github.com/Immobilienscout24/lab-manager-light
34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready.
Automate the delivery chain from source till production.
Low Risk – Lots of Fun
http://go.schapiro.org/slides
Kontakt:
Immobilien Scout GmbH
Andreasstraße 10
10243 Berlin
Fon: +49 30 243 01-1229
Email: schlomo.schapiro@immobilienscout24.de
URL: www.immobilienscout24.de
Thank you very much!
Please contact me for further
questions and discussions.

More Related Content

What's hot

Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...
Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...
Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...Eran Stiller
 
Vered Flis: Because performance matters! Architecture Next 20
Vered Flis: Because performance matters! Architecture Next 20Vered Flis: Because performance matters! Architecture Next 20
Vered Flis: Because performance matters! Architecture Next 20CodeValue
 
Building Serverless Web Applications with OpenWhisk
Building Serverless Web Applications with OpenWhiskBuilding Serverless Web Applications with OpenWhisk
Building Serverless Web Applications with OpenWhiskNiklas Heidloff
 
Create Alexa Skills using IBM Watson Conversation and Apache OpenWhisk
Create Alexa Skills using IBM Watson Conversation and Apache OpenWhiskCreate Alexa Skills using IBM Watson Conversation and Apache OpenWhisk
Create Alexa Skills using IBM Watson Conversation and Apache OpenWhisk Niklas Heidloff
 
Cloud native principles
Cloud native principlesCloud native principles
Cloud native principlesDiego Pacheco
 
Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...
Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...
Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...CodeValue
 

What's hot (9)

Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...
Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...
Net Conf Israel - Intro & Building Cloud Native Apps with .NET Core 3.0 and K...
 
Vered Flis: Because performance matters! Architecture Next 20
Vered Flis: Because performance matters! Architecture Next 20Vered Flis: Because performance matters! Architecture Next 20
Vered Flis: Because performance matters! Architecture Next 20
 
Building Serverless Web Applications with OpenWhisk
Building Serverless Web Applications with OpenWhiskBuilding Serverless Web Applications with OpenWhisk
Building Serverless Web Applications with OpenWhisk
 
Create Alexa Skills using IBM Watson Conversation and Apache OpenWhisk
Create Alexa Skills using IBM Watson Conversation and Apache OpenWhiskCreate Alexa Skills using IBM Watson Conversation and Apache OpenWhisk
Create Alexa Skills using IBM Watson Conversation and Apache OpenWhisk
 
Generative models in the arts
Generative models in the artsGenerative models in the arts
Generative models in the arts
 
Cloud native principles
Cloud native principlesCloud native principles
Cloud native principles
 
Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...
Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...
Nir Doboviski: In Space No One Can Hear Microservices Scream – a Microservice...
 
use case ibm k8s_service+devops
use case ibm k8s_service+devopsuse case ibm k8s_service+devops
use case ibm k8s_service+devops
 
GitHub Actions 101
GitHub Actions 101GitHub Actions 101
GitHub Actions 101
 

Viewers also liked

Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24Schlomo Schapiro
 
OSDC 2016 - Hybrid Cloud - A Cloud Migration Strategy
OSDC 2016 - Hybrid Cloud - A Cloud Migration StrategyOSDC 2016 - Hybrid Cloud - A Cloud Migration Strategy
OSDC 2016 - Hybrid Cloud - A Cloud Migration StrategySchlomo Schapiro
 
DevOps, Agile and Open Source at ImmobilienScout24
DevOps, Agile and Open Source at ImmobilienScout24DevOps, Agile and Open Source at ImmobilienScout24
DevOps, Agile and Open Source at ImmobilienScout24Schlomo Schapiro
 
PyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugen
PyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugenPyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugen
PyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugenSchlomo Schapiro
 
GUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital Age
GUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital AgeGUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital Age
GUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital AgeSchlomo Schapiro
 
Rational Theology of Judaism - Introduction
Rational Theology of Judaism - IntroductionRational Theology of Judaism - Introduction
Rational Theology of Judaism - Introductionhellaschapiro
 

Viewers also liked (6)

Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
 
OSDC 2016 - Hybrid Cloud - A Cloud Migration Strategy
OSDC 2016 - Hybrid Cloud - A Cloud Migration StrategyOSDC 2016 - Hybrid Cloud - A Cloud Migration Strategy
OSDC 2016 - Hybrid Cloud - A Cloud Migration Strategy
 
DevOps, Agile and Open Source at ImmobilienScout24
DevOps, Agile and Open Source at ImmobilienScout24DevOps, Agile and Open Source at ImmobilienScout24
DevOps, Agile and Open Source at ImmobilienScout24
 
PyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugen
PyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugenPyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugen
PyCon 2013 - Open Source Sponsoring - und den Chef damit überzeugen
 
GUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital Age
GUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital AgeGUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital Age
GUUG FFG 2017 - DevOps for Everybody - A Workplace Strategy for the Digital Age
 
Rational Theology of Judaism - Introduction
Rational Theology of Judaism - IntroductionRational Theology of Judaism - Introduction
Rational Theology of Judaism - Introduction
 

Similar to EuroPython 2014 Devops Risk Mitigation

PyCon 2013 Test Driven Infrastructure
PyCon 2013 Test Driven InfrastructurePyCon 2013 Test Driven Infrastructure
PyCon 2013 Test Driven InfrastructureSchlomo Schapiro
 
OSDC 2014 Test Driven Infrastructure
OSDC 2014 Test Driven InfrastructureOSDC 2014 Test Driven Infrastructure
OSDC 2014 Test Driven InfrastructureSchlomo Schapiro
 
OSDC 2014: Schlomo Schapiro - Test Driven Infrastructure
OSDC 2014: Schlomo Schapiro -  Test Driven InfrastructureOSDC 2014: Schlomo Schapiro -  Test Driven Infrastructure
OSDC 2014: Schlomo Schapiro - Test Driven InfrastructureNETWAYS
 
Seven steps to better security
Seven steps to better securitySeven steps to better security
Seven steps to better securityMichael Pignataro
 
Continuous Integration using Cruise Control
Continuous Integration using Cruise ControlContinuous Integration using Cruise Control
Continuous Integration using Cruise Controlelliando dias
 
SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability EnvironmentsSaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability EnvironmentsSaltStack
 
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...WebCamp
 
Making the most of your Test Suite
Making the most of your Test SuiteMaking the most of your Test Suite
Making the most of your Test Suiteericholscher
 
PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agents
PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agentsPVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agents
PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agentsAndrey Karpov
 
Good practices for debugging Selenium and Appium tests
Good practices for debugging Selenium and Appium testsGood practices for debugging Selenium and Appium tests
Good practices for debugging Selenium and Appium testsAbhijeet Vaikar
 
Groovy for System Administrators
Groovy for System AdministratorsGroovy for System Administrators
Groovy for System AdministratorsDaniel Woods
 
Tools and Process for Streamlining Mac Deployment
Tools and Process for Streamlining Mac DeploymentTools and Process for Streamlining Mac Deployment
Tools and Process for Streamlining Mac DeploymentTimothy Sutton
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsGraham Dumpleton
 
Profiling PHP with Xdebug / Webgrind
Profiling PHP with Xdebug / WebgrindProfiling PHP with Xdebug / Webgrind
Profiling PHP with Xdebug / WebgrindSam Keen
 
CCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShell
CCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShellCCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShell
CCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShellwalk2talk srl
 
Website Testing Practices
Website Testing PracticesWebsite Testing Practices
Website Testing Practicesdeseomar
 
Windows Azure Web Sites - Things they don't teach kids in school - NDC London
Windows Azure Web Sites - Things they don't teach kids in school - NDC LondonWindows Azure Web Sites - Things they don't teach kids in school - NDC London
Windows Azure Web Sites - Things they don't teach kids in school - NDC LondonMaarten Balliauw
 

Similar to EuroPython 2014 Devops Risk Mitigation (20)

PyCon 2013 Test Driven Infrastructure
PyCon 2013 Test Driven InfrastructurePyCon 2013 Test Driven Infrastructure
PyCon 2013 Test Driven Infrastructure
 
OSDC 2014 Test Driven Infrastructure
OSDC 2014 Test Driven InfrastructureOSDC 2014 Test Driven Infrastructure
OSDC 2014 Test Driven Infrastructure
 
OSDC 2014: Schlomo Schapiro - Test Driven Infrastructure
OSDC 2014: Schlomo Schapiro -  Test Driven InfrastructureOSDC 2014: Schlomo Schapiro -  Test Driven Infrastructure
OSDC 2014: Schlomo Schapiro - Test Driven Infrastructure
 
Seven steps to better security
Seven steps to better securitySeven steps to better security
Seven steps to better security
 
Continuous Integration using Cruise Control
Continuous Integration using Cruise ControlContinuous Integration using Cruise Control
Continuous Integration using Cruise Control
 
SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability EnvironmentsSaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
 
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
 
Making the most of your Test Suite
Making the most of your Test SuiteMaking the most of your Test Suite
Making the most of your Test Suite
 
Deployment talk dpc 13
Deployment talk dpc 13Deployment talk dpc 13
Deployment talk dpc 13
 
PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agents
PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agentsPVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agents
PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agents
 
Good practices for debugging Selenium and Appium tests
Good practices for debugging Selenium and Appium testsGood practices for debugging Selenium and Appium tests
Good practices for debugging Selenium and Appium tests
 
Groovy for System Administrators
Groovy for System AdministratorsGroovy for System Administrators
Groovy for System Administrators
 
Tools and Process for Streamlining Mac Deployment
Tools and Process for Streamlining Mac DeploymentTools and Process for Streamlining Mac Deployment
Tools and Process for Streamlining Mac Deployment
 
My Saminar On Php
My Saminar On PhpMy Saminar On Php
My Saminar On Php
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web Applications
 
Profiling PHP with Xdebug / Webgrind
Profiling PHP with Xdebug / WebgrindProfiling PHP with Xdebug / Webgrind
Profiling PHP with Xdebug / Webgrind
 
CCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShell
CCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShellCCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShell
CCI2018 - Automatizzare la creazione di risorse con ARM template e PowerShell
 
Website Testing Practices
Website Testing PracticesWebsite Testing Practices
Website Testing Practices
 
Revoke-Obfuscation
Revoke-ObfuscationRevoke-Obfuscation
Revoke-Obfuscation
 
Windows Azure Web Sites - Things they don't teach kids in school - NDC London
Windows Azure Web Sites - Things they don't teach kids in school - NDC LondonWindows Azure Web Sites - Things they don't teach kids in school - NDC London
Windows Azure Web Sites - Things they don't teach kids in school - NDC London
 

More from Schlomo Schapiro

The Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo Schapiro
The Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo SchapiroThe Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo Schapiro
The Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo SchapiroSchlomo Schapiro
 
The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11
The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11
The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11Schlomo Schapiro
 
The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...
The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...
The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...Schlomo Schapiro
 
Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...
Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...
Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...Schlomo Schapiro
 
The GitOps Journey - GitOpsCon EU 2021 - Schlomo Schapiro
The GitOps Journey - GitOpsCon EU 2021 - Schlomo SchapiroThe GitOps Journey - GitOpsCon EU 2021 - Schlomo Schapiro
The GitOps Journey - GitOpsCon EU 2021 - Schlomo SchapiroSchlomo Schapiro
 
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo SchapiroCompliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo SchapiroSchlomo Schapiro
 
The Devops Driving School - DevOps Gathering 2019 - Schlomo Schapiro
The Devops Driving School - DevOps Gathering 2019 - Schlomo SchapiroThe Devops Driving School - DevOps Gathering 2019 - Schlomo Schapiro
The Devops Driving School - DevOps Gathering 2019 - Schlomo SchapiroSchlomo Schapiro
 
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...Schlomo Schapiro
 
DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...
DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...
DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...Schlomo Schapiro
 
Root for all - measuring DevOps adoption - microxchg 2018 - Schlomo Schapiro
Root for all - measuring DevOps adoption - microxchg 2018 - Schlomo SchapiroRoot for all - measuring DevOps adoption - microxchg 2018 - Schlomo Schapiro
Root for all - measuring DevOps adoption - microxchg 2018 - Schlomo SchapiroSchlomo Schapiro
 
WARNING is a waste of my time
WARNING is a waste of my timeWARNING is a waste of my time
WARNING is a waste of my timeSchlomo Schapiro
 
SE 2015 DevOps Risk Mitigation - Test Driven Infrastructure
SE 2015 DevOps Risk Mitigation - Test Driven InfrastructureSE 2015 DevOps Risk Mitigation - Test Driven Infrastructure
SE 2015 DevOps Risk Mitigation - Test Driven InfrastructureSchlomo Schapiro
 
EuroPython 2014 YAML Reader Lightning Talk
EuroPython 2014 YAML Reader Lightning TalkEuroPython 2014 YAML Reader Lightning Talk
EuroPython 2014 YAML Reader Lightning TalkSchlomo Schapiro
 
PyCon 2013 - Distributed Monitoring Configuration
PyCon 2013 - Distributed Monitoring ConfigurationPyCon 2013 - Distributed Monitoring Configuration
PyCon 2013 - Distributed Monitoring ConfigurationSchlomo Schapiro
 
LinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI Systems
LinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI SystemsLinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI Systems
LinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI SystemsSchlomo Schapiro
 
Linux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADTLinux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADTSchlomo Schapiro
 

More from Schlomo Schapiro (16)

The Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo Schapiro
The Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo SchapiroThe Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo Schapiro
The Role of GitOps in IT-Strategy v2 - July 2022 - Schlomo Schapiro
 
The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11
The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11
The GitOps Journey - Schlomo Schapiro - Berlin DevOps Meetup 2021-11
 
The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...
The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...
The Role of GitOps in IT-Strategy - November 2021 - Schlomo Schapiro - Contin...
 
Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...
Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...
Schlomo Schapiro - Why I like to use the proprietary Cloud services without f...
 
The GitOps Journey - GitOpsCon EU 2021 - Schlomo Schapiro
The GitOps Journey - GitOpsCon EU 2021 - Schlomo SchapiroThe GitOps Journey - GitOpsCon EU 2021 - Schlomo Schapiro
The GitOps Journey - GitOpsCon EU 2021 - Schlomo Schapiro
 
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo SchapiroCompliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
 
The Devops Driving School - DevOps Gathering 2019 - Schlomo Schapiro
The Devops Driving School - DevOps Gathering 2019 - Schlomo SchapiroThe Devops Driving School - DevOps Gathering 2019 - Schlomo Schapiro
The Devops Driving School - DevOps Gathering 2019 - Schlomo Schapiro
 
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
 
DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...
DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...
DevOps + Continuous Delivery + Cloud: The Three Drivers of Enterprise Agility...
 
Root for all - measuring DevOps adoption - microxchg 2018 - Schlomo Schapiro
Root for all - measuring DevOps adoption - microxchg 2018 - Schlomo SchapiroRoot for all - measuring DevOps adoption - microxchg 2018 - Schlomo Schapiro
Root for all - measuring DevOps adoption - microxchg 2018 - Schlomo Schapiro
 
WARNING is a waste of my time
WARNING is a waste of my timeWARNING is a waste of my time
WARNING is a waste of my time
 
SE 2015 DevOps Risk Mitigation - Test Driven Infrastructure
SE 2015 DevOps Risk Mitigation - Test Driven InfrastructureSE 2015 DevOps Risk Mitigation - Test Driven Infrastructure
SE 2015 DevOps Risk Mitigation - Test Driven Infrastructure
 
EuroPython 2014 YAML Reader Lightning Talk
EuroPython 2014 YAML Reader Lightning TalkEuroPython 2014 YAML Reader Lightning Talk
EuroPython 2014 YAML Reader Lightning Talk
 
PyCon 2013 - Distributed Monitoring Configuration
PyCon 2013 - Distributed Monitoring ConfigurationPyCon 2013 - Distributed Monitoring Configuration
PyCon 2013 - Distributed Monitoring Configuration
 
LinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI Systems
LinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI SystemsLinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI Systems
LinuxTag 2013 Relax and Recover - Disaster Recovery for UEFI Systems
 
Linux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADTLinux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADT
 

Recently uploaded

SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predieusebiomeyer
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxmibuzondetrabajo
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书rnrncn29
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxAndrieCagasanAkio
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxNIMMANAGANTI RAMAKRISHNA
 
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书rnrncn29
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxMario
 
Cybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best PracticesCybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best PracticesLumiverse Solutions Pvt Ltd
 

Recently uploaded (9)

SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predi
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptx
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptx
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptx
 
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptx
 
Cybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best PracticesCybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best Practices
 

EuroPython 2014 Devops Risk Mitigation

  • 1. DevOps Risk Mitigation www.immobilienscout24.de Berlin | 23.07.2014 | Schlomo Schapiro Systems Architect, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ Test Driven Infrastructure @schlomoschapiro
  • 4. Plan Design Budget Develop Test Plan Design Budget Develop Test RUN Proxy config Database borked „Buy Now“ button broken MTA drop all mail Load Balancer Configuration Everything costs only 0 € Login possible without password NFS not available DB Replication stopped No ads shown Broken CSS / JS Tomcat won't start/stop Service user not defined sudoers invalid
  • 5. DevOps: Respect & Learning DEV learn from OPS to think about: ● Resources (CPU, RAM, Disk) ● Services (Start, Stop, Status) ● Dependencies (Start DB before App) ● Logfiles (Rotate, Remove) ● Disk Space ● Monitoring and Alarming ● ... OPS learn from DEV to think about: ● Incremental Improvement ● Infrastructure as Code ● Version Control System ● Coding (OO, Functions, Libraries …) ● Code Quality ● Unit & Integration Tests ● Test Automation ● ...
  • 7. Unit Tests Test the smallest possible components in an artifical environment. System Tests Test the entire application in a real(istic) environment together with other applications.
  • 8. Part of build process Syntax checks  Scripts  Config Files  Data Files Unit tests for functions/libs Run program with test data  Check result  Check program behaviour with wrong/broken test data Also run on Developer desktop Quick feedback (~ seconds) Install on test server Run tests from outside  HTTP calls  Send emails  Try to login Run tests from inside  Remote Exec (rsh, ssh …) http://go.schapiro.org/rshpitfall  Service Start, Stop & Status  Modify server to create good & bad test scenarios  Reboot Unit Tests System Tests
  • 10. %prep %setup -q %install install … %{buildroot}/… install … %{buildroot}/… %files %defattr(-,root,root,-) /...
  • 11. BuildRequires: sudo %build set -e visudo -c -f sudoers %install install -m 0440 sudoers -D %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-) /etc/suoders.d/%{name}
  • 12. BuildRequires: PyYAML, pylint %build set -e # syntax checks bash -n my_script.sh # Should be valid python code pylint -E yum-repo-propagate # should be valid YAML file python -c "↩ import yaml↩ yaml.safe_load(open('config.yaml'))↩ "
  • 14. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  • 15. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  • 16. SAN mount service Test via rsh Mock SAN devices with losetup service start, stop mounts/umounts Error handling
  • 17. HTTP Proxy Configuration X-Forwarded-For header spoofs source Check result for ERR_ACCESS_DENIED Run tests for all function groups! Test Server GET http://external.com/ X-Forwarded-For: 10.11.12.01 502 Bad Gateway✔ GET http://external.com/ X-Forwarded-For: 10.34.56.01 403 Forbidden ERR_ACCESS_DENIED by proxy server ✘ ✘
  • 18. VM Provisioning & Kickstart Installation Test via HTTP API Create broken VMs and check error reports Create valid VM and install Linux OS Scrape VM screen via OCR http://github.com/Immobilienscout24/lab-manager-light
  • 19. 34 35 36 2 3 53 87 88 89 90 91 92 93 94 95 96 97 TIME Continous Live Deployment Deploy every application when it is ready. Automate the delivery chain from source till production.
  • 20. Low Risk – Lots of Fun http://go.schapiro.org/slides
  • 21. Kontakt: Immobilien Scout GmbH Andreasstraße 10 10243 Berlin Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.de URL: www.immobilienscout24.de Thank you very much! Please contact me for further questions and discussions.