Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DevOps Risk Mitigation
www.immobilienscout24.de
Berlin | 23.07.2014 | Schlomo Schapiro
Systems Architect, Open Source Evan...
Expensive
Fix
Cheap
Fix
Costs Of Finding
Bugs In Production
Plan
Design
Budget
Develop Test
RUN
DEV
Plan
Design
Budget
Develop Test RUNOPS
TIME
Plan
Design
Budget
Develop Test
Plan
Design
Budget
Develop Test RUN
Proxy
config
Database
borked
„Buy Now“
button broken
M...
DevOps: Respect & Learning
DEV learn
from OPS to
think about:
●
Resources (CPU, RAM, Disk)
●
Services (Start, Stop, Status...
Untested
=
Broken
Unit Tests
Test the smallest possible components in an
artifical environment.
System Tests
Test the entire application in ...
Part of build process
Syntax checks
 Scripts
 Config Files
 Data Files
Unit tests for functions/libs
Run program with t...
Unit
Tests
%prep
%setup -q
%install
install … %{buildroot}/…
install … %{buildroot}/…
%files
%defattr(-,root,root,-)
/...
BuildRequires: sudo
%build
set -e
visudo -c -f sudoers
%install
install -m 0440 sudoers -D 
%{buildroot}/etc/suoders.d/%{n...
BuildRequires: PyYAML, pylint
%build
set -e
# syntax checks
bash -n my_script.sh
# Should be valid python code
pylint -E y...
System
Tests
http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Crea...
Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Crea...
SAN mount service
Test via rsh
Mock SAN devices
with losetup
service start, stop
mounts/umounts
Error handling
HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all functio...
VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and i...
34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready....
Low Risk – Lots of Fun
http://go.schapiro.org/slides
Kontakt:
Immobilien Scout GmbH
Andreasstraße 10
10243 Berlin
Fon: +49 30 243 01-1229
Email: schlomo.schapiro@immobiliensco...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
Next

of

EuroPython 2014 Devops Risk Mitigation Slide 1

YouTube videos are no longer supported on SlideShare

View original on YouTube

EuroPython 2014 Devops Risk Mitigation Slide 3 EuroPython 2014 Devops Risk Mitigation Slide 4 EuroPython 2014 Devops Risk Mitigation Slide 5 EuroPython 2014 Devops Risk Mitigation Slide 6 EuroPython 2014 Devops Risk Mitigation Slide 7 EuroPython 2014 Devops Risk Mitigation Slide 8 EuroPython 2014 Devops Risk Mitigation Slide 9 EuroPython 2014 Devops Risk Mitigation Slide 10 EuroPython 2014 Devops Risk Mitigation Slide 11 EuroPython 2014 Devops Risk Mitigation Slide 12 EuroPython 2014 Devops Risk Mitigation Slide 13 EuroPython 2014 Devops Risk Mitigation Slide 14 EuroPython 2014 Devops Risk Mitigation Slide 15 EuroPython 2014 Devops Risk Mitigation Slide 16 EuroPython 2014 Devops Risk Mitigation Slide 17 EuroPython 2014 Devops Risk Mitigation Slide 18 EuroPython 2014 Devops Risk Mitigation Slide 19 EuroPython 2014 Devops Risk Mitigation Slide 20 EuroPython 2014 Devops Risk Mitigation Slide 21 EuroPython 2014 Devops Risk Mitigation Slide 22
Upcoming SlideShare
Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24
Next
Download to read offline and view in fullscreen.

Share

EuroPython 2014 Devops Risk Mitigation

Download to read offline

The (perceived) risk of the DevOps is that too many people get the right to "break" the platform.

Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others.

This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers.

See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

EuroPython 2014 Devops Risk Mitigation

  1. 1. DevOps Risk Mitigation www.immobilienscout24.de Berlin | 23.07.2014 | Schlomo Schapiro Systems Architect, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ Test Driven Infrastructure @schlomoschapiro
  2. 2. Expensive Fix Cheap Fix Costs Of Finding Bugs In Production
  3. 3. Plan Design Budget Develop Test RUN DEV Plan Design Budget Develop Test RUNOPS TIME
  4. 4. Plan Design Budget Develop Test Plan Design Budget Develop Test RUN Proxy config Database borked „Buy Now“ button broken MTA drop all mail Load Balancer Configuration Everything costs only 0 € Login possible without password NFS not available DB Replication stopped No ads shown Broken CSS / JS Tomcat won't start/stop Service user not defined sudoers invalid
  5. 5. DevOps: Respect & Learning DEV learn from OPS to think about: ● Resources (CPU, RAM, Disk) ● Services (Start, Stop, Status) ● Dependencies (Start DB before App) ● Logfiles (Rotate, Remove) ● Disk Space ● Monitoring and Alarming ● ... OPS learn from DEV to think about: ● Incremental Improvement ● Infrastructure as Code ● Version Control System ● Coding (OO, Functions, Libraries …) ● Code Quality ● Unit & Integration Tests ● Test Automation ● ...
  6. 6. Untested = Broken
  7. 7. Unit Tests Test the smallest possible components in an artifical environment. System Tests Test the entire application in a real(istic) environment together with other applications.
  8. 8. Part of build process Syntax checks  Scripts  Config Files  Data Files Unit tests for functions/libs Run program with test data  Check result  Check program behaviour with wrong/broken test data Also run on Developer desktop Quick feedback (~ seconds) Install on test server Run tests from outside  HTTP calls  Send emails  Try to login Run tests from inside  Remote Exec (rsh, ssh …) http://go.schapiro.org/rshpitfall  Service Start, Stop & Status  Modify server to create good & bad test scenarios  Reboot Unit Tests System Tests
  9. 9. Unit Tests
  10. 10. %prep %setup -q %install install … %{buildroot}/… install … %{buildroot}/… %files %defattr(-,root,root,-) /...
  11. 11. BuildRequires: sudo %build set -e visudo -c -f sudoers %install install -m 0440 sudoers -D %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-) /etc/suoders.d/%{name}
  12. 12. BuildRequires: PyYAML, pylint %build set -e # syntax checks bash -n my_script.sh # Should be valid python code pylint -E yum-repo-propagate # should be valid YAML file python -c "↩ import yaml↩ yaml.safe_load(open('config.yaml'))↩ "
  13. 13. System Tests http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
  14. 14. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  15. 15. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  16. 16. SAN mount service Test via rsh Mock SAN devices with losetup service start, stop mounts/umounts Error handling
  17. 17. HTTP Proxy Configuration X-Forwarded-For header spoofs source Check result for ERR_ACCESS_DENIED Run tests for all function groups! Test Server GET http://external.com/ X-Forwarded-For: 10.11.12.01 502 Bad Gateway✔ GET http://external.com/ X-Forwarded-For: 10.34.56.01 403 Forbidden ERR_ACCESS_DENIED by proxy server ✘ ✘
  18. 18. VM Provisioning & Kickstart Installation Test via HTTP API Create broken VMs and check error reports Create valid VM and install Linux OS Scrape VM screen via OCR http://github.com/Immobilienscout24/lab-manager-light
  19. 19. 34 35 36 2 3 53 87 88 89 90 91 92 93 94 95 96 97 TIME Continous Live Deployment Deploy every application when it is ready. Automate the delivery chain from source till production.
  20. 20. Low Risk – Lots of Fun http://go.schapiro.org/slides
  21. 21. Kontakt: Immobilien Scout GmbH Andreasstraße 10 10243 Berlin Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.de URL: www.immobilienscout24.de Thank you very much! Please contact me for further questions and discussions.
  • arturmartins

    May. 23, 2015
  • Aquarius1973

    Jul. 23, 2014

The (perceived) risk of the DevOps is that too many people get the right to "break" the platform. Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others. This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers. See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.

Views

Total views

3,340

On Slideshare

0

From embeds

0

Number of embeds

1,801

Actions

Downloads

22

Shares

0

Comments

0

Likes

2

×