The (perceived) risk of the DevOps is that too many people get the right to "break" the platform.
Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others.
This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers.
See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.
4. Plan
Design
Budget
Develop Test
Plan
Design
Budget
Develop Test RUN
Proxy
config
Database
borked
„Buy Now“
button broken
MTA drop
all mail
Load Balancer
Configuration
Everything costs
only 0 €
Login possible
without password
NFS not
available
DB Replication
stopped
No ads
shown
Broken
CSS / JS
Tomcat won't
start/stop
Service user
not defined
sudoers
invalid
5. DevOps: Respect & Learning
DEV learn
from OPS to
think about:
●
Resources (CPU, RAM, Disk)
●
Services (Start, Stop, Status)
●
Dependencies (Start DB before App)
●
Logfiles (Rotate, Remove)
●
Disk Space
●
Monitoring and Alarming
●
...
OPS learn
from DEV to
think about:
●
Incremental Improvement
●
Infrastructure as Code
●
Version Control System
●
Coding (OO, Functions, Libraries …)
●
Code Quality
●
Unit & Integration Tests
●
Test Automation
●
...
7. Unit Tests
Test the smallest possible components in an
artifical environment.
System Tests
Test the entire application in a real(istic)
environment together with other
applications.
8. Part of build process
Syntax checks
Scripts
Config Files
Data Files
Unit tests for functions/libs
Run program with test data
Check result
Check program behaviour
with wrong/broken test data
Also run on Developer desktop
Quick feedback (~ seconds)
Install on test server
Run tests from outside
HTTP calls
Send emails
Try to login
Run tests from inside
Remote Exec (rsh, ssh …)
http://go.schapiro.org/rshpitfall
Service Start, Stop & Status
Modify server to create
good & bad test scenarios
Reboot
Unit Tests System Tests
16. SAN mount service
Test via rsh
Mock SAN devices
with losetup
service start, stop
mounts/umounts
Error handling
17. HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all function groups!
Test Server
GET http://external.com/
X-Forwarded-For: 10.11.12.01
502 Bad Gateway✔
GET http://external.com/
X-Forwarded-For: 10.34.56.01
403 Forbidden
ERR_ACCESS_DENIED by proxy server
✘
✘
18. VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and install Linux OS
Scrape VM screen via OCR
http://github.com/Immobilienscout24/lab-manager-light
19. 34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready.
Automate the delivery chain from source till production.
20. Low Risk – Lots of Fun
http://go.schapiro.org/slides
21. Kontakt:
Immobilien Scout GmbH
Andreasstraße 10
10243 Berlin
Fon: +49 30 243 01-1229
Email: schlomo.schapiro@immobilienscout24.de
URL: www.immobilienscout24.de
Thank you very much!
Please contact me for further
questions and discussions.