OSDC 2014: Schlomo Schapiro -  Test Driven Infrastructure
Upcoming SlideShare
Loading in...5
×
 

OSDC 2014: Schlomo Schapiro - Test Driven Infrastructure

on

  • 140 views

Common wisdom has it that the test effort should be related to the risk of a change. However, the reality is different: Developers build elaborate automated test chains to test every single commit of ...

Common wisdom has it that the test effort should be related to the risk of a change. However, the reality is different: Developers build elaborate automated test chains to test every single commit of their application. Admins regularly “test” changes on the live platform in production. But which change carries a higher risk of taking the live platform down?

What about the software that runs at the “lower levels” of your platform, e.g. systems automation, provisioning, proxy configuration, mail server configuration, database systems etc. An outage of any of those systems can have a financial impact that is as severe as a bug in the “main” software!

One of the biggest learnings that any Ops person can learn from a Dev person is Test Driven Development. Easy to say - difficult to apply is my personal experience with the TDD challenge.

This talk throws some light on recent developments at ImmobilienScout24 that help us to develop the core of our infrastructure services with a test driven approach:
* How to do unit tests, integration tests and systems tests for infrastructure services?
* How to automatically verify Proxy, DNS, Postfix configurations before deploying them on live servers?
* How to test “dangerous” services like our PXE boot environment or the automated SAN mounting scripts?
* How to add a little bit of test coverage to everything we do.
* Test Driven: First write a failing test and then the code that fixes it.

Statistics

Views

Total Views
140
Views on SlideShare
117
Embed Views
23

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 23

http://www.netways.de 23

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

OSDC 2014: Schlomo Schapiro -  Test Driven Infrastructure OSDC 2014: Schlomo Schapiro - Test Driven Infrastructure Presentation Transcript

  • Test Driven Infrastructure www.immobilienscout24.de Berlin | 09.04.2014 | Schlomo Schapiro Systemarchitekt, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ DevOps Risk Mitigation @schlomoschapiro schlomo.schapiro@immobilienscout24.de
  • Slide 2 | Test Driven Infrastructure | Schlomo Schapiro www.ImmobilienScout24.de >2 billion PI per month 2 data center with ~1600 VM ~2.5 million outgoing email/day total of ~600 employees  ~30 crossfunctional IT teams  ~160 in IT 15 years in business part of Deutsche Telekom
  • Slide 3 | Test Driven Infrastructure | Schlomo Schapiro Expensive Fix Cheap Fix Costs Of Finding Bugs In Production
  • Slide 4 | Test Driven Infrastructure | Schlomo Schapiro Plan Design Budget Develop Test RUN DEV Plan Design Budget Develop Test RUNOPS TIME
  • Slide 5 | Test Driven Infrastructure | Schlomo Schapiro Plan Design Budget Develop Test Plan Design Budget Develop Test RUN Proxy config Database borked „Buy Now“ button broken MTA drop all mail Load Balancer Configuration Everything costs only 0 € Login possible without password NFS not available DB Replication stopped No ads shown Broken CSS / JS Tomcat won't start/stop Service user not defined sudoers invalid
  • Slide 6 | Test Driven Infrastructure | Schlomo Schapiro DevOps: Respect & Learning DEV learn from OPS to think about: ● Resources (CPU, RAM, Disk) ● Services (Start, Stop, Status) ● Dependencies (Start DB before App) ● Logfiles (Rotate, Remove) ● Disk Space ● Monitoring and Alarming ● ... OPS learn from DEV to think about: ● Incremental Improvement ● Infrastructure as Code ● Version Control System ● Coding (OO, Functions, Libraries …) ● Code Quality ● Unit & Integration Tests ● Test Automation ● ...
  • Slide 7 | Test Driven Infrastructure | Schlomo Schapiro Untested = Broken
  • Slide 8 | Test Driven Infrastructure | Schlomo Schapiro Unit Tests Test the smallest possible components in an artifical environment. System Tests Test the entire application in a real(istic) environment together with other applications.
  • Slide 9 | Test Driven Infrastructure | Schlomo Schapiro Part of build process Syntax checks  Scripts  Config Files  Data Files Unit tests for functions/libs Run program with test data  Check result  Check program behaviour with wrong/broken test data Also run on Developer desktop Quick feedback (~ seconds) Install on test server Run tests from outside  HTTP calls  Send emails  Try to login Run tests from inside  Remote Exec (rsh, ssh …) http://go.schapiro.org/rshpitfall  Service Start, Stop & Status  Modify server to create good & bad test scenarios  Reboot Unit Tests System Tests
  • Slide 10 | Test Driven Infrastructure | Schlomo Schapiro Unit Tests
  • Slide 11 | Test Driven Infrastructure | Schlomo Schapiro %prep %setup -q %install install … %{buildroot}/… install … %{buildroot}/… %files %defattr(-,root,root,-) /...
  • Slide 12 | Test Driven Infrastructure | Schlomo Schapiro BuildRequires: sudo %build set -e visudo -c -f sudoers %install install -m 0440 sudoers -D %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-) /etc/suoders.d/%{name}
  • Slide 13 | Test Driven Infrastructure | Schlomo Schapiro BuildRequires: PyYAML, pylint %build set -e # syntax checks bash -n my_script.sh # Should be valid python code pylint -E yum-repo-propagate # should be valid YAML file python -c "↩ import yaml↩ yaml.safe_load(open('config.yaml'))↩ " ...
  • Slide 14 | Test Driven Infrastructure | Schlomo Schapiro BuildRequires: python-unittest2, python-teamcity-messages, ... %build set +x +e # for teamcity exec 1>&2 # join stdout to stderr to synchronize between them for teamcity output FAILED=0 TEAMCITY_PROJECT_NAME=1 python unit_tests.py || let FAILED++ bash -n oldhomeinfo.sh || let FAILED++ (( FAILED == 0 )) || exit 1
  • Slide 15 | Test Driven Infrastructure | Schlomo Schapiro More Examples for Unit Tests Patching nsswitch.conf and PAM files Syntax checking HTTPD, DNS, DHCP ... configuration files Checking SSH Server & Client configurations http://go.schapiro.org/sshconfigtest ...
  • Slide 16 | Test Driven Infrastructure | Schlomo Schapiro System Tests http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
  • Slide 17 | Test Driven Infrastructure | Schlomo Schapiro Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  • Slide 18 | Test Driven Infrastructure | Schlomo Schapiro Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  • Slide 19 | Test Driven Infrastructure | Schlomo Schapiro SAN mount service Test via rsh Mock SAN devices with losetup service start, stop mounts/umounts Error handling
  • Slide 20 | Test Driven Infrastructure | Schlomo Schapiro HTTP Proxy Configuration X-Forwarded-For header spoofs source Check result for ERR_ACCESS_DENIED Run tests for all function groups! Test Server GET http://external.com/ X-Forwarded-For: 10.11.12.01 502 Bad Gateway✔ GET http://external.com/ X-Forwarded-For: 10.34.56.01 403 Forbidden ERR_ACCESS_DENIED by proxy server ✘ ✘
  • Slide 21 | Test Driven Infrastructure | Schlomo Schapiro Subversion Server Configuration Tests Config RPM Maker https://github.com/yadt/yadt-config-rpm-maker Tests 2 servers: Master & Slave Replication Failure and Recovery Backup and Restore Master SVN svnsync Backup Slave SVN Backup
  • Slide 22 | Test Driven Infrastructure | Schlomo Schapiro PAM & NSS Configuration PAM & nsswitch.conf patching Mock setup via rsh Mock AD groups and users with nss_db Check service status Test login via ssh Thorben Wengert / pixelio.de
  • Slide 23 | Test Driven Infrastructure | Schlomo Schapiro VM Provisioning & Kickstart Installation Test via HTTP API Create broken VMs and check error reports Create valid VM and install Linux OS Scrape VM screen via OCR http://github.com/Immobilienscout24/lab-manager-light
  • Slide 24 | Test Driven Infrastructure | Schlomo Schapiro 34 35 36 2 3 53 87 88 89 90 91 92 93 94 95 96 97 TIME Continous Live Deployment Deploy every application when it is ready. Automate the delivery chain from source till production.
  • Slide 25 | Test Driven Infrastructure | Schlomo Schapiro Low Risk – Lots of Fun http://go.schapiro.org/slides
  • Slide 26 | Test Driven Infrastructure | Schlomo Schapiro Kontakt: Immobilien Scout GmbH Andreasstraße 10 10243 Berlin Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.de URL: www.immobilienscout24.de Thank you very much! Please contact me for further questions and discussions.