EuroPython 2014 Devops Risk Mitigation

1,547 views

Published on

The (perceived) risk of the DevOps is that too many people get the right to "break" the platform.

Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others.

This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers.

See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.

Published in: Internet
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,547
On SlideShare
0
From Embeds
0
Number of Embeds
536
Actions
Shares
0
Downloads
18
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

EuroPython 2014 Devops Risk Mitigation

  1. 1. DevOps Risk Mitigation www.immobilienscout24.de Berlin | 23.07.2014 | Schlomo Schapiro Systems Architect, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ Test Driven Infrastructure @schlomoschapiro
  2. 2. Expensive Fix Cheap Fix Costs Of Finding Bugs In Production
  3. 3. Plan Design Budget Develop Test RUN DEV Plan Design Budget Develop Test RUNOPS TIME
  4. 4. Plan Design Budget Develop Test Plan Design Budget Develop Test RUN Proxy config Database borked „Buy Now“ button broken MTA drop all mail Load Balancer Configuration Everything costs only 0 € Login possible without password NFS not available DB Replication stopped No ads shown Broken CSS / JS Tomcat won't start/stop Service user not defined sudoers invalid
  5. 5. DevOps: Respect & Learning DEV learn from OPS to think about: ● Resources (CPU, RAM, Disk) ● Services (Start, Stop, Status) ● Dependencies (Start DB before App) ● Logfiles (Rotate, Remove) ● Disk Space ● Monitoring and Alarming ● ... OPS learn from DEV to think about: ● Incremental Improvement ● Infrastructure as Code ● Version Control System ● Coding (OO, Functions, Libraries …) ● Code Quality ● Unit & Integration Tests ● Test Automation ● ...
  6. 6. Untested = Broken
  7. 7. Unit Tests Test the smallest possible components in an artifical environment. System Tests Test the entire application in a real(istic) environment together with other applications.
  8. 8. Part of build process Syntax checks  Scripts  Config Files  Data Files Unit tests for functions/libs Run program with test data  Check result  Check program behaviour with wrong/broken test data Also run on Developer desktop Quick feedback (~ seconds) Install on test server Run tests from outside  HTTP calls  Send emails  Try to login Run tests from inside  Remote Exec (rsh, ssh …) http://go.schapiro.org/rshpitfall  Service Start, Stop & Status  Modify server to create good & bad test scenarios  Reboot Unit Tests System Tests
  9. 9. Unit Tests
  10. 10. %prep %setup -q %install install … %{buildroot}/… install … %{buildroot}/… %files %defattr(-,root,root,-) /...
  11. 11. BuildRequires: sudo %build set -e visudo -c -f sudoers %install install -m 0440 sudoers -D %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-) /etc/suoders.d/%{name}
  12. 12. BuildRequires: PyYAML, pylint %build set -e # syntax checks bash -n my_script.sh # Should be valid python code pylint -E yum-repo-propagate # should be valid YAML file python -c "↩ import yaml↩ yaml.safe_load(open('config.yaml'))↩ "
  13. 13. System Tests http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
  14. 14. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  15. 15. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  16. 16. SAN mount service Test via rsh Mock SAN devices with losetup service start, stop mounts/umounts Error handling
  17. 17. HTTP Proxy Configuration X-Forwarded-For header spoofs source Check result for ERR_ACCESS_DENIED Run tests for all function groups! Test Server GET http://external.com/ X-Forwarded-For: 10.11.12.01 502 Bad Gateway✔ GET http://external.com/ X-Forwarded-For: 10.34.56.01 403 Forbidden ERR_ACCESS_DENIED by proxy server ✘ ✘
  18. 18. VM Provisioning & Kickstart Installation Test via HTTP API Create broken VMs and check error reports Create valid VM and install Linux OS Scrape VM screen via OCR http://github.com/Immobilienscout24/lab-manager-light
  19. 19. 34 35 36 2 3 53 87 88 89 90 91 92 93 94 95 96 97 TIME Continous Live Deployment Deploy every application when it is ready. Automate the delivery chain from source till production.
  20. 20. Low Risk – Lots of Fun http://go.schapiro.org/slides
  21. 21. Kontakt: Immobilien Scout GmbH Andreasstraße 10 10243 Berlin Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.de URL: www.immobilienscout24.de Thank you very much! Please contact me for further questions and discussions.

×