1. Increasing agility through DevOps
Jon Milsom - Co-founder & CTO
@jonmilsom
Leeds DevOps
15th November 2016

2. An online platform for amateur and semi-pro sports teams

3. Last time_
August 2014_

4. “agility”

5. “agility”
“ability of a system to rapidly respond to change by adapting
its initial stable configuration”
Wikipedia - Business agility
https://en.wikipedia.org/wiki/Business_agility

6. Examples
Things you can do if you have good agility
● Release software frequently and with confidence
● Develop new features without being bogged down by technical debt
● Onboard team members and bring them up to speed quickly
● Move systems between data centres / clouds

7. Configuration management

8. Configuration management
Containerised development environment
Aggregated logging (via Ansible)

9. Configuration management
Before Time After Time
Get a new server
into prod
Runbook hours Ansible minutes
Setup local dev
environment
- days Vagrant & Ansible minutes
Create test server
for new software
version
Runbook
Individual log files
hours Ansible
Aggregated logging
minutes

10. Configuration management - Example
PHP Versions 5.3. to 7.0

11. Configuration management - Example

12. Communication

13. Communication

14. Communication
“Business”
Marketing
Sales
Support
“Technology”
Development
Operations
Security

15. Communication
“Business”
I need it yesterday, for free
What are you working on?
We can’t do our job unless you deliver
this
“Technology”
Not possible
Last weeks last emergency project
:(

16. Jared Dunn, Silicon Valley
http://www.theyoungfolks.com/wp-content/uploads/2014/05/episode-05-05-1024.jpg

17. Communication
How to write a good bug report
Why security is important

18. Communication
Self service for other departments
Requires trust & process

19. Communication
Summary:
● Be transparent - increase visibility with other departments
● Communicate openly - good or bad
● Trust - remove unnecessary bottlenecks (self service where possible)

20. Communication
Summary:
● Be transparent - increase visibility with other departments
● Communicate openly - good or bad
● Trust - remove unnecessary bottlenecks (self service where possible)
“growing up”

21. Managing technical debt & fixing bugs

22. Managing technical debt & fixing bugs
Measure / assess

23. Managing technical debt & fixing bugs
Prioritise

24. Managing technical debt & fixing bugs
Commit

25. Managing technical debt & fixing bugs
Going forward, every sprint:
a. Fix bugs & regressions quickly
b. Refactor (retrospectively) poor decisions
c. Increase test coverage

26. Embrace open source & PaaS

27. Embrace open source & PaaS

28. Embrace open source & PaaS
RDS

29. The Database

30. The Database - why is this hard?
● Live transactional data
○ Always changing (writes), must be atomic & consistent i.e. NOT immutable!
● Size of data set
○ Hard (slow) to move around
● Changes to schema / indexes performed manually etc
○ Seen as high risk, hard to maintain across environments

31. The Database - problems we had/have
● Made changes in production, not replicated to other environments
○ Application now out of sync with database schema or indexes
● Accidently deleted data from production which was required for tests
○ Now tests fail & difficult to re-assemble required state of dataset
● Deployed schema changes with application changes & could not revert
○ Data set polluted based on application code, hard to reverse engineer

32. This might lead you to fear change...

33. “We don’t like change”

34. You can increase agility / deployability

35. The Database
https://skeltonthatcher.com/blog/database-deployability-1-minimize-changes-in-production/
Database Deployability

36. The Database
https://skeltonthatcher.com/blog/database-deployability-1-minimize-changes-in-production/
Problem: Changes originate in production

37. The Database
https://skeltonthatcher.com/blog/database-deployability-1-minimize-changes-in-production/
Solution: Isolate flow of production data

38. The Database - so far
1. Codified and version controlled changes to schema / indexes
2. Decouple tests from database contents (work in progress)
3. Deploy (backwards compatible) schema changes before code

39. Project:
“Blue Sheep”

40. Blue Sheep

41. Blue Sheep
English (UK) =>
English (USA)
English (CA)
French (CA)
French (FR)
X (Y)

42. Blue Sheep
High risk

43. Blue Sheep
Complex / Big

44. Blue Sheep
How?

45. Blue Sheep
1. “Responsive” feature toggle
Release the code without changing the UX
Users saw minimal change - we reduced risk

46. Blue Sheep
2. Re-built template by template
Continually delivered small chunks of work each sprint
Caught regressions & errors quickly

47. Blue Sheep - Tech results
● Refactored majority of public facing pages to use Laravel
● Implemented platform wide translations engine
● Converted frontend to be responsive
● Refactored CSS to use ITCSS / BEM principals

48. Blue Sheep - Business results
● Pages per session up 51%
● Session duration up 97%
● Bounce rate down 37%
● Marketing team now able to send out location specific campaigns

49. We were on TV :)

50. We were on TV :(

55. Ego vs team

56. Your ego has its place

57. Often ego win = team fail

58. We want the team to win...

59. Personal examples
Me Team
Aim to work with (or hire) people
who are better than you / have
other skills
No longer “the best” Skillset increased & augmented
Everyone learns
Had a pull request rejected Feel like a n00b! Standards have increased
Trust people to make big
decisions
Scary to delegate Shared domain knowledge
Trusted & empowered

61. “Leave your ego at the door”
#egoless technology teams
https://www.allaboutjazz.com/an-evening-with-a-legend-quincy-jones-quincy-jones-by-solomon-leflore.php

62. The Future
Codify entire infrastructure (currently just individual servers)
Continue to invest in reduction of tech debt
Remove noise from error logs
Consolidate monitoring & analytics
Increase quality and quantity of automated testing
Parallelise CI

63. @PitcheroTech / @jonmilsom
Questions?
We are hiring! pitchero.com/careers