Devopsdays 2019 talk

1. Imagine this:
● Developer of something
technical
● Domain expert, but developing
in isolation.
● Not data-driven
● Little feedback

2. Operations
● Need to make it run smoothly
● But everything is reactive not
proactive.
● Only know when things go
wrong
● Deep understanding of how
things really work

3. Sound familiar?
● In case you didn't spot the
title, I'm actually talking about
Bus Companies
● Disclaimer: I may have
exaggerated a little!
● I’m going to work through an
analogy between buses and
software

4. Buses Software
Scheduler Developer
Operations Operations
Schedule Software
Buses/Stops Servers, Containers
GPS pings, etc Instrumentation
Passengers Users
Reliability Uptime
Efficiency Latency

5. ● Great they’re the same, let’s
do DevOps!
● The analogy breaks down
when it comes to tools
● No bus equivalent for Git,
Kubernetes, Terraform, Cloud
etc

6. One view of what we’re doing is Devops for Buses.
Creating a toolbox with DevOps in my mind,
rather than tools that match the current process.
Devops
for Buses

7. Cityswift create more efficient and reliable bus timetables
using:
● Lots of Data
● Machine learning
● Mathematical optimisation
For instance, how can we avoid
all the buses arriving at once?
Let’s look at 3 of our tools and
their software equivalents.

8. 3 tools and their analogues
SWIFT INSIGHTS
● View historical and
predicted data on journey
times and passengers
● Schedulers get detailed
feedback on the impact of
their changes
● Datadog to view latency
and throughput
● A/B Testing
BUS SOFTWARE

9. 3 tools and their analogues
SWIFT SCHEDULE
● Automated creation of bus
schedules
● Version controlled
● Terraform, Ansible to
provision infrastructure
● Git
● Jenkins
BUS SOFTWARE

10. 3 tools and their analogues
SWIFT OPS (Coming Soon!)
● Dashboard for Operations
● Solve issues before they
happen
● Log incidents
● Datadog alerts, OpsGenie
● No blame post-mortems
BUS SOFTWARE

11. Motivation for Bus Companies
We’ve all been on a bus when things went wrong
● Weather
● Break down
● Bus full at a football match
● Its bad for passengers and what's bad for
passengers is bad for bus companies

12. Benefit to Bus Companies
● Add more buses for reliability?
● But running a very reliable service is expensive
and inefficient, need a happy balance
● Think # servers vs response times
● Bus companies can get fined by regulators for not
hitting KPIs (not nine-nines though!)

13. 13
Aren’t we doing it the
wrong way around?
● Should be People before Process
before Tools
● Bus companies have the right
people, they're specialists in a
difficult domain.

14. 14
● We didn't dive headlong into tools
● We talked with them and worked out
a process collaboratively
● From seeing how they work, we
know that the existing tools don’t
create the right process
Aren’t we doing it the
wrong way around?

15. Process
15
Data-driven
All the data is
available rather than a
subset
Automation
Automating the
boring parts, free
up the experts
Continuous Delivery
Automation allows
more frequent
schedule changes

16. Process
16
Version control of
Schedules
Treating the schedule
as you would treat
code
Monitoring and
Feedback
Correlate
schedules, KPIs and
incidents
Collaboration
No strict split in
tools for Dev and
Ops.

17. In Praise of Tools
● We shouldn't dive headlong into creating tools to
solve problems
● But we should create tools that encourage the
process we want.
● I think that tools help create the de facto process,
whatever the de jure process is

18. Where does the Analogy
Breakdown?
● Timeframes are much longer, and
feedback is slower. Think months
not minutes
● And you can't just run
kubectl make more buses
(But, if you know how to solve
that, you're hired! )

19. What can Software Engineers
learn from
Bus Companies?
● Tools shouldn't automate everything
● But should make it easier for the expert to make
decisions.
● Experts will spot the patterns and apply some
tricks (E.g. anomaly detection is hard!)

20. Thank You!
Check us out at cityswift.com
whoami
Frank Farrell
Lead Developer and Cloud Architect
@CitySwift
github.com/frankfarrell
linkedin.com/in/frank-farrell-44529a93
ofearghp@gmail.com