Fastly Altitude - June 25, 2015. Ines Sombra, a Systems Engineer at Fastly, talks about lessons learned in rapid systems development. Video of the talk: http://fastly.us/Altitude2015_Fallacy-of-Fast Ines' bio: Ines Sombra is a Systems Engineer at Fastly, where she spends her time helping the Web go faster. Ines holds an M.S. in Computer Science and an M.S. in Information Management from Washington University in Saint Louis. Being a true Argentine, she has a fondness for steak, fernet, and a pug named Gordo.

2. the Fallacy
Of Fast

3. @Randommood
INES
Sombra

6. Common
Mistakes
What
Matters

7. Common
Mistakes

8. Accidentally
de-emphasizing
long term quality
& system Stability
1

9. De-prioritizing Testing
Cutting corners on testing
carries a hidden cost
Test the full system: client,
code, & provisioning code
Code reviews != tests.
Have both
Continuous Integration
(CI) is critical to velocity,
quality, & transparency

10. Automation shortcuts taken while
in a rush will come back to haunt
you
Playbooks (docs) are a must
Release stability is often tied to
system stability. Iron out your
deploy process!
De-prioritizing Ops
👈

11. “Future you monitoring” is
bad, make it part of MVP
Alert fatigue has a cost,
don’t let it get that far
Link alerts to playbooks
Don’t ignore error handling
& security
De-prioritizing Insight
✨

12. The inner workings of data
components matter.
Learn about them
System boundaries ought
to be made explicit
Deprecate your go-to
person
De-prioritizing Knowledge

13. Don’t sacrifice tests.
Test the FULL system
Service ownership implies
leveling-up operationally
Architectural choices
made in a rush have a
long life
Whatwe learned
✨

14. What
Matters

15. Mind system design
Simple & utilitarian design
takes you a long way
Use well understood
components (mind NIH)
Use feature flags & on/off
switches (test them!)
YEEHAW!

16. Mind system Limits
Rate limit your API calls
especially if they are public or
expensive to run
Instrument / add metrics to
track them
Rank your services & data
(what can you drop?)
Capacity analysis is not dead
✨

17. Mind system Configs
System assumptions are
dangerous, make them
explicit
Consolidate system
configuration (data bags,
config file, etc)
Hardcoding is the devil

18. Distrust is healthy
Distrust client behavior, even
if they are internal
Decisions have an expiration
date. Periodically re-
evaluate them as past
you was much dumber
A revisionist culture produces
more robust systems
✨

19. Keep track of your
technical debt &
repay it regularly
It’s about lowering
the risk of change
with tools & culture
Mind assumptions
Whatwe learned
✨

20. TL;DR
Easy to sacrifice things
may be harder to
correct later
Culture is established
daily
TESTING MATTERS!
Not all process is evil
Keep in Mind
System boundaries &
dependencies ought
to be explicit
Playbooks are your
friends, have them
Use kill switches & limits
Prioritize your services
Distributed systems

21. github.com/Randommood/FallacyOfFast