Scaling to hundreds of hosts and serving thousands of requests per second is interesting and hard. But what about when everything is smaller? How can a tiny team—or even one systems person working alone—make sure small but vital systems stay reliable? Are SRE practices still relevant? We explain why small scale engineering is just as challenging as large-scale engineering and offer ideas on how to survive technical debt, poor communication, and other everyday challenges.

1. effie mouzeli
Velocity NY ‘18
manjiki

2. About
jijiki/effie @freenode
effie@wikimedia.org
manjiki

3. Who cares?
• Solo systems engineers
• Single “sysadm*” teams
• Small orgs
• Startups
• Juniors!
* ops/SRE/systems/DevOps(?)3

4. Do we care?
• Most startups are small scale
• Small scale is everywhere
• Everything was small scale once
• We rely on products of small orgs
4

5. Life at small scale
• Intimate
• Time to grow up
• Make decisions
• Take responsibility
• Be resourceful and creative
• A.M.A. Engineering
5

6. Life at small scale
• Stressful
• Lonely
• Fighting many battles at once
• No safety nets
• Always on call
• A.M.A. Engineering
• Invited late to the party
manjiki6

7. How we got here?
• Lack of processes and experience
• Adoption of immature tech
• Re-inventing the wheel and cargo
cult
• Frequent change of requirements
• Overestimation
7

8. How we got here?
Startups
• Short term planning
• A lot of duct tape
• Early adopters of fancy tech
• No documentation, processes,
boundaries
• No Backups :)
8

9. How we got here?
Small Organisations
• More duct tape
• Lack of original information
• Waiting for a hero or heroes
• Communication issues
• Untouchable systems
9

10. What’s our role
• Set standards
• Create processes
• Identify and solve problems
• Bridge the gap between
production and developers
• Design and advise for the future
10

11. Challenges
• New environment
• Unknown infrastructure
• A lot of debt
• Bad habits
• Finding information
• What’s coming?
11 manjiki

12. The 5 stages of technical debt
• Denial
• Anger
• Bargaining
• Depression
• Acceptance
12

13. It’s all about Control
• Ask the right questions
• Understand your product
• Write down everything
• Task dashboard
• Know your current limits
• Choose software wisely
• Learn to let go
• Be consistent and create rules
13

14. The big picture
• Provisioning and automations
• Environments
• Monitoring & Alerting
• Metrics & Logging
• Update management
• Viable backups
• Security
• Legacy systems
14

15. The Dev team is YOUR team
• Build relationships and trust
• Be approachable
• Don’t prioritise alone
• Stay in touch
• Share what you know and train
• Break bad habits
• Don’t be arrogant
15

16. Killing the sysadmn (or the team)
• Lack of recognition
• Overworking
• Burnout
• Toxicity
• Martyrs and Messiahs
16 manjiki

17. Creating a culture
• Guard your processes
• Junior engineers
• Adopt good practices
• To ERR is human
• Following the Golden Rule* can go
a long way
* https://en.wikipedia.org/wiki/Golden_Rule17

18. TL;DR
18

19. On a final note
• Working in small orgs is not a waste of
time
• Small scale is hard
• Good foundations will make scaling
easier
• Always say “we”
• Embrace failure
• Remember to breathe
• No systems are perfect, just manageable
19

20. Thank you!
jijiki/effie @freenode
effie@wikimedia.org
manjiki