At Ustream the teams developing the streaming technology stack are also responsible for operating it. This means we have our monitoring and alerting in place (including those based on error logs I mentioned above, but many others too) which alert the engineers themselves. I would like to talk about how we made this transition from the traditional setup where the devs did the coding and the sysops did the operation - what lessons we learned, how we convinced the sysop guys to give us permissions and so on

1. Implementing DevOps
In practice
Zoltán Németh
Engineering Manager, Core Systems

2. About the
presenter  Age 36, Married, father of
two
 Geek since ~30 years
 Ustreamer for 5 years
 Dev, Ops, Management

3. at a
glance
Live streaming
Free broadcastingSoftware as a Service
Big events
• Sony Playstation
• Nintendo
• Lady Gaga
• Festivals
Citizen journalism
• Rescue of Chilean miners
• Earthquake in Japan
• Revolutions in Egypt, Syria
• Protests in Ukraine
• Obama campaign
• Climate Reality
• Football
• Justin Bieber
Concurrent viewership peaks at almost 2 million

4. Ustream timeline
 2007: founded
 2010-11: first big expansion
 2013: shift focus to Pro Broadcasting
 2014: SaaS product line launched
 2015: current engineering setup: 12 teams, ~100
people

5. Agenda
 What is DevOps?
 Why DevOps?
 Implementing
 Organizational changes
 Takeaways

6. Origins  Agile 2008: “Agile
infrastructure”
 DevOps Days, from 2009
Belgium
 Cloud, scaling, automation
 command line is not
enough
 Netflix, Etsy, Github
 O’Reilly Velocity

7. Devs
Before DevOps
Ops
QA

8. Devs Ops
 They build it
 Development is single
responsibility
 No access to production
systems
 “It worked on my machine”
 They deploy and run it
 Control and operate all
prod servers
 Restrict access
 On duty rotation
 “You mess it up and I have
to wake up when it fails”

9. Devs
The release engineer
Ops
QA

10. Cross functional teams
The Agile way – first step towards DevOps

11. No strict team boundaries
No more silos

12. What is DevOps?
The DevOps movement emphasizes communication,
collaboration and integration between software
developers and IT operations (New Relic)
The set of cultural norms and technical practices that
enable organizations to have a fast flow of work from
development through test and deployment, while
preserving world-class reliability, availability, and
security. (Gene Kim)

13. What is DevOps?  Agile Manifesto
 Continuous Integration
 Continuous Deployment
 Cross functionality
 Shared responsibilities,
trust
 Everyone on Duty
 Metrics and Monitoring
 Toolchain

14. Implementing  Redefine roles
 “Old guard” problem
 Knowledge sharing
 Permissions and
responsibility
 Audit logs
 Ops reserved areas

15. Metrics  Dashboards
 Graphite
 Dashing
 Code metrics
 Static analysis
 Test coverage
 Real user metrics

17. Automate all the
things
 Build
 Jenkins
 Gradle
 Deployment
 Deb packages
 Immutable images
 Automatic scaling

18. Alerting & Duty  Everyone involved
 Meaningful checks
 Self-aware applications
 Tooling
 Admin
 Logs
 Debug

19. OpenDuty  Open source alerting tool
 Supports different
notification methods
 Incident handling
 Personal configuration
 Escalation policies
 PagerDuty compatible API
 Mobile Web View

20. DevOps Culture  Ownership
 Autonomy
 Alignment
 Challenge
 Culture as a Weapon

21. Takeaways  We all row the same boat
 Ownership leads to
responsibility
 Automate all the things
 Metrics, Monitoring,
Alerting
 Tools

22. Questions
syntaxerror@ustream.tv