How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022

1. How to screw up learn safely
in Production
By Pedro Gustavo Torres
Senior Director of Engineering @ Salsify

2. Hello there!
Pedro Gustavo Torres
Senior Engineering Leader
Leading Engineering for 14 years
Portuguese living and working from Portugal
CTO Portugal co-founder
Investor and Advisor
Star Wars and Lego fan
Vegetarian and Non-drinker

3. Dev vs Ops

4. DevOps
We want the
same thing
We want the
same thing
DevOps

5. SRE

6. Developers On-Call

7. Developers On-Call

8. Developers On-Call
We want the
same thing
We want the
same thing
Developers On-Call

9. Developers On-Call
• What is key on the On-Call process?
• Psychological Safety
• Autonomy
Psychological
safety
Autonomy

10. Developers On-Call
• What is key on the On-Call process?
• Psychological Safety
• Autonomy
Psychological
safety
Autonomy
Learning
in
Production

11. Pschycological Safety
• Hospital patient care teams’ errors study
> “Building a psychologically safe workplace” by Amy Edmondson @
TEDxHGSE (https://www.youtube.com/watch?v=BxC1Bl-4Zv)
TL;DR
Ø The Better teams were reporting more mistakes than the other
teams on the study (which was confusing)
Ø The Better teams are better because they are open to discuss
the mistakes and learn from them

12. Pschycological Safety
• Google Project Aristotle
> https://rework.withgoogle.com/blog/five-keys-to-a-successful-
google-team/
TL;DR
What makes a team effective at Google?
Ø Psychological Safety
Ø Dependability
Ø Structure and Clarity
Ø Meaning
Ø Impact

13. Clarity, Competence, and Autonomy

14. Purpose, Mastery, and Autonomy

15. Screwing up Learning in Production (10 tips)
1. No Execs in public channels asking for updates
2. No pressure from middle management
3. Full autonomy to declare incidents (e.g. Support, Engineering)
4. Full autonomy to update the status page (with messaging agreed by Support + Engineering)
5. Full transparency on the incident (e.g. slack channel, zoom call)
6. Primary and secondary roles on each on-call shift
7. Incident Commander (someone that it is not solving the incident… ideally middle mng)
8. Blameless (and anonymized) Post mortems… with Action Points with clear owners
9. Discuss openly incidents and potential issues (don’t swipe them under the rug)
10. Be solidary and help if needed (i.e. don’t shield behind the “it’s not my team/system/shift”)

16. Who is this person?

17. Career Honours

18. Source: https://en.as.com/en/2018/05/10/soccer/1525950671_256207.html

19. Never give in

20. How to screw up learn safely
in Production
By Pedro Gustavo Torres
Senior Director of Engineering @ Salsify