I gave that talk at the 2016 Sysadmin Days to explain how I work when taking over a new infrastructure.

1. THE COMMANDO
DEVOPS
FRED DE VILLAMIL
@FDEVILLAMIL

2. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
BACKGROUND
▸ 37 years old, Linux and FreebSD user, Ruby lover
▸ Managed infrastructures for various SAAS companies
▸ Specialised in taking over forsaken, outdated,
undocumented platforms
▸ Loves working under pressure
▸ A sick sense of humour

3. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
PREREQUISITES
▸ Being in control of the budget, and having enough cash to
do what needs to be done
▸ Total control over the team hirings
▸ Freedom on how to make things viable again
▸ Involved in the product roadmap

4. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
PRIORITIES… ACCORDING TO YOUR COLLEAGUES
▸ Sales / Client Success / Marketing: Stability
▸ Sales / Client Success / Marketing / Devs: Delivering new
features
▸ Devs: On demand environments / Automation
▸ Management: Documentation / Conformity

5. RESTORE CONFIDENCE IN THE
INFRASTRUCTURE
Priority is a singular word
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016

6. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
COMMANDO DEVOPS MANAGEMENT
▸ Scrum is cool, but it lacks flexibility, Kanban lacks visibility
▸ 1 week sprints, 50% for planned projects, 50% for ongoing
tasks
▸ 1 admin == 1 project, systematic code reviews for quality and
knowledge sharing
▸ Team based sprint planning on Friday: we know what to do
Monday morning
▸ Daily standup every morning, 10:00AM at the coffee machine

7. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
ONGOING MANAGEMENT
▸ Everybody solves the incident, not only the oncall guy
▸ Week based contact point
▸ Ongoing tasks are added to the current sprint or put on the
backlog
▸ “No" or “later" are acceptable answers
▸ Always give an ETA!

8. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
DEFINE YOUR RELATIONSHIP WITH YOUR COLLEAGUES
▸ Your clients are people you’re facing directly. They’re the one
you need to satisfy.
▸ Are you in contact with the company’s client?
▸ The relation is not immutable. You can switch from a service
provider to an integrated component of the product team.

9. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
WHEN YOU HAVE NO TEAM
▸ Hire freelances while you’re looking for permanent hires
▸ Hire techno specific vendors
▸ Work with the developers who setup the existing platform

10. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
BUILDING THE TEAM
▸ Ensure we hire at least 2 people with the same skills
▸ We look for autonomous people
▸ Importance to know the technologies in place
▸ People who can learn quickly
▸ These people are senior profiles, therefore expensive!!!

11. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
COMMUNICATION
▸ Unified communication to avoid confusing messages or
situation
▸ Avoid alarming messages at all cost
▸ Be totally transparent when reporting to the management
▸ Give ETAs as soon as you can. “I don’t know” is a totally
acceptable answer.

12. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
WEEKLY REPORT
▸ Weekly report, 3-4 slides
▸ KPIs : budget, uptime, critical incidents, oncall hours
▸ Components at risk
▸ Current projects (with ETAs)

13. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
PLATFORM DISCOVERY
▸ Monitoring is your best friend, when it exists, often
incomplete
▸ nmap + ssh + facter in a CSV: useful but lots of surprises
▸ tcpdump on network nodes (load balancers, firewalls…)
▸ Puppet / Ansible: often messy when they exist
▸ “Great old ones"
▸ Hand over with the former team (not sure they accept)

14. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
FROM THE BEGINNING
▸ Let the teams that manage part of the components keep
their baby so you can focus on something else
▸ Build an inventory, rationalise the machine naming
▸ Restart ALL services one by one, many of them won’t come
back
▸ Focus on things that work but you don’t know about instead
of what you know about but doesn’t work

15. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
DOCUMENTATION
▸ Takes lot of time, but you desperately need it
▸ Track information flows between your components
▸ Add the operation manual in the monitoring alert messages

16. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
WORKING IN HOSTILE ENVIRONMENT
▸ Private clouds
▸ Networks you don’t manage
▸ AWS is a pain in the …
▸ Communication with your vendors: below a certain bill
(amount depends on the vendor), the client is always wrong

17. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
DEVELOPPERS BUILT INFRASTRUCTURES
▸ One server == one vertical feature
▸ Hard to debug
▸ Impossible to scale
▸ Need to be split in smaller VMs, by service
▸ Adds lot of pressure on your network addressing

18. QUESTIONS ?