Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Commando Devops

1,774 views

Published on

I gave that talk at the 2016 Sysadmin Days to explain how I work when taking over a new infrastructure.

Published in: Career
  • Be the first to comment

The Commando Devops

  1. 1. THE COMMANDO DEVOPS FRED DE VILLAMIL @FDEVILLAMIL
  2. 2. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 BACKGROUND ▸ 37 years old, Linux and FreebSD user, Ruby lover ▸ Managed infrastructures for various SAAS companies ▸ Specialised in taking over forsaken, outdated, undocumented platforms ▸ Loves working under pressure ▸ A sick sense of humour
  3. 3. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 PREREQUISITES ▸ Being in control of the budget, and having enough cash to do what needs to be done ▸ Total control over the team hirings ▸ Freedom on how to make things viable again ▸ Involved in the product roadmap
  4. 4. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 PRIORITIES… ACCORDING TO YOUR COLLEAGUES ▸ Sales / Client Success / Marketing: Stability ▸ Sales / Client Success / Marketing / Devs: Delivering new features ▸ Devs: On demand environments / Automation ▸ Management: Documentation / Conformity
  5. 5. RESTORE CONFIDENCE IN THE INFRASTRUCTURE Priority is a singular word COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
  6. 6. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 COMMANDO DEVOPS MANAGEMENT ▸ Scrum is cool, but it lacks flexibility, Kanban lacks visibility ▸ 1 week sprints, 50% for planned projects, 50% for ongoing tasks ▸ 1 admin == 1 project, systematic code reviews for quality and knowledge sharing ▸ Team based sprint planning on Friday: we know what to do Monday morning ▸ Daily standup every morning, 10:00AM at the coffee machine
  7. 7. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 ONGOING MANAGEMENT ▸ Everybody solves the incident, not only the oncall guy ▸ Week based contact point ▸ Ongoing tasks are added to the current sprint or put on the backlog ▸ “No" or “later" are acceptable answers ▸ Always give an ETA!
  8. 8. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 DEFINE YOUR RELATIONSHIP WITH YOUR COLLEAGUES ▸ Your clients are people you’re facing directly. They’re the one you need to satisfy. ▸ Are you in contact with the company’s client? ▸ The relation is not immutable. You can switch from a service provider to an integrated component of the product team.
  9. 9. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 WHEN YOU HAVE NO TEAM ▸ Hire freelances while you’re looking for permanent hires ▸ Hire techno specific vendors ▸ Work with the developers who setup the existing platform
  10. 10. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 BUILDING THE TEAM ▸ Ensure we hire at least 2 people with the same skills ▸ We look for autonomous people ▸ Importance to know the technologies in place ▸ People who can learn quickly ▸ These people are senior profiles, therefore expensive!!!
  11. 11. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 COMMUNICATION ▸ Unified communication to avoid confusing messages or situation ▸ Avoid alarming messages at all cost ▸ Be totally transparent when reporting to the management ▸ Give ETAs as soon as you can. “I don’t know” is a totally acceptable answer.
  12. 12. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 WEEKLY REPORT ▸ Weekly report, 3-4 slides ▸ KPIs : budget, uptime, critical incidents, oncall hours ▸ Components at risk ▸ Current projects (with ETAs)
  13. 13. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 PLATFORM DISCOVERY ▸ Monitoring is your best friend, when it exists, often incomplete ▸ nmap + ssh + facter in a CSV: useful but lots of surprises ▸ tcpdump on network nodes (load balancers, firewalls…) ▸ Puppet / Ansible: often messy when they exist ▸ “Great old ones" ▸ Hand over with the former team (not sure they accept)
  14. 14. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 FROM THE BEGINNING ▸ Let the teams that manage part of the components keep their baby so you can focus on something else ▸ Build an inventory, rationalise the machine naming ▸ Restart ALL services one by one, many of them won’t come back ▸ Focus on things that work but you don’t know about instead of what you know about but doesn’t work
  15. 15. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 DOCUMENTATION ▸ Takes lot of time, but you desperately need it ▸ Track information flows between your components ▸ Add the operation manual in the monitoring alert messages
  16. 16. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 WORKING IN HOSTILE ENVIRONMENT ▸ Private clouds ▸ Networks you don’t manage ▸ AWS is a pain in the … ▸ Communication with your vendors: below a certain bill (amount depends on the vendor), the client is always wrong
  17. 17. COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016 DEVELOPPERS BUILT INFRASTRUCTURES ▸ One server == one vertical feature ▸ Hard to debug ▸ Impossible to scale ▸ Need to be split in smaller VMs, by service ▸ Adds lot of pressure on your network addressing
  18. 18. QUESTIONS ?

×