DevOps Picc12 Management Talk

DevOps Management: Herding Cats at Michael Rembetsy Director, Operations Engineering @mrembetsy

What is Etsy? “Etsy is the marketplace we make together. We enablepeople anywhere to easily build and directly exchange with independent, creative businesses”

The Numbers!

The Numbers!• 15 million members

The Numbers!• 15 million members• 875,000 active sellers

The Numbers!• 15 million members• 875,000 active sellers• Sellers in over 150 countries

The Numbers!• 15 million members• 875,000 active sellers• Sellers in over 150 countries• 2011 sellers grossed 525 million in sales

The Numbers!• 15 million members• 875,000 active sellers• Sellers in over 150 countries• 2011 sellers grossed 525 million in sales• 1.4 billion page views a month and growing

Who am I?

Who am I?• Responsible for day to day collaboration and operations (datacenter / corporate)

Who am I?• Responsible for day to day collaboration and operations (datacenter / corporate)• I am a mentor

Who am I?• Responsible for day to day collaboration and operations (datacenter / corporate)• I am a mentor• I enable

Who am I?• Responsible for day to day collaboration and operations (datacenter / corporate)• I am a mentor• I enable• Remove barriers and encourage shipping

Culture

The old Culture didn’t work at Etsy

So what is the new way?

Starts with Hiring

Starts with Hiring• Manager != smartest person on the team, hire smart people

Starts with Hiring• Manager != smartest person on the team, hire smart people• You need to be more than just a technologist

Starts with Hiring• Manager != smartest person on the team, hire smart people• You need to be more than just a technologist• Experiences count for managers and team members

Required

Principles for a DevOps Managers

Principles for a DevOps Managers• Accept failures not lower standards

Principles for a DevOps Managers• Accept failures not lower standards• Доверяй, но проверяй

Principles for a DevOps Managers• Accept failures not lower standards• Доверяй, но проверяй• Celebrate success and failures (blameless post mortems)

Principles for a DevOps Managers• Accept failures not lower standards• Доверяй, но проверяй• Celebrate success and failures (blameless post mortems)• Be open minded

Principles for a DevOps Managers• Accept failures not lower standards• Доверяй, но проверяй• Celebrate success and failures (blameless post mortems)• Be open minded• Move fast (collaboration and deployment)

The Team

The Team• Flexible. Able to work in a dynamic environment anywhere

The Team• Flexible. Able to work in a dynamic environment anywhere• There is no single variable to success. There are many.

The Team• Flexible. Able to work in a dynamic environment anywhere• There is no single variable to success. There are many.• Team members and managers must know they are liaisons for all other teams

How do I manage my team during...

Non-Stressful Times

Stressful Times

The Same Way!

The Same Way!• Failures / outages

The Same Way!• Failures / outages• Personality conﬂicts

The Same Way!• Failures / outages• Personality conﬂicts• Personal issues

The Same Way!• Failures / outages• Personality conﬂicts• Personal issues• Last minute whoopsies

The Same Way!• Failures / outages• Personality conﬂicts• Personal issues• Last minute whoopsies• Unexpected context switching (work)

One on One

“What’s the point? We talk all the time in IRC?”“I don’t have time, just email me an update.”

NO!!!!!

NO!!!!!Provide a scheduled outlet for talk (30minutes minimum)

NO!!!!!Provide a scheduled outlet for talk (30minutes minimum)Respect their time and yours, show up ontime and prepared to discuss the last week

http://www.randsinrepose.com/archives/2010/09/22/ the_update_the_vent_and_the_disaster.html

Freedom

Engineers are creative.So let them be creative....

Freedom but with Guidelines• Communication: send everyone updates• Ask for help if you need it• Ask for direction on work• Day to day work is done (queue, job description)

Open Door Policy

Problems Managing People

Problems Managing People• You will have them and each is diﬀerent

Problems Managing People• You will have them and each is diﬀerent• Stay calm

Problems Managing People• You will have them and each is diﬀerent• Stay calm• Talk to your peers for advice

Problems Managing People• You will have them and each is diﬀerent• Stay calm• Talk to your peers for advice• Think outside the box

Problems Managing People• You will have them and each is diﬀerent• Stay calm• Talk to your peers for advice• Think outside the box• Don’t fear having hard conversations. These make you a better manager

“the job of a manager is to be willing to have ahard conversation, any time during the weekwith anyone anywhere.” Marc Hedlund

We also changed ourworkﬂow in operations

Why?We weren’t shipping enoughContext switching = low morale“what are you working on?”

Kanban

Kanban• Provided the whole organization Transparency into what we were doing

Kanban• Provided the whole organization Transparency into what we were doing• Less context switching = better morale

Kanban• Provided the whole organization Transparency into what we were doing• Less context switching = better morale• Better throughput of work

Kanban• Provided the whole organization Transparency into what we were doing• Less context switching = better morale• Better throughput of work• Better Prioritization

How Does it ﬂow?

We use a virtual board

Give them tools &Encourage buildingcollaborative tools!

Devops Tools - managers• Jira- tickets All can see• Kanban - workﬂow that you just saw• but wait theres more.....

Metrics

Metrics• We have ~280K now and growing!

Metrics• We have ~280K now and growing!• Use open source tools (ganglia, graphite)

Metrics• We have ~280K now and growing!• Use open source tools (ganglia, graphite)• Dashboards are key

Monitoring• We use tools like Nagios, Pagerduty and Pingdom.

Monitoring• We use tools like Nagios, Pagerduty and Pingdom.• More than ops are on call

Monitoring• We use tools like Nagios, Pagerduty and Pingdom.• More than ops are on call • ops on call rotation

Monitoring• We use tools like Nagios, Pagerduty and Pingdom.• More than ops are on call • ops on call rotation • dev on call rotation

Monitoring• We use tools like Nagios, Pagerduty and Pingdom.• More than ops are on call • ops on call rotation • dev on call rotation • search on call rotation

Human escalation

Asset Management

Chef deployment istransparent to everyone Deployed Chef Change

That wasn’t enough collaboration

Eventinator

Let’s review

• Culture• One on One meeting• Humor• Freedom to be creative and explore• Open Door Policy• Encourge the use and building of collabrative tools• DevOps Princples for managers• Be transparent with your work to everyone

My Inﬂuences• Michael Lopp (http://www.randsinrepose.com)• Kate Matsudaira (http://www.katemats.com)• Chad Dickerson (http://blog.chaddickerson.com)• John Allspaw (http://kitchensoap.com/)• Marc Hedlund (http://blog.precipice.org/)• Kellan Elliot-McCrea (https://twitter.com/#!/kellan)

Thank you for listening Questions?

http://dcxwiki.energy.com.au	111
http://www.scoop.it	67
http://www2.fundacioninlea.org	38
http://blog.sahsu.mobi	17
http://www.onlydoo.com	10
http://us-w1.rockmelt.com	6
http://pinterest.com	4
http://localhost	3
https://twitter.com	3
http://tonihopponen2.sparkboxes.net	3
http://dev.tabimpact.com	2
http://new.fluidinfo.com	2
https://twimg0-a.akamaihd.net	1
http://sundayparade.com	1
http://comtel.in	1
http://10.64.136.239	1
http://www.draac.com	1
http://dcxjira.energy.com.au	1
http://www.weebly.com	1

DevOps Picc12 Management Talk

by Michael Rembetsy

Accessibility

Categories

Upload Details

Usage Rights

19 Embeds 273

Statistics