SlideShare a Scribd company logo
The Importance of Culture
Building and Sustaining Effective Engineering
Organizations
Randy Shoup
@randyshoup
linkedin.com/in/randyshoup
Background
CTO at KIXEYE
• Real-time strategy games for web and mobile
Director of Engineering for Google App Engine
• World’s largest Platform-as-a-Service
Chief Engineer at eBay
• Multiple generations of eBay’s real-time search
infrastructure
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Hire and Retain the Best
Hire „A‟ Players
• In creative disciplines, top performers are 10x
more productive (!)
Confidence
• A players bring A players
• B players bring C players
Google Hiring
Goal: Only hire top talent
• False negatives are OK; false positives are
not
Hiring Process
• Famously challenging interviews
• Very detailed interviewer feedback
• Hiring committee decides whether to hire
• Separately assign new Googler to group
 Highly talented and engaged employees
Respect People
People are not interchangeable
• Different skills, interests, capabilities
• Create a Symphony, not a Factory
Most valuable and irreplaceable asset
• Treat people with care and respect
• If the company values its people, people will
provide value to the company
eBay “Train Seats”
eBay‟s development process (circa 2006)
• Design and estimate project
(“Train Seat” == 2 engineer-weeks)
• Assign engineers from common pool to implement
tasks
• Designer does not implement; implementers do not
design
 Dysfunctional engineering culture
• (-) Engineers treated as interchangeable “cogs”
• (-) No regard for skill, interest, experience
• (-) No pride of ownership in task implementation
• (-) No long-term ownership of codebase
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Service Teams
• Small, focused teams
• Single service or set of related services
• Minimal, well-defined “interface”
• Vendor – Customer relationships
• Clear “contract” between teams
• Functionality: agreed-upon scope of
responsibility
• Service levels and performance
Google Services
• All engineering groups organized into
“services”
• Gmail, App Engine, Bigtable, etc.
• Self-sufficient and autonomous
• Layered on one another
 Very small teams achieve great things
Autonomy and Accountability
• Give teams autonomy
• Freedom to choose technology, methodology,
working environment
• Responsibility for the results of those choices
• Hold team accountable for *results*
• Give a team a goal, not a solution
• Let team own the best way to achieve the
goal
KIXEYE Service Chassis
• Goal: Produce a “chassis” for building scalable
game services
• Minimal resources, minimal direction
• 3 people x 1 month
• Consider building on open source projects
 Team exceeded expectations
• Co-developed chassis, transport layer, service
template, build pipeline, red-black deployment, etc.
• Heavy use of Netflix open source projects
• 15 minutes from no code to running service in AWS
(!)
• Plan to open-source several parts of this work
Google and DevOps
Ops Support is a privilege, not a right
• Developers carry pager for first 6+ months
• Service “graduates” to SRE after intensive
review of monitoring, reliability, resilience, etc.
• SRE collaborates with service to move
forward
Everyone‟s incentives are aligned
• Everyone is responsible for production
• Everyone strongly motivated to have solid
instrumentation and monitoring
Collaboration
• One team across
engineering, product, operations, etc.
• Solve problems instead of pointing fingers
Google Co-Location
Multiple Organizations
• Engineering
• Product
• Operations
• Support
• Different reporting structures to different VPs
Virtual Team with Single Goal
• All work to make Google App Engine successful
• Coworkers are “Us”, not “Them”
• When asked which teams we need to sit next to, it
never occurred to us that other organizations were
not “our team”
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Quality over Quantity
Whole user / player experience
• Think holistically about the full end-to-end
experience of the user
• UX, functionality, performance, bugs, etc.
Less is more
• Solve 100% of one problem rather than 50%
of two
• Users prefer one great feature instead of two
partially-completed features
Institutionalize Quality
Development Practices
• Code reviews
• Continuous Testing
• Continuous Integration
Quality Automation
• Automated testing frameworks
• Canary releases to production
“Make it easy to do the right thing, and hard to do the
wrong thing”
Google Engineering Discipline
Solid Development Practices
• Code reviews before submission
• Automated tests for everything
• Single logical source code repository
 Internal Open Source Model
• Not “here is a bug report”
• Instead “here is the bug; here is the code fix;
here is the test that verifies the fix” 
Technical Tradeoffs
Make Tradeoffs Explicit
• Triangle: date vs. quality vs. features
• When you choose date and features, you
implicitly choose a level of quality
Manage Technical Debt
• Plan for how and when you will pay it off
• Maintain sustainable and well-understood level of
debt
“Don‟t have time to do it right” ?
• WRONG – Don’t have time to do it twice (!)
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Constant Learning
Any process, organization, or product can
always be improved
Mistakes are a learning opportunity
• What did you do -> What did you *learn*
• Take emotion and personalization out of it
Encourage iteration and velocity
• “Failure is not falling down but refusing to get
back up” – Theodore Roosevelt
Google Blame-Free Post-
Mortems
Post-mortem After Every Incident
• Document exactly what happened
• What went right
• What went wrong
Open and Honest Discussion
• What contributed to the incident?
• What could we have done better?
Engineers compete to take personal
responsibility (!)
Google Blame-Free Post-
Mortems
Action Items
• How will we change process, technology,
documentation, etc.
• How could we have automated the problems
away?
• How could we have diagnosed more quickly?
• How could we have restored service more
quickly?
Follow up (!)
Iteration and Experimentation
*Engineer* successes
• Constant iteration
• Launch is only the first step
• Assume you will not get it perfect on the first
try
• A / B Testing needs to be a core competence
Many small experiments sum to big wins
eBay Machine-Learned Ranking
Ranking function for search results
• Which item should appear 1st, 10th, 100th, 1000th
• Before: Small number of hand-tuned factors
• Goal: Thousands of factors
Experimentation Process
• Predictive models: query->view, view-
>purchase, etc.
• Hundreds of parallel A|B tests
• Full year of steady, incremental improvements
 2% increase in eBay revenue (~$120M)
Recap: Building Blocks of
Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Thank you!
rshoup@kixeye.com
@randyshoup
linkedin.com/in/randyshoup

More Related Content

What's hot

DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps TransitionDOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
Gene Kim
 
A CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling OrganizationsA CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling Organizations
Randy Shoup
 
Evolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBayEvolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBay
Randy Shoup
 
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine Learning
Randy Shoup
 
One Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us BetterOne Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us Better
Randy Shoup
 
DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How?
John Willis
 
Minimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughMinimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good Enough
Randy Shoup
 
Turning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational CapitalTurning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational Capital
John Willis
 
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
All daydevops   2016 - Turning Human Capital into High Performance Organizati...All daydevops   2016 - Turning Human Capital into High Performance Organizati...
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
John Willis
 
Immutable Service Delivery Shenzhen 2016
Immutable Service Delivery   Shenzhen 2016Immutable Service Delivery   Shenzhen 2016
Immutable Service Delivery Shenzhen 2016
John Willis
 
Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Software devops engineer in test (SDET)
Software devops engineer in test (SDET)
Sriram Angajala
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and Microservices
Randy Shoup
 
DevOps Picc12 Management Talk
DevOps Picc12 Management TalkDevOps Picc12 Management Talk
DevOps Picc12 Management Talk
Michael Rembetsy
 
Moving Fast at Scale
Moving Fast at ScaleMoving Fast at Scale
Moving Fast at Scale
Randy Shoup
 
Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness
John Willis
 
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
Yulia Shcherbachova
 
Tales from the Platform Trade
Tales from the Platform TradeTales from the Platform Trade
Tales from the Platform Trade
William Grosso
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
Aviran Mordo
 
DevOps for CTOs
DevOps for CTOsDevOps for CTOs
DevOps for CTOs
Burke Autrey
 
Serverless Toronto helps Startups
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps Startups
Daniel Zivkovic
 

What's hot (20)

DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps TransitionDOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
 
A CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling OrganizationsA CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling Organizations
 
Evolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBayEvolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBay
 
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine Learning
 
One Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us BetterOne Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us Better
 
DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How?
 
Minimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughMinimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good Enough
 
Turning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational CapitalTurning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational Capital
 
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
All daydevops   2016 - Turning Human Capital into High Performance Organizati...All daydevops   2016 - Turning Human Capital into High Performance Organizati...
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
 
Immutable Service Delivery Shenzhen 2016
Immutable Service Delivery   Shenzhen 2016Immutable Service Delivery   Shenzhen 2016
Immutable Service Delivery Shenzhen 2016
 
Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Software devops engineer in test (SDET)
Software devops engineer in test (SDET)
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and Microservices
 
DevOps Picc12 Management Talk
DevOps Picc12 Management TalkDevOps Picc12 Management Talk
DevOps Picc12 Management Talk
 
Moving Fast at Scale
Moving Fast at ScaleMoving Fast at Scale
Moving Fast at Scale
 
Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness
 
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
 
Tales from the Platform Trade
Tales from the Platform TradeTales from the Platform Trade
Tales from the Platform Trade
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
DevOps for CTOs
DevOps for CTOsDevOps for CTOs
DevOps for CTOs
 
Serverless Toronto helps Startups
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps Startups
 

Viewers also liked

The Importance of Culture
The Importance of CultureThe Importance of Culture
The Importance of Culture
Tristan White
 
Company Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right TeamsCompany Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right Teams
e27
 
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
Randy Shoup
 
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYEQCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
Randy Shoup
 
Concurrency at Scale: Evolution to Micro-Services
Concurrency at Scale:  Evolution to Micro-ServicesConcurrency at Scale:  Evolution to Micro-Services
Concurrency at Scale: Evolution to Micro-Services
Randy Shoup
 
Why Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudWhy Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the Cloud
Randy Shoup
 
Importance of Organizational Development
Importance of Organizational DevelopmentImportance of Organizational Development
Importance of Organizational Development
Amanda Jamison
 
Nonverbalcommunication 1
Nonverbalcommunication 1Nonverbalcommunication 1
Nonverbalcommunication 1
Sulman Ahmed
 
รายงานการตรวจรถ
รายงานการตรวจรถรายงานการตรวจรถ
รายงานการตรวจรถseedow
 
Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2
Donald Malloy
 
Ind eng-734-doc
Ind eng-734-docInd eng-734-doc
Ind eng-734-doc
Riverside School
 
Katrina sigma xi oct 2009 bob gilbert
Katrina sigma xi oct 2009 bob gilbertKatrina sigma xi oct 2009 bob gilbert
Katrina sigma xi oct 2009 bob gilbert
University of Texas at Austin
 
Social Experimentation
Social ExperimentationSocial Experimentation
Social Experimentation
muhammad shoaib
 
Professional societies
Professional societiesProfessional societies
Professional societies
Sulman Ahmed
 
Human Subjects Regulations
Human Subjects RegulationsHuman Subjects Regulations
Human Subjects Regulations
Janet Stemwedel
 
Beyond Disaster Recovery
Beyond Disaster RecoveryBeyond Disaster Recovery
Beyond Disaster Recovery
mkinzie
 
Law Enforcement Oath of Honor
Law Enforcement Oath of HonorLaw Enforcement Oath of Honor
Law Enforcement Oath of Honor
DiscoverPolicing
 
Discovering a Career in Law Enforcement
Discovering a Career in Law EnforcementDiscovering a Career in Law Enforcement
Discovering a Career in Law Enforcement
DiscoverPolicing
 
Software Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEESoftware Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEE
Samsuddoha Sams
 
From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015
Randy Shoup
 

Viewers also liked (20)

The Importance of Culture
The Importance of CultureThe Importance of Culture
The Importance of Culture
 
Company Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right TeamsCompany Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right Teams
 
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
 
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYEQCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
 
Concurrency at Scale: Evolution to Micro-Services
Concurrency at Scale:  Evolution to Micro-ServicesConcurrency at Scale:  Evolution to Micro-Services
Concurrency at Scale: Evolution to Micro-Services
 
Why Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudWhy Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the Cloud
 
Importance of Organizational Development
Importance of Organizational DevelopmentImportance of Organizational Development
Importance of Organizational Development
 
Nonverbalcommunication 1
Nonverbalcommunication 1Nonverbalcommunication 1
Nonverbalcommunication 1
 
รายงานการตรวจรถ
รายงานการตรวจรถรายงานการตรวจรถ
รายงานการตรวจรถ
 
Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2
 
Ind eng-734-doc
Ind eng-734-docInd eng-734-doc
Ind eng-734-doc
 
Katrina sigma xi oct 2009 bob gilbert
Katrina sigma xi oct 2009 bob gilbertKatrina sigma xi oct 2009 bob gilbert
Katrina sigma xi oct 2009 bob gilbert
 
Social Experimentation
Social ExperimentationSocial Experimentation
Social Experimentation
 
Professional societies
Professional societiesProfessional societies
Professional societies
 
Human Subjects Regulations
Human Subjects RegulationsHuman Subjects Regulations
Human Subjects Regulations
 
Beyond Disaster Recovery
Beyond Disaster RecoveryBeyond Disaster Recovery
Beyond Disaster Recovery
 
Law Enforcement Oath of Honor
Law Enforcement Oath of HonorLaw Enforcement Oath of Honor
Law Enforcement Oath of Honor
 
Discovering a Career in Law Enforcement
Discovering a Career in Law EnforcementDiscovering a Career in Law Enforcement
Discovering a Career in Law Enforcement
 
Software Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEESoftware Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEE
 
From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015
 

Similar to The Importance of Culture: Building and Sustaining Effective Engineering Organizations [QCon Beijing 2014]

Java DevOps at Enterprise Scale
Java DevOps at Enterprise ScaleJava DevOps at Enterprise Scale
Java DevOps at Enterprise Scale
Ryan McGuinness
 
How to create awesome customer experiences
How to create awesome customer experiencesHow to create awesome customer experiences
How to create awesome customer experiences
Morgan Simonsen
 
UX in Action: IBM Watson
UX in Action: IBM WatsonUX in Action: IBM Watson
UX in Action: IBM Watson
UserTesting
 
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisationLean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Craig Sullivan
 
Lean UX in an Agency Environment
Lean UX in an Agency EnvironmentLean UX in an Agency Environment
Lean UX in an Agency Environment
Jef Bekes
 
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Maurizio Mancini
 
Rethinking Your DevOps Strategy
Rethinking Your DevOps StrategyRethinking Your DevOps Strategy
Rethinking Your DevOps Strategy
Mandi Walls
 
Holistic Product Development
Holistic Product DevelopmentHolistic Product Development
Holistic Product Development
Gary Pedretti
 
Designing for efficiency.pdf
Designing for efficiency.pdfDesigning for efficiency.pdf
Designing for efficiency.pdf
Gabriela Véghová
 
Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...
Agile India
 
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
TEST Huddle
 
Adapt your testing for Agile
Adapt your testing for AgileAdapt your testing for Agile
Adapt your testing for Agile
Emma Armstrong
 
Agile ux fullday-uxpa2016
Agile ux fullday-uxpa2016Agile ux fullday-uxpa2016
Agile ux fullday-uxpa2016
UXPA International
 
Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test Automation
Naveen Kumar Singh
 
QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
Praveen Dosapati (CSM)
 
Agile Testing - What, why and how.
Agile Testing - What, why and how.Agile Testing - What, why and how.
Agile Testing - What, why and how.
Asim Kazmi
 
Ten Lessons of the DevOps Transition
Ten Lessons of the DevOps TransitionTen Lessons of the DevOps Transition
Ten Lessons of the DevOps Transition
Randy Shoup
 
Getting By Without "QA"
Getting By Without "QA"Getting By Without "QA"
Getting By Without "QA"
Dave King
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
Kari Kakkonen
 
Kanban testing
Kanban testingKanban testing
Kanban testing
Cprime
 

Similar to The Importance of Culture: Building and Sustaining Effective Engineering Organizations [QCon Beijing 2014] (20)

Java DevOps at Enterprise Scale
Java DevOps at Enterprise ScaleJava DevOps at Enterprise Scale
Java DevOps at Enterprise Scale
 
How to create awesome customer experiences
How to create awesome customer experiencesHow to create awesome customer experiences
How to create awesome customer experiences
 
UX in Action: IBM Watson
UX in Action: IBM WatsonUX in Action: IBM Watson
UX in Action: IBM Watson
 
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisationLean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
 
Lean UX in an Agency Environment
Lean UX in an Agency EnvironmentLean UX in an Agency Environment
Lean UX in an Agency Environment
 
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
 
Rethinking Your DevOps Strategy
Rethinking Your DevOps StrategyRethinking Your DevOps Strategy
Rethinking Your DevOps Strategy
 
Holistic Product Development
Holistic Product DevelopmentHolistic Product Development
Holistic Product Development
 
Designing for efficiency.pdf
Designing for efficiency.pdfDesigning for efficiency.pdf
Designing for efficiency.pdf
 
Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...
 
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
 
Adapt your testing for Agile
Adapt your testing for AgileAdapt your testing for Agile
Adapt your testing for Agile
 
Agile ux fullday-uxpa2016
Agile ux fullday-uxpa2016Agile ux fullday-uxpa2016
Agile ux fullday-uxpa2016
 
Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test Automation
 
QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
 
Agile Testing - What, why and how.
Agile Testing - What, why and how.Agile Testing - What, why and how.
Agile Testing - What, why and how.
 
Ten Lessons of the DevOps Transition
Ten Lessons of the DevOps TransitionTen Lessons of the DevOps Transition
Ten Lessons of the DevOps Transition
 
Getting By Without "QA"
Getting By Without "QA"Getting By Without "QA"
Getting By Without "QA"
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
 
Kanban testing
Kanban testingKanban testing
Kanban testing
 

More from Randy Shoup

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Randy Shoup
 
Anatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsAnatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and Lessons
Randy Shoup
 
Scaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermScaling Your Architecture for the Long Term
Scaling Your Architecture for the Long Term
Randy Shoup
 
Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020
Randy Shoup
 
Breaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsBreaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building Teams
Randy Shoup
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and Events
Randy Shoup
 
Learning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three IncidentsLearning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three Incidents
Randy Shoup
 
Managing Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsManaging Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and Events
Randy Shoup
 
Service Architectures at Scale
Service Architectures at ScaleService Architectures at Scale
Service Architectures at Scale
Randy Shoup
 
Managing Data in Microservices
Managing Data in MicroservicesManaging Data in Microservices
Managing Data in Microservices
Randy Shoup
 
Effective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldEffective Microservices In a Data-centric World
Effective Microservices In a Data-centric World
Randy Shoup
 

More from Randy Shoup (11)

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
Anatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsAnatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and Lessons
 
Scaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermScaling Your Architecture for the Long Term
Scaling Your Architecture for the Long Term
 
Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020
 
Breaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsBreaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building Teams
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and Events
 
Learning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three IncidentsLearning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three Incidents
 
Managing Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsManaging Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and Events
 
Service Architectures at Scale
Service Architectures at ScaleService Architectures at Scale
Service Architectures at Scale
 
Managing Data in Microservices
Managing Data in MicroservicesManaging Data in Microservices
Managing Data in Microservices
 
Effective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldEffective Microservices In a Data-centric World
Effective Microservices In a Data-centric World
 

Recently uploaded

peru primero de la alianza con el pacifico
peru primero de la alianza con el pacificoperu primero de la alianza con el pacifico
peru primero de la alianza con el pacifico
FernandoGuevaraVentu2
 
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call GirlsBangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
narwatsonia7
 
The Principal Up-and-Coming Risks to Cloud-Based Security!
The Principal Up-and-Coming Risks to Cloud-Based Security!The Principal Up-and-Coming Risks to Cloud-Based Security!
The Principal Up-and-Coming Risks to Cloud-Based Security!
Alec Kassir cozmozone
 
IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024
IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024
IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024
APNIC
 
cyber crime.pptx..........................
cyber crime.pptx..........................cyber crime.pptx..........................
cyber crime.pptx..........................
GNAMBIKARAO
 
169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...
169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...
169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...
tanichadda371 #v08
 
DocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptxDocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptx
AmitTuteja9
 
KubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial IntelligentKubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial Intelligent
Emre Gündoğdu
 
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
thezot
 
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
APNIC
 
India Cyber Threat Report of 2024 with year
India Cyber Threat Report of 2024 with yearIndia Cyber Threat Report of 2024 with year
India Cyber Threat Report of 2024 with year
AkashKumar1733
 
HistorySrSec2024 daahi sadhin sgg-25.pdf
HistorySrSec2024 daahi sadhin sgg-25.pdfHistorySrSec2024 daahi sadhin sgg-25.pdf
HistorySrSec2024 daahi sadhin sgg-25.pdf
AdiySgh
 
Decentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and EsportsDecentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and Esports
Federico Ast
 
Unlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENT
Unlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENTUnlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENT
Unlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENT
keshavtiwari584
 
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
Web Inspire
 
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENTUnlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
rajesh344555
 
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
dtagbe
 
Lesson6 in spreadsheet 2024 for g12..ppt
Lesson6 in spreadsheet 2024 for g12..pptLesson6 in spreadsheet 2024 for g12..ppt
Lesson6 in spreadsheet 2024 for g12..ppt
ReyLouieSedigo1
 
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENTUnlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
rajesh344555
 
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. ITNetwork Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Sarthak Sobti
 

Recently uploaded (20)

peru primero de la alianza con el pacifico
peru primero de la alianza con el pacificoperu primero de la alianza con el pacifico
peru primero de la alianza con el pacifico
 
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call GirlsBangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
 
The Principal Up-and-Coming Risks to Cloud-Based Security!
The Principal Up-and-Coming Risks to Cloud-Based Security!The Principal Up-and-Coming Risks to Cloud-Based Security!
The Principal Up-and-Coming Risks to Cloud-Based Security!
 
IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024
IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024
IPv6: Unlocking the Potential, presented by Paul Wilson at CommunicAsia 2024
 
cyber crime.pptx..........................
cyber crime.pptx..........................cyber crime.pptx..........................
cyber crime.pptx..........................
 
169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...
169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...
169+ Call Girls In Navi Mumbai | 9930245274 | Reliability Escort Service Near...
 
DocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptxDocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptx
 
KubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial IntelligentKubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial Intelligent
 
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
 
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
 
India Cyber Threat Report of 2024 with year
India Cyber Threat Report of 2024 with yearIndia Cyber Threat Report of 2024 with year
India Cyber Threat Report of 2024 with year
 
HistorySrSec2024 daahi sadhin sgg-25.pdf
HistorySrSec2024 daahi sadhin sgg-25.pdfHistorySrSec2024 daahi sadhin sgg-25.pdf
HistorySrSec2024 daahi sadhin sgg-25.pdf
 
Decentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and EsportsDecentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and Esports
 
Unlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENT
Unlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENTUnlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENT
Unlimited Fun With Call Girls Hyderabad ✅ 7737669865 💘 FULL CASH PAYMENT
 
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
 
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENTUnlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
 
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
 
Lesson6 in spreadsheet 2024 for g12..ppt
Lesson6 in spreadsheet 2024 for g12..pptLesson6 in spreadsheet 2024 for g12..ppt
Lesson6 in spreadsheet 2024 for g12..ppt
 
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENTUnlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
 
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. ITNetwork Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
 

The Importance of Culture: Building and Sustaining Effective Engineering Organizations [QCon Beijing 2014]

  • 1. The Importance of Culture Building and Sustaining Effective Engineering Organizations Randy Shoup @randyshoup linkedin.com/in/randyshoup
  • 2. Background CTO at KIXEYE • Real-time strategy games for web and mobile Director of Engineering for Google App Engine • World’s largest Platform-as-a-Service Chief Engineer at eBay • Multiple generations of eBay’s real-time search infrastructure
  • 3. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 4. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 5. Hire and Retain the Best Hire „A‟ Players • In creative disciplines, top performers are 10x more productive (!) Confidence • A players bring A players • B players bring C players
  • 6. Google Hiring Goal: Only hire top talent • False negatives are OK; false positives are not Hiring Process • Famously challenging interviews • Very detailed interviewer feedback • Hiring committee decides whether to hire • Separately assign new Googler to group  Highly talented and engaged employees
  • 7. Respect People People are not interchangeable • Different skills, interests, capabilities • Create a Symphony, not a Factory Most valuable and irreplaceable asset • Treat people with care and respect • If the company values its people, people will provide value to the company
  • 8. eBay “Train Seats” eBay‟s development process (circa 2006) • Design and estimate project (“Train Seat” == 2 engineer-weeks) • Assign engineers from common pool to implement tasks • Designer does not implement; implementers do not design  Dysfunctional engineering culture • (-) Engineers treated as interchangeable “cogs” • (-) No regard for skill, interest, experience • (-) No pride of ownership in task implementation • (-) No long-term ownership of codebase
  • 9. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 10. Service Teams • Small, focused teams • Single service or set of related services • Minimal, well-defined “interface” • Vendor – Customer relationships • Clear “contract” between teams • Functionality: agreed-upon scope of responsibility • Service levels and performance
  • 11. Google Services • All engineering groups organized into “services” • Gmail, App Engine, Bigtable, etc. • Self-sufficient and autonomous • Layered on one another  Very small teams achieve great things
  • 12. Autonomy and Accountability • Give teams autonomy • Freedom to choose technology, methodology, working environment • Responsibility for the results of those choices • Hold team accountable for *results* • Give a team a goal, not a solution • Let team own the best way to achieve the goal
  • 13. KIXEYE Service Chassis • Goal: Produce a “chassis” for building scalable game services • Minimal resources, minimal direction • 3 people x 1 month • Consider building on open source projects  Team exceeded expectations • Co-developed chassis, transport layer, service template, build pipeline, red-black deployment, etc. • Heavy use of Netflix open source projects • 15 minutes from no code to running service in AWS (!) • Plan to open-source several parts of this work
  • 14. Google and DevOps Ops Support is a privilege, not a right • Developers carry pager for first 6+ months • Service “graduates” to SRE after intensive review of monitoring, reliability, resilience, etc. • SRE collaborates with service to move forward Everyone‟s incentives are aligned • Everyone is responsible for production • Everyone strongly motivated to have solid instrumentation and monitoring
  • 15. Collaboration • One team across engineering, product, operations, etc. • Solve problems instead of pointing fingers
  • 16. Google Co-Location Multiple Organizations • Engineering • Product • Operations • Support • Different reporting structures to different VPs Virtual Team with Single Goal • All work to make Google App Engine successful • Coworkers are “Us”, not “Them” • When asked which teams we need to sit next to, it never occurred to us that other organizations were not “our team”
  • 17. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 18. Quality over Quantity Whole user / player experience • Think holistically about the full end-to-end experience of the user • UX, functionality, performance, bugs, etc. Less is more • Solve 100% of one problem rather than 50% of two • Users prefer one great feature instead of two partially-completed features
  • 19. Institutionalize Quality Development Practices • Code reviews • Continuous Testing • Continuous Integration Quality Automation • Automated testing frameworks • Canary releases to production “Make it easy to do the right thing, and hard to do the wrong thing”
  • 20. Google Engineering Discipline Solid Development Practices • Code reviews before submission • Automated tests for everything • Single logical source code repository  Internal Open Source Model • Not “here is a bug report” • Instead “here is the bug; here is the code fix; here is the test that verifies the fix” 
  • 21. Technical Tradeoffs Make Tradeoffs Explicit • Triangle: date vs. quality vs. features • When you choose date and features, you implicitly choose a level of quality Manage Technical Debt • Plan for how and when you will pay it off • Maintain sustainable and well-understood level of debt “Don‟t have time to do it right” ? • WRONG – Don’t have time to do it twice (!)
  • 22. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 23. Constant Learning Any process, organization, or product can always be improved Mistakes are a learning opportunity • What did you do -> What did you *learn* • Take emotion and personalization out of it Encourage iteration and velocity • “Failure is not falling down but refusing to get back up” – Theodore Roosevelt
  • 24. Google Blame-Free Post- Mortems Post-mortem After Every Incident • Document exactly what happened • What went right • What went wrong Open and Honest Discussion • What contributed to the incident? • What could we have done better? Engineers compete to take personal responsibility (!)
  • 25. Google Blame-Free Post- Mortems Action Items • How will we change process, technology, documentation, etc. • How could we have automated the problems away? • How could we have diagnosed more quickly? • How could we have restored service more quickly? Follow up (!)
  • 26. Iteration and Experimentation *Engineer* successes • Constant iteration • Launch is only the first step • Assume you will not get it perfect on the first try • A / B Testing needs to be a core competence Many small experiments sum to big wins
  • 27. eBay Machine-Learned Ranking Ranking function for search results • Which item should appear 1st, 10th, 100th, 1000th • Before: Small number of hand-tuned factors • Goal: Thousands of factors Experimentation Process • Predictive models: query->view, view- >purchase, etc. • Hundreds of parallel A|B tests • Full year of steady, incremental improvements  2% increase in eBay revenue (~$120M)
  • 28. Recap: Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation