Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK
Similar to Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK (20)
Recently uploaded
Recently uploaded (20)
Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK
- 1. Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK @4ndyHumphrey
- 2. The Auto Trader Bit
- 3. Our Magazine History 1977 1988 1995 2008 2013 • Online
- 4. Our Online History – AutoTrader.co.uk 1996 2002 2010 2011 2015
- 5. Our People PRIVATE Car Sellers Trade Car Dealers 30,000 15,000 Auto Trader Staff Product & Tech Teams 850 275 Our Customers
- 6. Our Technology Platform Servers2400 UK Datacentres2 Services switched routinely
- 7. Our Technology Platform 1.2 billion page views per month 70 million peak page views per day 15 million unique visitors per month Supported by 100 live applications
- 8. Technology Architecture, Tech Stack Organisation Structure, Culture, Location, Finance Product Project Methodology, Product Design Service Transition
- 9. A process that’s evolving ITIL Foundation 1 Organise as a Digital Company Agile Transformation Continuous Product Discovery 32 4
- 10. Our ITIL Foundation “What gets measured gets improved” Peter Drucker 1
- 11. Strong Service Management Team
- 12. Getting our platform under control Reporting and measurement Continuous Improvement Pragmatic, Focused on Business Value Our ITIL foundation
- 13. Challenges we faced Started to hit limitations Too bureaucratic Safety through control (change freeze?) Un-scalable Focussed on process over relationships
- 14. Complicated manual processes Detailed planning Extensive approval process Releases are complicated and costly Infrequent Releases Example: Service Transition
- 16. What is Agile? - Values Individuals and interactions over Processes and tools Working software over Comprehensive documentation Customer collaboration over Contract negotiation Responding to change over Following a plan http://www.agilemanifesto.org/
- 17. What is Agile? - Practices Incrementally Instead of all at once
- 18. Effect on Service Transition
- 19. More face to face communication
- 21. Kanban for small changes Kanban for small changes
- 22. More automation of build and test
- 23. Less, more often Smaller releases - Faster
- 24. Release all week Safer releases – Open the floodgates
- 25. Release Team as Go-Betweens Scaled using Release Team as go-betweens
- 26. “Monitoring is a tool for building a relationship between ops, developers and business” Noah Zucker: https://twitter.com/noahlz
- 27. Effects on Release volume / success FY10 FY11 FY12 FY13 Overall Success 495 699 899 1293 Total Releases 582 781 980 1388 0 200 400 600 800 1000 1200 1400 1600 Release Volume over Time
- 28. Effects on Release volume / success FY10 FY11 FY12 FY13 Success Rate 85.05% 89.50% 91.73% 93.16% 80.00% 85.00% 90.00% 95.00% 100.00% Release Success rate over Time
- 29. Organize as a Digital Comapany3
- 30. till some Organisational Problems Still some organisational problems
- 31. Still some Release Problems Still some Release Process problems
- 32. Re-organization 2013 Product and Technology Teams brought together Manchester
- 33. Spotify Engineering model Small, multi-disciplinary team Autonomous Based around a Purpose Led by Tech and Product experts https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
- 35. TRIBE – Audience + Brand TRIBE – Retail ProductsTRIBE – Customer Experience
- 36. What did that mean for Service Transition?
- 38. Auto Deploy Automate deployments in Production
- 40. Long Automated Release migration 0 10 20 30 40 50 60 70 Mar 2014 Apr 2014 May 2014 Jun 2014 Jul 2014 Aug 2014 Sep 2014 Oct 2014 Nov 2014 Dec 2014 Jan 2015 Feb 2015 Mar 2015 Apr 2015 May 2015 Jun 2015 Auto Deployments Manual Deployments Deployments by week: Auto vs Manual
- 41. Changing Release process again Gradually given Delivery Squads the ability to LIVE release Start to push Operational responsibility back to Delivery Squads Our purpose: “To lead Continuous Delivery and innovation of the Auto Trader platform” Operations teams then Re-organized as Squads.
- 42. Effect on our Service Transition Process 0 500 1000 1500 2000 2500 3000 FY11 FY12 FY13 FY14 FY15 FY16 Release Totals Overall Success Total Releases
- 43. Effect on our Service Transition Process 93.0% 94.0% 95.0% 96.0% 97.0% 98.0% 99.0% FY11 FY12 FY13 FY14 FY15 FY16 Release Success
- 44. Impact on working culture End of Project Culture End of Contractor Army Operational and Technical issues highly valued Improved operational responsibility within Squads
- 45. Continuous Product Discovery “Prediction is difficult, especially about the future” Niels Bohr 4
- 46. Still some Organisational Problems Starting to build the things right… but are we building the right things? How do we build the right things right?
- 47. Product Discovery Discover which changes add value by iterating designs and experiments with your customers
- 48. Change in approach New Requirements Become Assumptions We know Becomes We believe Let’s build it Becomes Lets test it! http://www.slideshare.net/jgothelf/better-product-definition-with-lean-ux-and-design-thinking
- 49. Data centered Scientific Iterative approach Not just Software - All Business Changes e.g. Process!
- 50. Product Discovery – effects on Service Transition Squads need to prototype and experiment more with their customers All ideas and decisions are validated with Data Another step change in the numbers of changes Needs new levels of of automation and self service
- 51. Our Business is continuing to grow and change http://www.londonstockexchange.com/
Editor's Notes
- Intro to me: Why I’m here Introduce Jono and Pete What we’re going to cover Tell you a bit about Auto Trader – bit of context Then talk about how Service Transition has changed as our organisation has changed. DevOps is not mentioned much here. But hopefully you’ll see the themes of cultural change, collaboration, automation , lean thinking coming through really strongly.
- AutoTrader grew as a publishing institution in the UK Selling classified magazines with private and trade cars and motorbikes for sale Bikes, Commercial vehicles etc. 400,000 magazines per week at it’s heyday It’s a really well known brand- 92% of the UK population know what we do and what our brand stands for. Last magazine printed in 2013
- We’re also a great story about moving a business from print to digital A journey that started for us in 1996 Rapidly that part of the the business expanded until the vast majority of revenue was generated from digital services and the magazine part of our business slowly reduced in size. POINT IS We are a digital business and aspire to be one of the best digital businesses in the world BUT We are not a start-up 20 years of digital history and decades of print history http://www.autotrader.co.uk/digital http://about-us.autotrader.co.uk/
- Private Sellers: us selling our Cars Trade Car Dealers 15,000 - Independent dealers, Franchise dealers, Car Supermarkets
- Availability at 99.99%
- Availability at 99.99% supporting products for Consumers, Private Sellers and Trade Retailers. Supporting access across multiple platforms Supporting Commercial and International Autotrader sites e.g. Dealer Websites Automotive leader for dealer websites with just under 5000 dealers’ sites hosted
- So I’m going to talk about Service Transition – how we safely build and release new services and changes to services. It’s where I’ve worked for the past few years and its really interesting because it’s affected by loads of factors – how our organqisatio is structured, out culture, how we deiliver work, the technology we use, architecture etc. Service Transistion is a key enabler for the business as well – something that will either drive your business forward, or hold you back. Hopefully it will be useful to look at the changes, and how we’ve adapted and improved our Service to our Customers What factors can affect Service Transition Organisational structure / Changes Financial model of organisation Project Methodologies Approach to product development Physical location People / Culture / Habits Risk aversion, Regulatory pressure Tech stack Architectural decisions Service Transition Why it’s cool / important Getting an idea out of someone’s head, to delivering value for a customer (in a way that is supportable and reliable) Touches loads of people, skillsets, departments Can be a technical/logistical challenge Should be quick and painless Our jobs are to make it speedy , painless and safe
- SO I’m going to talk through a few aspects of how our Service Transition process has changed Although this is a timeline – and this has been a journey – not one stage has replaced the previous – it might be better to think of this as layers on top of eachother that all contribute to the way we work at the moment. Hopefully it will be useful to look at the changes, and how we’ve adapted and improved our Service to our Customers
- When we started implementing our ITIL processes, our Digital Services were pretty unreliable I’d like to tell you how unreliable, but we didn’t measure too much!! We had major incidents weekly (some periods daily) - our main website down for hours at a time. A lot of our working practices were uncontrolled and inconsistent
- We had a strong core team of Service Managers Initially in dedicated Service Management team to implement ITIL based processes. Then distributed as leads throughout our Operations team
- Positives – that we keep TO THIS DAY – and still really value. reporting and measurement key to management Continuous Improvement is part of our day to day roles we are pragmatic – we adopt what’s useful and adapt to our business Focussed on Customers and Business Value
- As our business started to change more rapidly, Too bureaucratic – 2 hours CAB meetings for each development department -Insisting all product changes are documented for CAB approval. unnecessary documentation/approvals Safety through control Big Change Freezes - taking access/responsibility away Un-scalable e.g. Focussed on planning, implementing and recovering from infrequent complicated, risky releases Focussed on process over relationships- handovers, tickets and documentation between teams and departments.
- Out of hours implementation Frequent failures Complicated manual processes Focussed on detailed planning Extensive documentation and approval process Releases of software were infrequent, complicated and costly Big batch size – INFREQUENT releases
- 2009 started to change how we deliver our projects 1 project scaling out to all projects Need to still be responsive Cannot have more and more change meetings
- AGILE is an iterative approach to software delivery that builds software incrementally from the start of the project, instead of trying to deliver it all at once near the end. Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan
- It works by breaking projects down into little bits of user functionality called user stories, prioritizing them, and then continuously delivering them in short two week cycles called iterations.
- The most common complaint I heard from Ops during the “agile migration” was that the devs would no longer document new services. In reality, most of that documentation was useless anyway as no-one maintained it. What agile actually did was break down barriers for communication, through standups, retros etc. Through bringing stakeholders into the project. Through making project walls/backlogs visible. Documentation largely replaced by standups, showcases, retro’s = better/more immediate conversation Ditch pointless paperwork. But only the pointless stuff. Anything with a point can stay. Basically, for releases we trimmed out all the fat and were left with a calendar entry & release note The documentation dev used to produce for ops was replaced with a wiki page, maintained by the release team The devs also create documentation, for their own use.
- Daily stand-ups, RETROS, SHowcases.
- CAB as a 5-10min standup focussing on the next few weeks, instead of a lengthy meeting CAB just for service-affecting changes
- Kanban for small changes, allowing the Ops teams to be open about the prioritisation of work etc..
- Change in build practices for test environments. Many builds in non-live per day. Automated tests!!!! Continuous Integration But it stopped there
- Smaller, more frequent releases, to de-risk the act of releasing
- With the greater volume of releases, and the improvement in safety, No more out of hours deployment (very few – maybe 1%_2%)) Change freezes become much reduced (maybe not on Christmas Day eh?) it was not only necessary to widen the release window, but finally possible.
- Scaled using Release Team as go - betweens Release team link Development and Operations teams Attending standups and being the public face of Ops Ensuring documentation is done Preparing the way with Infrastructure teams Scheduling and managing releases Fixing issues
- We started to see interest from Developers who were keen to know how their apps were performing. Firstly in Test environments then quickly in LIVE GRAPHITE became the common language to interact with Development teams. Development and Operations teams starting working together a lot more
- Releasing less (smaller change) more frequently has the obvious effect on volume. It also allows QA’s to focus on fewer changes, which increases the chance of catching something before it goes live. “The implementation of Release Management within AutoTrader is one of the most mature that Pink Elephant has ever witnessed and AutoTrader is firmly within the top 10 organisations in terms of Release maturity scoring.” - Pink Elephant
- Releasing less (smaller change) more frequently has the obvious effect on volume. It also allows QA’s to focus on fewer changes, which increases the chance of catching something before it goes live. Pink Elephant quote The implementation of Release Management within TMG is one of the most mature that Pink Elephant has ever witnessed and TMG is firmly within the top 10 organisations in terms of Release maturity scoring. More importantly, the process reports show a clear year on year improvement since 2010 in terms of real business benefit, despite a year on year increase in workload as the business model of TMG has changed and more agility and flexibility has been required of the IT department. The Release management process has seen its workload increase 300% in terms of numbers of Releases handled, since 2010 but has also increased the success rate from 85% to 93% during that timeframe.
- Different Reporting Lines for Product and Technology Many physical locations Focus on PROJECTS - New agile shiny products Big bursts of activity in one area while other products get neglected. Capital vs Operational budget lines
- Manual Processes for deployment – can’t hire any more people! Lack of ‘Operational Mindset’ within teams – no way to get issues addressed Technology roles not ‘decision makers’ Still little focus on quality after launch – developers move off the project on launch day Lack of ownership - developers moving from one project to the next Large number of contractors – 40%
- We got a new CEO Stopped producing a Magazine. Now a DIGITAL BUSINESS Product and Technology Teams brought together Consolidate into 2 offices, all tech teams in Manchester Every team has a Product Lead and a Tech Lead
- Stopped producing a Magazine. Now a DIGITAL BUSINESS Product and Technology Teams brought together Consolidate into 2 offices, all tech teams in Manchester Every team has a Product Lead and a Tech Lead
- Each squad will have a purpose around a particular service or product Squads aligned to our customers With freedom (***) to carry out that purpose as best they can
- Where squads are working towards a similar market or customer base they are grouped together into a TRIBE Tribes then have leaders who set a wider strategic goal We’ve got about 4 tribes across our organisation
- WE DID NOT RE-STRUCTURE OPERATIONS We created a Squad within Operations #Continuous Delivery Squad Ops Engineers and Developers Chance to treat our Operational tools and processes properly! Focussing on Automated Deployment Monitoring Provisioning of environments Auto Deploy Bit of a stop-gap Single Click deployment to LIVE environments Use Continuous Integration pipelines and deployment scripts Triggered by Release Team Reduce the time to release and manual effort No more constraints on Deploy Engineers’ time Less need to schedule Squads Release software within a few minutes of passing QA We start treating Operational tools the same as External Services
- Single Click deployment to LIVE environments Mostly goverened by the same Release team Reduce the manual effort of releasing Remove constraint around scheduling and massively improve release bandwidth We have up to 5+ releases ongoing at the same time We kept Calendar entries Recording and monitoring more automated Admin less automated
- Improved Monitoring and Visualisation AGILE – visibility of information
- Competative advantage: Changing Data provider Gone from 40 manual releases a week to 4 Gone from 4 Auto Releases to 45 Note the 41 auto releases for CAP Switchover
- Start to give Delivery teams the ability to LIVE release! Start to push responsibility back to Delivery teams Restricted by Access to live systems Complexity of fixing anything Interdependent services (some too tightly coupled) Delivery Teams not aware of wider platform/operational issues Real desire within our Business to experiment and try things out Product Discovery,
- So that gets us almost up today But what challenges do we face now - and how do we want to improve? We’ve focused a lot on how to build things RIGHT Product discovery is about making sure we build the right thing
- Hands up: Who works on projects in your job? Keep your hands up if you’ve worked hard on a project that never saw the light of day? (never got released to a customer?) EXAMPLE – Damien working on 2 projects for 18 months that didn’t go live. An endemic problem for everyone in a digital world is knowing what the right thing is to build Lean Enterprise p47 Long product development cycles dramatically reduce the potential return on investment we can achieve from successful new products. • Long development cycles delay the time it takes to get customer feedback on whether we are building something valuable. • Typical market research activities are poor at predicting a product/market fit, especially in new product categories. Research said that minivans and iPods would not be successful. • In the absence of good data, people tend to get their pet projects funded. Particularly in enterprise IT, we often see spectacular amounts of money poured down the drain on systems replacement
- http://www.slideshare.net/jgothelf/better-product-definition-with-lean-ux-and-design-thinking?next_slideshow=1 Key questions: How long do we wait for launch How do we define the right requirements for our product What signals are we looking for from the market REQUIREMENTS are actually ASSUMPOTIONS e,.g early assumptions Who is our customer AMAZON Run thousands of experiments per day Roughly on third will add some value, one third will do nothing and one third will be detrimental They have not found a way of predicting which experiment will delivery which outcome. Lean Enterprise p32 Data gathered from evolving web-based systems reveals that the plan-based approach to feature development is very poor at creating value for customers and the organization. Amazon and Microsoft (along with many startups) use a technique called A/B testing to gather data on whether a feature will actually deliver value to users before it gets built in full. Ronny Kohavi, who directed Amazon’s Data Mining and Personalization group before joining Microsoft as General Manager of its Experimentation Platform, reveals the “humbling statistics”: 60%–90% of ideas do not improve the metrics they were intended to improve. Based on experiments at Microsoft, 1/3 of ideas created a statistically significant positive change, 1/3 produced no statistically significant difference and 1/3 created a statistically significant negative change.12 All of the ideas tested were thought to be good ones—but neither intuition nor expert opinion are good gauges of the value our ideas have for users.
- Starting to build the things right… but are we building the right things? EXAMPLE – Damien working on 2 projects for 18 months that didn’t go live. Call what we think will deliver value to our customers ASSUMPTIONS (not requirements) Start with the highest risk assumptions Design a testable hypothesis so we can run an experiment with a measurable result. Do the minimum amount of work required to measure success of that experiment Outcomes are important – it’s not what we want to build but what outcome we want from the problem we solve. Outcomes not outputs – examples.
- Creation of prototypes and experiments Getting to a world where change is continual. Each of these changes is likely to be a release A/B testing, MVT, changes to logging , metrics,
- Leading Continuous Delivery Watching for big batches! (Big batch police) Forcing changes to be easier and safer. e.g More reliance on signals/metrics/data to quickly interpret Identifying constraints within that value stream and working with Squads to fix them Limiting factors for Release process Culture within delivery teams – manual testing, admin, How quickly we can get feedback and change again Automated recording of releases Still measure success/ failure and take actions to improve
- So our business is continuing to grow and change We floated on the London Stock exchange last year and I naively thought this would be a good slide to demonstrate how we are changing. (If anyone knows what’s happended to the stock markets since December then you’ll know this was a good time to take a picture) But we’re outperforming the market and we’re outperforming our competition When our CEO talks to investors about Auto Trader as a company - he will talk about how we work, our culture, our agility and responsiveness to market changes – our ability to know what delivers value to customers and focus on that and not waste time and money. Service Transition is a great indicator about how our company works, but also been a key enabler to delivering value to our customers.