Ian Margetts - ASOS’ Journey to Continuous Deployment
•
2 likes•1,696 views
ASOS has been improving its world for the last 8 years. It has grown from 2 teams in 2008 to over 50 in 2017, increased revenues massively in the same period and then embarked on a major re-architecture of its codebase with a large emphasis on cloud. This talk is about how we have approached the evolution of DevOps during that period – some of the mistakes we have identified and how it’s so not about tools but people, getting good people to care about Platform Engineering and engendering that behaviour in to teams.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Ian Margetts - ASOS’ Journey to Continuous Deployment
Similar to Ian Margetts - ASOS’ Journey to Continuous Deployment (20)
Recently uploaded
Recently uploaded (20)
Ian Margetts - ASOS’ Journey to Continuous Deployment
- 2. JOURNEY TO CONTINUOUS DEPLOYMENT IAN MAR GETTS
- 3. 2016 confidential AGENDA. • Who are we? • The Background • Why Re-architect? • V1 of DevOps • Alignment to Re-Platforming • The Results • What’s Next 2016 confidential
- 4. 2016 confidential T H E N U M B E R 1 F A S H I O N D E S T I N AT I O N F O R T W E N T Y S O M E T H I N G S W E S E L L O V E R 8 5 , 0 0 0 P R O D U C T S 1 3 9 M S I T E V I S I T S I N D E C E M B E R 2 0 1 6 1 4 M A C T I V E C U S T O M E R S F R O M A R O U N D T H E W O R L D W E S E L L T O A L M O S T E V E R Y C O U N T R Y I N T H E W O R L D S T I L L G R O W I N G O V E R 2 5 % Y O Y
- 5. 2016 confidential2016 confidential DIGITAL PLATFORM PRODUCT & CONTENT CUSTOMER ENGAGEMENT 3rd Party Solutions E.g Fred Hopper 3rd Party Solutions E.g Fred Hopper CORE eCOMMERCE Bag Service Bag Service Checkout Service Checkout Service Payment Service Payment Service Search Service Search Service Customer Profile Service Customer Profile Service Loyalty Service Loyalty Service STOCK & FULFILMENT StockStock Delivery OptionsDelivery Options ASOS CHANNELSASOS CHANNELS Product Service Product Service INTEGRATION PLATFORM ENTERPRISE PLATFORM Supply ChainSupply Chain FinanceFinance DATA PLATFORM Data Services PeoplePeople Data Lake StockStock RetailRetail OrderOrder ProductProduct IdentityIdentity PropensityPropensity
- 6. 2016 confidential2016 confidential DOMAIN ... Development Teams (1 or more) Platform Team Head Of Technology For Domain Principal Architect Principal Engineer Lead Business Analyst Lead QA Service Delivery Programme / Project Manager Shared Across Domain Supporting Domain Resources Solution Architect (1 or more) Platform Team Platform Team Platform Team Platform Lead Product Manager UX Designer (Team Dependent) Technology Core Functions PMO SecurityEnterprise Architect Change Mngmt Etc. Senior Business Analyst Business Analyst Agile Delivery Manager Dev Team Software Engineers QA (1 or more) Data Engineer (Team dependent)
- 7. 2016 confidential THE BACKGROUND 17.8 37.7 71.7 149.3 205 324 482 754 955 1119 1444 0 200 400 600 800 1000 1200 1400 1600 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Revenue (£m) Total 2 0 0 8 • £72m Revenue • 2 x software engineering teams • Single, monolithic code base hosted in traditional datacentre • Semi-automatic deployment process 2 0 1 6 • £1.5b Revenue • 54 x software engineering teams • Codebase largely re-architected and hosted in Azure • Fully automated deployment process
- 8. 2016 confidential2016 confidential • Technology constraints • Monolithic Application • Slow to change – Test and Deploy • Presentation & Logic intertwined • Unstable • 3rd Party Proxies hacked in • No resiliency • Business constraints • Single language • Single currency • Single channel (Web) Why re-architect? Content Delivery Network (Globally Deployed) Upto7seconds
- 9. V1 of DevOps A Centralised Framework • Used by everyone to push code from test to production • Build once, deploy many • Shared ownership (anyone can develop) • Custom to the needs of ASOS Centralised ALM / DevOps • ‘Own’ deployments and path to live to ensure consistency • Make core changes to framework • Support teams on making extensions • ‘Own’ environment management to ensure consistency
- 10. 2016 confidential THE RESULTS… R ELEA SES IN C R EA SED EXPONENTIALLY IN 4 YEARS, ATTR IB U TA B LE TO: • ALM framework development • Improved environments • Release management improvements • Start of the re-architecture into Azure 20 40 141 272 332 0 50 100 150 200 250 300 350 2011 2012 2013 2014 2015 Releases 2016 confidential
- 11. Aligning DevOps to Re-Platforming Re-architecture of the code base • From one code base to hundreds • Truly decoupled micro-services architecture • Move to Azure Emergence of the Platform Teams • From project to platform based • More responsibility for code and infrastructure • Both development and support responsibilities Release Demand • Business hunger for more regular features
- 12. What did we do? Goals • Empower teams • Improve team competency • Devolve team support • Support cloud How • Tooling • Rollout of competency model • Platform engineering – central and devolved
- 13. Did it work? • Teams taking ownership • Reduction in release effort • Resolving challenges more quickly (and earlier) • Metrics based conversations • All components released prior to Black Friday Challenges • Shifting Accountability to Platforms • Doing it at Scale • Business Demands
- 14. 20 40 141 272 332 487 1305 0 200 400 600 800 1000 1200 1400 2011 2012 2013 2014 2015 2016 2017 Releases
- 15. 2016 confidential WHAT’S NEXT? CULTURE • Get to grips with Autonomy • Drive Platform Engineering to Teams • Metrics based conversations to drive Continuous Improvement • DevOps to Ops TECHNOLOGY • Automate (where appropriate) • Data, data, data • Operate at scale • Information 2016 confidential
Editor's Notes
- So today’s talk will be a brief tour of the DevOps journey that ASOS has gone through – how we have responded to the needs of the wider business and where we are today My name is Ian Margetts, I have been with ASOS after leaving Lloyds TSB in 2008 – starting out as a Dev Manager in an organisation that not many people had heard of and now run the Platform Engineering capability in an internationally recognised company, responsible for supporting the rollout of DevOps approaches and tooling across the organisation. Its been a challenging and interesting 8 years and that’s what I will talk about now
- So, a bit about the company first – we were established in 2000 and currently have more than 3000 employees – we are on a mission to be the number 1 fashion destination for 20 somethings in the world So, who has heard of ASOS? Either you are shopping for yourselves or you have teenage kids who are busy filling your credit card up with our name The last bit of information is important for the rest of the session today – the extraordinary growth that we have had and continue to experience gives us an almost unique challenge, whatever we develop now has to be able to handle at least double that volume in 3 years I want to underpin this by starting at the end of the story - Black Friday 2016. In today’s environment, for many companies this is probably the biggest test of any IT solution. I will talk more about this later but 2016 was the year that we landed a significant portion of our re-architecture (predominantly in to the Azure cloud) and so we had a lot of new technology coming together at a time when we were about to experience our biggest weekend ever. Some stats from that period were: Sales grew 45% YoY for Black Friday week At its peak we were taking 33 orders per second through our systems – that’s 100% YoY Mobile traffic was bigger than ever – 68% of our total traffic On Black Friday we served 167m API requests with an average response time of 48ms Product service peaked out at 3083 requests per second Orders platform team had zero incidents, Platform Engineering were stood down
- We run 24 Platforms, supported by 54 teams – this equates to nearly 400 Engineers (mostly co-located)
- So now we have talked about the present let’s flip back and compare back to when I joined the company to today, this will hopefully give you some context for the challenges that we have had to overcome in the last 8 years Growth 20x in 8 years Our legacy platform was what ASOS was built on. It was functionally very rich and stable, but… It became more difficult to scale to match our growth rates It became too difficult to keep up with the velocity of change we wanted.. and it was all deployed in a single data centre in the UK which means it was not as globally distributed or as resilient as we wanted …and back in 2008 the process of deployment was a semi-automated one: a reasonably well developed set of PowerShell and other scripts orchestrated manually by the DevOps team. A fairly typical set up back then.
- So in 2011/12 we started to look at the challenges we had and ways in which to address them From this investigation we picked the approach that we ended up with – a predominantly Micro-Service based architecture in the Azure Cloud
- Similarly, prior to 2011 we largely carried out deployments in a semi-manual fashion, In 2011 /12 we took a step back from our manual deployment model and started the first steps in our DevOps journey. We addressed what we saw as the 2 key areas of pains for (the 6) teams, namely: The lack of a repeatable deployment process Unstable and unrepresentative test environments Click We developed an in-house custom solution that addressed the former – this was used by all teams to deploy code to test environments and had many good features, to get to production we continued to use our central DevOps function To address the latter – we created templated (but still manual) environments that represented the essence of our live environments Click All delivered by a centralised DevOps function – responsible for driving through change and to ensure consistency, as well as supporting the teams in their usage of this new way of working This was a significant first step in our DevOps journey: We had embedded the concept of repeatable, automated processes and deployments with the teams Started the cultural journey with teams so they understood the benefits of investing time early in path to live activities Set the precedent with the organisation that it was necessary (and beneficial) to invest heavily in the underlying plumbing, tools and competency support to maximise the benefit
- So what did these developments give us? We saw nearly a 9x increase in releases to production from 2012 to 2015, to now over 300 releases a year Now this sounds great (who is releasing to these today?) but the figures don’t tell the whole story… However, we understood that the centralised model could only take us so far. We were very aware that this model, albeit important to drive through the message and consistency of approach in the early days would not scale and get us to where we needed to be and so we again reviewed the options. To truly realise the potential of automation a more devolved model was required that would address the inherent limitations of the existing model
- During 2012 to 2015 a lot changed for ASOS that forced us to change the way in which we thought about DevOps Click The re-architecture of the code base. This largely happened form 2012 to 2016 and is important from a DevOps perspective, we now had the technical underpinning that we required to accelerate change: Teams could now think about Feature based releases rather than project based ones They were largely no longer dependent on each other Therefore the ability to release more regularly suddenly becomes a reality for them, in fact – there is little benefit in having such an architecture unless the ability to release regularly and safely is there Move to Azure was huge for us – suddenly could provision infrastructure and deploy code at will with less of the traditional constraints – this had its problems as well but overall, opened up our thinking Click Emergence of the Platforms During the process of re-architecture we set the mechanism up for teams to continue to manage their services after the programme had completed, what did this mean for ASOS? They now have a long term interest in the health of the platform They need / want to take on more responsibility over all aspects of the code base – whether it be in development or production The ability to understand their infrastructure and be able to release regularly is vital Most importantly, teams have now taken on support responsibility as well development so the need to have easy access to up to date, reliable information and be able to act on that quickly Click Release Demand ASOS is customer obsessed – as we started to unlock the potential to release more we needed to remove the impediments to teams. The existence of a central deployment model was seen as the biggest blocker to achieving the company goals and so we had to resolve this
- We recognised that to fully support the goals of the business we needed to make some key changes – the largest of which was to let go of central control of many of our areas so that we could scale in multiple directions Click What does this actually mean in practice? We needed to empower the teams to take on as much responsibility as they can – some examples of this might be their ability to create environments, create new release pipelines, release on demand but also easily hook up to monitoring solutions and drive improved support from all of these We needed to improve team competency – the goal is to make teams self sufficient (as much as possible) in a DevOps sense but many have a long way to go. DevOps is much like Agile as a term – a lot of people can say they ‘do’ Agile or ‘do’ DevOps but do not really understand what this means and that needs support and guidance over a long period We need to get support close to the team – This is closely linked to the above point but we cannot just be a competency, Platform Engineers at ASOS need to get involved and implement and support as well and so having just a central function will not meet that goal We need to support cloud development – I have called this out as a separate thing as even though ASOS is heavily invested in Azure we still have teams going in to it for the first time or adopting new technologies which require different thinking and approaches. As such there is a key role for any DevOps function to help drive this story and ensure consistency of approach ad understanding So that’s all great, we now had a list of things to go after but how have we met these goals? Click Tooling – VSTS / Octopus / Team City Tooling is not the answer to what is essentially a People and Practices challenge, however in ASOS we needed a reboot of DevOps and the tooling gave us a great way of breaking the link and connotations of the previous solution. We picked best of breed tooling that was widely used externally and had an eager audience, this then made the rest of the mission far easier to attack. Specific things this gave us was: Reliable, easy to use tooling - really started to focus the teams on development rather than deployment CI / CD Pipelines – did not have these before and gives much faster feedback and drives improvement Rollout of competency model Dedicated staff for each of the key areas of Engineering – Software development, QA, Data, BA, Platform Engineering Mixture of coaching, mentoring, hands on problem solving and retrospectives to improve both individuals and team skills and behaviours Cross competency sharing and initiatives to ensure that any enterprise challenges are addressed Introducing Platform Engineering – both central and devolved From the devolved perspective, we needed to get DevOps skills closer to the teams to support them in their day to day tasks but more importantly, to act as a competency and improve their capabilities – primarily from a metrics based discussion. We still have a need for a central function – to be the home of the competency, support the central infrastructure, support the guys in the teams and drive through Enterprise level concerns e.g. as an organisation we are just looking at Azure Service Fabric and my team are leading the DevOps aspects of that. For teams this means that they can just consume the results of those central efforts and focus on delivering features, rather than the underlying plumbing To bring this to life let’s talk about our Orders platform – Prior to M2 they looked like this: Fully on prem All deployments through centralised DevOps No control over infrastructure Dev / Test not fully representative of live Result – infrequent and large painful releases What we did We made the team responsible for DevOps activities so by August 2016 Partially in azure - Full control over both Azure and on prem through pipelines – where possible using dsc / arm to provision and configure these environments and Kibana to monitor them Competency support – PSE and PE, ensured that we did not compromise quality over delivery, ensured the health of the platform post delivery Migrated to new stack – All deployments carried out through CI / CD pipelines, gave the teams higher confidence through test automation and repeatable deployment patterns Much richer non-prod environments – They now reflected live (at least for cloud) so tests were trusted and environments could be rebuilt at will Result – Team moved to a model of releasing daily, lower post release issues and able to understand and share issues better through improved logging
- Throughout 2016 we saw the following things happen: Click Teams taking ownership Wanting to take on development of the pipelines themselves Actively involved in resolving issues and understanding root causes Reduction in release effort The vast majority of the increase in releases since 2016 has been via the new process and technology We have not upped the number of DevOps or Release Management staff carrying out releases centrally but will achieve a four fold increase in releases in 2 years Resolving challenges more quickly (and earlier) Use of pipelines, reliable, automated environments and automated tests within the lifecycle are identifying issues earlier Especially relevant where teams are purely Azure – they can create as production like environments as they need Starting to have metrics based conversations Up to now releases has been the Holy Grail, we have not focused on a lot of the other things that are equally, if not more important Now starting to work with teams on a wider set of metrics that defines their DevOps world and then driving a Continuous Improvement culture out of that And now for the punchline that underpins all of those: Click All components released prior to Black Friday Without the shift to the new tooling, efforts to allow teams to take on more and greater support we would not have released the large amount of change necessary to make our re-platforming efforts a success
- So, if we put it all together what has this all meant? In the space of 5 years since we started driving a DevOps programme within ASOS we have increased our release volume by 600x, over the next few years we can easily see this hitting 10,000 releases a year if we continue to follow and extend the culture of Continuous Improvement within ASOS.
- Everything we do is focused at continuing the devolvement of responsibility to the teams Culture Drive Platform Engineering to Teams Metrics based conversations to drive Continuous Improvement DevOps to Ops Technology Automate everything – Provide governance through automation (define the sandpit) and then let the teams get on and deliver Data, data, data Scale – By this I mean scaling the deployment infrastructure so that its way ahead of demand Information – Give the teams what they need to manage their estates and make good decisions (providing monitoring and finance info as a service), then let them act on it