Welcome to Introduction to DevOps with Chocolate and LEGO game. You are about to participate in the end-to-end gamified simulation of product development. This workshop has been played in 7 countries and has been modified each time based on the feedback. I am looking forward to hearing your ideas and to getting them into next iteration!
Hello everyone, as you walk into the room, please pick a table and ask your neighbor 2 questions. What did you learn? Who at your table is using DevOps practices today? Who is just starting to learn about DevOps?
1:34 My name is Dana Pylayeva, I am an agile coach at HBC digital. Based in NYC, The game you are about to play was inspired by The Phoenix project as well as my own experience being on both side of the Wall of Confusion. First as a Java developer and later as a DBA manager.
1:35 How many of you have heard the term “Wall of confusion“ – Coined by Andrew Shafer, this term refers to common misunderstandings between dev and ops in traditional organizations. These include goals misalignment, tooling and processes.
1:36 Let’s look closely at what happens in a typical development organization. Typical dev organization is using Scrum. Scrum team is focused on delivering features faster. At the ends of each sprint they are expected to deliver potentially shippable increment of the product. They are throwing it over the wall and start the next sprint. What happens to it afterwards, is not their concern – they are DONE!
Scrum Team – Focus on Delivery and Speed
1:37 What do we have on the other side of the wall? Life isn’t pretty – escalation procedure, on-call pager duty, monitoring – everything to ensure that current systems that generate revenue are stable, have adequate throughput. Operation team is focused on keeping the lights on. Operations resist change, they know, when all these increments accumulate in It operations, they can cause deployment issues and lead to post-deployment fires in production. Ops Team – Focus on Stability and Reliability
1:39 Why do we care about trust? Because in it’s absence we tend to introduce extra validation steps for each hand-off in the value stream. It doesn’t have to be this way!
1:40 According to 2016 State of DevOps Report by Puppet Labs, organization that use DevOps practices are able to achieve both – increase in speed and increase in stability.
DevOps movement came a long way since 2009. 2009 – 10 deploys per day at Flickr 2011 – 11.6 sec over 7 K deployments per day 2016 – Information from DevOps Enterprise summit. – all these companies are implementing Devops practices now and accelerating their delivery of services.
And if the Department of Homeland Security can do it, so can you! Let’s get some experience with DevOps transformation by playing a close-to real life DevOps transformation RPG
Inspired by The Phoenix Project and The Goal, this game will let you experience 3 sprints in the life of development and operations. We will simulate the flow of work through the enterprise, relationships between Business, Development and Operations and demonstrate the benefits of moving from cyclical to continuous delivery of value.
Welcome to Chocolate and LEGO Enterprise. All of you are new hires now and will be introduced to your new roles. We have 3 development teams in organization, each of them is using Scrum framework. Who is familiar with Scrum? We have one operations team, one security team and one business team. Our organization makes money by building LEGO Animals and selling them to the market.
1:46 – 1:47 Please open the bags on your tables, take out the roles package and pick a role for yourself. Take 2 min to read about your role
How do we know what and how much to build? Business team will determine the market demand – what animals are popular and how much they are willing to pay for those animals.
Market demand is regulated at the Animal Stock Exchange. This is where the animal prices are established and the orders are placed in Sprint 1. Orders can be placed in the increments of the batch sizes. Based on a quality and a quantity of products delivered in Sprint 1, the market demand will be adjusted for Sprint 2. In the first sprint, each order from the market (for example, “5 dogs” or “10 giraffes”) is considered as one story. To generate business value, the story must be delivered to the market in its entirety. Scrum Teams need to be careful to not over produce as business will not purchase the items above desired quantity indicated. Team will work with the Product Owner to determine which story has highest value for the market and prioritize what to work on for the next sprint.
------------------- In Sprint 2, 3 the animal prices will fluctuate during the sprints, hence providing an incentive to the teams to deliver products to the market faster. Any quantity delivered by a team over the desired quantity number is counted as negative ( company will not be able to sell it = waste) Team will work with the Product Owner to determine which story has highest value for the market and prioritize what to work on for the next sprint.
Finish at 1:52 Business will have a set of play money to be used for “payment” for accepted animals.
1:52 – 2:07
In the first sprint, each order from the market (for example, “5 dogs” or “10 giraffes”) is considered as one story. To generate business value, the story must be delivered to the market in its entirety. Team will work with the Product Owner to determine which story has highest value for the market and prioritize what to work on for the next sprint.
**Operations team** is a functional silo team in Sprint 1. Ask your **Security Engineer** to pick three random numbers between 1 – 20. These will be security bugs. When Scrum team is ready to deliver products into production, **Security Engineer** will “run the security scan”. If a product has a label with one of the three selected numbers, it will need to be returned back to a development team. **Release Engineer** will need to package features delivered into a deployment package and deploy them into production (place the small packages into a ZipLock back, write team name with a sprint number and deliver it to market analyst's table)
At the end of the Sprint 1, participants will discover that deployments are not allowed until the next deployment window. Everything that scrum teams built is now stuck in the queue in Operations department and no value gets delivered to the market.
In the Sprint 1 retrospective, **Scrum teams** will look at optimizations they can apply to their current process and the constraints that impede their productivity.
1. Invite **Sara Security** into the Scrum team. 2. Solicit an early feedback from the **Market**
2:07 – 2:12
How was your experience? Scrum Teams, how many product backlog items were you able to get done?
This is great! Why are the customers unhappy? Did you get any product delivered to you?
It took a long time to start – getting the stories, building environments… Then security testing at the end caused delays and rework. Then what happen – team built potentially shippable products, but wasn’t able to deploy due to the code freeze. They are getting no feedback. Why do you think there IS a code freeze? This is a function of deployment process being manual, brittle and potentially causing service disruption. With the Water-Scrum-fall it gets expensive to deliver changes and making them live in production. One of the advantages of embracing DevOps , is that you are able to re-define that!
There is more than one entry point into DevOps world and all of them will require changes in technology, people, process and culture.
Start with visualizing your flow. Get a better understanding of how a value delivering planned work moves through your organization. Learn to optimize that. Understand where unplanned work causes the most disruption and how you can minimize it’s impact. Think about all the places were you may have non-value added work and your biggest constraint.
End at 2:17
Bring in Operations into your team!They know how your system runs in production, what are some of the challenges it is having. Leverage that feedback to make it more resilient #1 How many of you are familiar with Conway Law – Architecture of the system reflect the structure of organization. By making the structure of org and team more open and collaborative we are positioning ourselves better for improving the architecture of our systems
#2 Enable everyone in the team to be a generalist – cross-train, build T-shaped skills
2:17 – 2:18 – explain X-training 2:18 - 2:33 – run the sprint
Was this better? 2 min
Rather than coming up with a bunch of features and planning a multi-month release, come up with new ideas continually and try them out individually on users. With enough thought, even big features or large-scale changes can be implemented as a series of smaller steps to get faster feedback, with the ability to pivot and course correct if needed. With a cross-functional team working to deliver these small increments in hours or days, you can be more innovative than your competition and maximize your return on investment
With Continuous Delivery your software is always release-ready, yet the timing of when to push it into production is a business decision, and so the final deployment is a manual step.
Ultimate feedback that we are looking for is the feedback from our end-users. Manual and infrequent deployments significantly reduce our opportunity to receive this feedback. Important step in your devops initiative will be environments standardization and automated provisioning. The idea is making things self-service so that everyone in the Scrum Team can build an environment, provision and deploy.
Start at 2:38 – 2:52
Shared on-call – everyone gets upstream feedback on architecutural or coding decision they’ve made
Blameless postmortem and share learnings (Morgue) Inject failure to test resilience “Human error is the consequence of the design of the tools we gave them” Dr, Sydney Dekker
AATC - Gamifying DevOps with Lego and Chocolate Game
Gamifying DevOps with
LEGO and Chocolate
Turn and Talk
DevOps with LEGO and Chocolate Game
CSM, CSPO, CSP,
NYC Scrum User Group
Agile Coach Camp US
My Favorite One
“A mix of patterns intended to
improve collaboration between
development and operations. DevOps
addresses shared goals and
incentives as well as shared
processes and tools.”
~ Michael Hüttermann
2017 - No
GE Capital, Disney,
US Department of
Pursue Further Feedback Loop
1. Telemetry – implement, collect, make
visible to all.
2. On-call rotation – involve developers
3. Have developers initially self-manage
their production servers.