The document discusses development and QA dilemmas in DevOps. It notes that traditionally development was done separately from testing and production environments, but with DevOps there is a focus on continuous delivery through automated pipelines. It acknowledges this is a big change that makes some developers and testers uncomfortable by removing safety nets. It emphasizes that DevOps is about culture and practices, not specific technologies, and that these practices can be used on-premises. It provides examples of how to handle database schema changes and feature rollouts transparently in production. It stresses measuring outcomes through metrics and analytics to learn from users and improve products iteratively.

1. Development and QA dilemmas
in DevOps
Matteo Emili
Microsoft MVP

2. Who am I?
 Visual Studio and DevelopmentTechnologies MVP (ALM)
 Professional Scrum Master I
 Systems EngineeringAdvisor @ One Identity
 Taunton Developers Meetup | London Microsoft DevOps Meetup
 GetLatestVersion.it | DomusDotNet
matteo.emili@live.com || @MattVSTS || mattvsts.blogspot.com

3. CDCycle
Monitor
Operate
Workingsoftware
inproduction
Valuerealization

4. Are we all ready for it?

5. The typical
(legacy)
approach
 Develop everything in a separate environment
 Countless hours of tests
 Big-bang migration then new feature for all the users
 Big release date, people freaking out
 …lots of fingers crossed!

6. Sometimes
things work
the other way
round…
 The first thing to do should be creating a MinimumViable Product,
including the infrastructure
 The Pipeline must work from the very first moment
 Dev andTest environments should be disposable, infrastructure
must be idempotent
 Testing can happen in Production

7. It is a shocking
change for
some
 Excerpts from interviews with developers, testers, managers…
 “I was frightened by not feeling a safety net”
 “Does it actually work?”
 “The feeling is not good at first”
 “We don’t want to be accountable for all of this”
 “It can’t work!”
 “How do we know what to build?”
 “It is hard to get the grip with these tools”

8. Prerequisite
The organisation must buy into DevOps!

9. “Should I go all-in with ‘the
Cloud’ then?”

10. Of course
not…
 It is all about culture and practices, there is little technology you
really need to know about
 This change puts the customer at the centre of the process, where
everybody across all the company work towards fulfilling the
customers’ requests and demands
 These practices can easily be reused with a totally on-premise
product, there is no need of targeting a technology you do not
need

11. Use your
infrastructure!
 Leverage on Infrastructure As Code
(Azure RM,Amazon
CloudFormations, PowerShell DSC)
to remove friction when factoring
the infrastructure in
 Spend time on defining what you
need to have, and the provider
(Azure, AWS, local deployment
service) will deploy that for you
 Idempotent and resilient – if there
is something wrong, redeploy

12. Package your
application for
distribution
 Leverage on packaging and deployment technologies to create a
flexible deployment and provisioning system
 These technologies are usually proven and tested
 MSDeploy
 Web Archive format
 NuGet
 They take centre stage with a Release Pipeline, together with
Infrastructure as Code

13. Demo
Build once, deploy everywhere

14. “How should I handle changes?”

15. Shhh! Don’t
make noise!
 Each big change must be split in smaller changes
 They are often invisible to the user
 Silent deployments can happen across multiple sprints
 If something breaks, rollback
 Build incrementally, migrate data before full rollout
 Start using your MVP straight away!

16. An example…
 You must bring a database schema change in production (a new
column)
 Cloud or mission critical web service
 Transparent for your users, regardless of public-facing or internal
change
 No downtime for migration – it must work!

17. Hang on. How
about breaking
changes?
 It is essential to factor in some sort of mitigative solution that keeps
the service online while the change is deployed and tested
 A real world example?TFS 2015 introduced a massive change that
resulted in large databases being unable to be upgraded in less than
48 hours
 Microsoft released a tool that performed an online transformation
of the affected tables so that the upgrade proper could happen in a
reasonable time
 Pro: feasible upgrade
 Con: resource cost
 More on that at https://mattvsts.blogspot.com/2017/10/successfully-
handle-disruptive-changes.html

18. Demo
Handle a database schema change in production

19. “Testing in Production?”

20. A/BTesting,
MVT
Testing…is it
actually about
testing?
 Little changes for QA in DevOps, except for automation and the
actual target of the system under test
 What is really new are practices ofTesting in Production -
extremely common especially withWeb Applications
 You deploy version B of your A application (or a permutation of
possibilities), direct a percentage of the traffic towards it and
compare the adoption results
 Implementing them is extremely easy, getting value out of them is
hard!

21. Feature Flags
to the rescue
 Championed by Martin Fowler, Feature Flags are around since at
least 2009
 Pretty much all the major services nowadays use them
 Flickr,Amazon, Microsoft, Facebook, Google, Netflix …
 They are used to enable or disable certain areas of your
application to your users
 Also extremely valuable as a kill switch in case of issues

22. “…wait. Does
it mean
Microservices
and all the new
stuff around?”
 No, it doesn’t!
 You can use Feature Flags with any architecture
 It is a practice, not necessarily a product
 It ties in with the previous scenario and makes testing easier

23. How?
 Practically speaking, it is about a Configuration Database coupled
with Identifiers
 The Configuration Database can be a csv file, XML file, key-value
model, a database (!)
 Identifiers are areas you will easily split your applications into
 They often go well together with Feature/Epics or even User Stories
 Countless OSS libraries

24. It doesn’t
come for free
 It is expensive (in terms of time, and sometimes refactoring) to
build it up from scratch in an existing application
 Not just enable/disable – logging, kill switch, selective adoption,
rollouts are all things to implement that take time
 OSS libraries help, as there is a choice for each platform, but there
is still an implementation cost to keep in mind

25. Demo
Feature Flags with LaunchDarkly

26. “How can I get better ROI
from my R&D investment?”

27. A bit of Lean…
 "There is no such thing as a free lunch“ – technology alone is not
going to deliver a successful product
 My take is to adopt the Build-Measure-Learn loop
 Start with a MinimumViable Product and measure its usage with
appropriate metrics
 Build on top of it, discard what doesn’t work

28. Getting
visibility on
what is
actually
happening
 Is my application performing as expected?
 What about availability?
 Are my users having a good experience?
 Above all, you need better, faster and more direct feedback

29. Learn from
your users and
improve your
product
 In the Build-Measure-Learn loop the Measure stage is the most
important one
 Metrics will tell if you are going in the right direction, so it is critical
to ‘get it right’
 Extend your product to include what really matters
 Get insights on your users before they react
 Proactively understand potential problems
 You can analyse your data and understand if your application is
behaving as expected

30. It is different
from logging
 We are not talking about plain application logs
 What user actions are meaningful to your business goals?
 Is the telemetry data answering this question?
 Is there any bottlenecks which is penalising the business?
 You need to have visibility on user events
 Actionable insights are mapped into behaviours (what content is
used, not how)
 E.g.: when a user enters in a certain page/area, log on/off,
Favourites, social stats, errors…

31. You won’t do
that from
scratch
 It is too expensive to build something from scratch
 You need to use an analytics provider
 The development experience must be cross-platform, unobtrusive
at first and then customisable with an SDK
 Easily add custom telemetry wherever needed

32. “Will I get a
meaningful
insight from
data alone?”
 Telemetry is not just reactive, it can also be proactive
 You will need to report on top of your data, with your metrics
 The alternative
 Easily add custom telemetry wherever needed

33. Demo
Learn from your users

34. So?
 These scenarios are very common when approaching DevOps
 Remember: DevOps is not the silver bullet, teams must buy-in into
the process
 Collaboration is key to success
 Use data to take decisions
 Don’t be afraid!

35. Dank!