1.
Development and QA dilemmas
in DevOps
Matteo Emili
Microsoft MVP

2.
Who am I?
 Visual Studio and DevelopmentTechnologies MVP (ALM)
 Professional Scrum Master I
 Systems EngineeringAdvisor @ One Identity
 Taunton Developers Meetup | London Microsoft DevOps Meetup
 GetLatestVersion.it | DomusDotNet
matteo.emili@live.com || @MattVSTS || mattvsts.blogspot.com

3.
CDCycle
Monitor
Operate
Workingsoftware
inproduction
Valuerealization

4.
Are we all ready for it?

5.
The typical
(legacy)
approach
 Develop everything in a separate environment
 Countless hours of tests
 Big-bang migration then new feature for all the users
 Big release date, people freaking out
 …lots of fingers crossed!

6.
Sometimes
things work
the other way
round…
 The first thing to do should be creating a MinimumViable Product,
including the infrastructure
 The Pipeline must work from the very first moment
 Dev andTest environments should be disposable, infrastructure
must be idempotent
 Testing can happen in Production

7.
It is a shocking
change for
some
 Excerpts from interviews with developers, testers, managers…
 “I was frightened by not feeling a safety net”
 “Does it actually work?”
 “The feeling is not good at first”
 “We don’t want to be accountable for all of this”
 “It can’t work!”
 “How do we know what to build?”
 “It is hard to get the grip with these tools”

8.
Prerequisite
The organisation must buy into DevOps!

9.
“Should I go all-in with ‘the
Cloud’ then?”

10.
Of course
not…
 It is all about culture and practices, there is little technology you
really need to know about
 This change puts the customer at the centre of the process, where
everybody across all the company work towards fulfilling the
customers’ requests and demands
 These practices can easily be reused with a totally on-premise
product, there is no need of targeting a technology you do not
need

11.
Use your
infrastructure!
 Leverage on Infrastructure As Code
(Azure RM,Amazon
CloudFormations, PowerShell DSC)
to remove friction when factoring
the infrastructure in
 Spend time on defining what you
need to have, and the provider
(Azure, AWS, local deployment
service) will deploy that for you
 Idempotent and resilient – if there
is something wrong, redeploy

12.
Package your
application for
distribution
 Leverage on packaging and deployment technologies to create a
flexible deployment and provisioning system
 These technologies are usually proven and tested
 MSDeploy
 Web Archive format
 NuGet
 They take centre stage with a Release Pipeline, together with
Infrastructure as Code

13.
Demo
Build once, deploy everywhere

14.
“How should I handle changes?”

15.
Shhh! Don’t
make noise!
 Each big change must be split in smaller changes
 They are often invisible to the user
 Silent deployments can happen across multiple sprints
 If something breaks, rollback
 Build incrementally, migrate data before full rollout
 Start using your MVP straight away!

16.
An example…
 You must bring a database schema change in production (a new
column)
 Cloud or mission critical web service
 Transparent for your users, regardless of public-facing or internal
change
 No downtime for migration – it must work!

17.
Hang on. How
about breaking
changes?
 It is essential to factor in some sort of mitigative solution that keeps
the service online while the change is deployed and tested
 A real world example?TFS 2015 introduced a massive change that
resulted in large databases being unable to be upgraded in less than
48 hours
 Microsoft released a tool that performed an online transformation
of the affected tables so that the upgrade proper could happen in a
reasonable time
 Pro: feasible upgrade
 Con: resource cost
 More on that at https://mattvsts.blogspot.com/2017/10/successfully-
handle-disruptive-changes.html

18.
Demo
Handle a database schema change in production

19.
“Testing in Production?”

20.
A/BTesting,
MVT
Testing…is it
actually about
testing?
 Little changes for QA in DevOps, except for automation and the
actual target of the system under test
 What is really new are practices ofTesting in Production -
extremely common especially withWeb Applications
 You deploy version B of your A application (or a permutation of
possibilities), direct a percentage of the traffic towards it and
compare the adoption results
 Implementing them is extremely easy, getting value out of them is
hard!

21.
Feature Flags
to the rescue
 Championed by Martin Fowler, Feature Flags are around since at
least 2009
 Pretty much all the major services nowadays use them
 Flickr,Amazon, Microsoft, Facebook, Google, Netflix …
 They are used to enable or disable certain areas of your
application to your users
 Also extremely valuable as a kill switch in case of issues

22.
“…wait. Does
it mean
Microservices
and all the new
stuff around?”
 No, it doesn’t!
 You can use Feature Flags with any architecture
 It is a practice, not necessarily a product
 It ties in with the previous scenario and makes testing easier

23.
How?
 Practically speaking, it is about a Configuration Database coupled
with Identifiers
 The Configuration Database can be a csv file, XML file, key-value
model, a database (!)
 Identifiers are areas you will easily split your applications into
 They often go well together with Feature/Epics or even User Stories
 Countless OSS libraries

24.
It doesn’t
come for free
 It is expensive (in terms of time, and sometimes refactoring) to
build it up from scratch in an existing application
 Not just enable/disable – logging, kill switch, selective adoption,
rollouts are all things to implement that take time
 OSS libraries help, as there is a choice for each platform, but there
is still an implementation cost to keep in mind

25.
Demo
Feature Flags with LaunchDarkly

26.
“How can I get better ROI
from my R&D investment?”

27.
A bit of Lean…
 "There is no such thing as a free lunch“ – technology alone is not
going to deliver a successful product
 My take is to adopt the Build-Measure-Learn loop
 Start with a MinimumViable Product and measure its usage with
appropriate metrics
 Build on top of it, discard what doesn’t work

28.
Getting
visibility on
what is
actually
happening
 Is my application performing as expected?
 What about availability?
 Are my users having a good experience?
 Above all, you need better, faster and more direct feedback

29.
Learn from
your users and
improve your
product
 In the Build-Measure-Learn loop the Measure stage is the most
important one
 Metrics will tell if you are going in the right direction, so it is critical
to ‘get it right’
 Extend your product to include what really matters
 Get insights on your users before they react
 Proactively understand potential problems
 You can analyse your data and understand if your application is
behaving as expected

30.
It is different
from logging
 We are not talking about plain application logs
 What user actions are meaningful to your business goals?
 Is the telemetry data answering this question?
 Is there any bottlenecks which is penalising the business?
 You need to have visibility on user events
 Actionable insights are mapped into behaviours (what content is
used, not how)
 E.g.: when a user enters in a certain page/area, log on/off,
Favourites, social stats, errors…

31.
You won’t do
that from
scratch
 It is too expensive to build something from scratch
 You need to use an analytics provider
 The development experience must be cross-platform, unobtrusive
at first and then customisable with an SDK
 Easily add custom telemetry wherever needed

32.
“Will I get a
meaningful
insight from
data alone?”
 Telemetry is not just reactive, it can also be proactive
 You will need to report on top of your data, with your metrics
 The alternative
 Easily add custom telemetry wherever needed

33.
Demo
Learn from your users

34.
So?
 These scenarios are very common when approaching DevOps
 Remember: DevOps is not the silver bullet, teams must buy-in into
the process
 Collaboration is key to success
 Use data to take decisions
 Don’t be afraid!

35.
Dank!