The document discusses building a quality system for software development. It begins by noting that while DevOps promises comprehensive testing throughout development and operations, in reality testing is often inconsistent due to a lack of focus and relentless pressure to deliver. It then outlines challenges in assigning responsibility for quality, having unfocused quality efforts, and dealing with increasing software complexity. The document proposes creating a quality system with three parts: specifying a quality strategy, defining a quality process, and automating testing. It provides examples of how to approach quality goals, roles, metrics, and investing in different phases. It emphasizes automating infrastructure, testing at all levels, and driving quality across the organization through communities of practice and continuous learning. Security quality is discussed as having its

1. © 2018 VERACODE INC.1
Software Quality as a Competitive
Differentiator
Maria Loughlin, VP Engineering
@marialoughlin

2. © 2018 VERACODE INC.2
On This Webinar
1.
Quality and
Business Success
2.
DevOps
Promises & Gaps
3.
Building a
Quality System

3. © 2018 VERACODE INC.3
Poll: Who’s Attending This Webinar?
• Quality Professional (Tester, QA Eng, SWET, Architect)
• Developer / DevOps / Operations Engineer
• Product Manager / UX Designer
• Engineering Manager / Executive
• Other

4. © 2018 VERACODE INC.4
Revenue and Net Promoter Score
SOURCE: The Net Promoter System. Bain &
Company, Inc.
“On average, an industry’s Net
Promoter leader outgrew its
competitors by a factor greater
than two times.”

5. © 2018 VERACODE INC.5
High Quality, High Trust
SOURCE: https://cxi.today/2018-cx-trends/analytics-infographic-5-trends-
shaping-cx-in-2018/

6. © 2018 VERACODE INC.6
Quality
Productivity
Predictability
Employee
Happiness
Innovation

7. © 2018 VERACODE INC.7
Maria Loughlin
VP Engineering, Veracode
• Two decades of software engineering
leadership
• Waterfall to Agile to DevOps
• Monolith to MicroServices
• Manage development and operations for the
FedRAMP instance of Veracode’s Application
Security products.
• Deep expertise in Secure SDLC and DevSecOps.

8. © 2018 VERACODE INC.8
Poll Results: Who’s Attending This Webinar?

9. © 2018 VERACODE INC.9 © 2018 VERACODE INC.
Part II: Quality
and DevOps

10. © 2018 VERACODE INC.10
DevOps Promise: Continuous Testing
Image: https://www.parasoft.com/solutions/continuous-testing

11. © 2018 VERACODE INC.11
DevOps Promise: Comprehensive Testing
Unit
Component
Integration
E2E
UI
Shift
Right
Shift
Left
Automation throughout the
stack
Automation throughout the
lifecycle

12. © 2018 VERACODE INC.12
Unit
Compo
nent
Integrat
ion
End to
End
DevOps Reality: Inconsistent Testing
• Quality investment often driven
by delivery team, independent
of overall strategy
• All sorts of tests with almost
equal priorities

13. © 2018 VERACODE INC.13
State of Software Quality
SOURCE: GitLab 2018 Global Developer Report, https://about.gitlab.com/developer-survey/2018/ /
42% sacrifice quality to
meet a deadline

14. © 2018 VERACODE INC.14
SOURCE: GitLab 2018 Global Developer Report, https://about.gitlab.com/developer-survey/2018/ /
Testing
causes
delays

15. © 2018 VERACODE INC.15
The Change Failure
Rate for high
performers is
5 times lower
than for low
performers
SOURCE: Puppet 2017 State of DevOps Report,
https://puppet.com/resources/whitepaper/2017-state-of-devops-report

16. © 2018 VERACODE INC.16
The Mean Time to
Repair (MTTR) for
high performers is
96 times faster
than for low
performers
SOURCE: Puppet 2017 State of DevOps Report,
https://puppet.com/resources/whitepaper/2017-state-of-devops-report

17. © 2018 VERACODE INC.17
85% of
applications are
vulnerable35.9%
33.5%
85.1%
84.9%
First Scan
Latest Scan
High or
Very High
Severity
Any
Severity
Percent of Applications with Findings
Source: Veracode SOSS Volume 9
SOURCE: Veracode SOSS Volume 9, https://www.veracode.com/state-of-software-security-report
State of Software Security

18. © 2018 VERACODE INC.18
The percent of
applications
passing OWASP
Top 10 Policy
on first scan is
consistent over
time
23% 77%
13% 87%
32.3% 67.7%
38.6% Passed 61.4% Did Not Pass
30.2% 69.8%
2010
2013
2015
2016
2017
Percentage of Applications Passing OWASP on First Scan
Source: Veracode SOSS Volume 9
SOURCE: Veracode SOSS Volume 9

19. © 2018 VERACODE INC.19
What’s The Challenge?
1. Reinvested quality process
2. Unfocused quality efforts
3. Relentless pressure to deliver
4. Complexity of software –
more than ever before

20. © 2018 VERACODE INC.20
Challenge 1: Who’s Responsible for Quality?
Dev Product
Tester
Designer Mgr
Dev Product
Tester
Designer Mgr
Monitoring
Analytics
SupportInfra-
structure
Dev Product
Tester
Designer Mgr
Waterfall Agile DevOps

21. © 2018 VERACODE INC.21
Challenge 2: Unfocused Quality Efforts
Quality can be subjective
and contextual.

22. © 2018 VERACODE INC.22
Challenge 3: Relentless Pressure To Deliver

23. © 2018 VERACODE INC.23
Challenge 4: Software Is Increasingly Complex
Today’s software is
• Distributed
• Embedded in complex systems
• Autonomously learning and
evolving
• Deployed to untrusted
environments

24. © 2018 VERACODE INC.24

25. © 2018 VERACODE INC.25 © 2018 VERACODE INC.
Part 3:
Creating a
Quality System

26. © 2018 VERACODE INC.26
Creating a Quality System
Specify
CI/CD across organization with recommended tools
Drive
Quality
Strategy
Quality
Process
Test
Automation

27. © 2018 VERACODE INC.27
Strategy: Quality Goals
`
Strategy Process Automation
Is Your Customer
Getting the Value
They Expect?
• Functional
• Great user experience
• Consistent, reliable
• High performing
Will Your Team
Remain Productive?
• Maintainable
• Scalable
• Secure

28. © 2018 VERACODE INC.28
Strategy: When and Where?
• Pre-production
– Test functionality, stability, security, customer satisfaction, compliance
• Production
– Test functionality, performance, resilience, stability
– Experiment to test new ideas
`
Strategy Process Automation

29. © 2018 VERACODE INC.29
Strategy: Who?
Unit
Component
Integration
E2E
UI Delivery team owns the tests
• Maturity of organization impacts
exact staffing
– Lower layers always developer
• Quality mindset always present
– QA architect and ever-present
voice of customer
`
Strategy Process Automation

30. © 2018 VERACODE INC.30
Process: Investment by Phase
• Inspect and adapt process
• Continuous production feedback
• Customer data
• Upper layers of test pyramid
• Quality dashboards
• Security testing
WALK
CRAWL
RUN
• Deployment automation, CI/CD
• Unit tests and mocking code
• Test infrastructure
`
Strategy Process Automation

31. © 2018 VERACODE INC.31
Process: Definition of Done
`
Strategy Process Automation
Test investment

32. © 2018 VERACODE INC.32
Process: Metrics
Internal View
• Test coverage
• Reopened issues
Customer View
• Escaped defects
• MTTR
• Service interruption
`
Strategy Process Automation

33. © 2018 VERACODE INC.33
Automate Everything
1. DevOps Infrastructure
`
Strategy Process Automation
Unit
Component
Integration
E2E
UI
2. Tests

34. © 2018 VERACODE INC.34
SOURCE: Atlassian Marketplace for DevOps Apps, https://marketplace.atlassian.com/categories/devops

35. © 2018 VERACODE INC.35
Automation: Infrastructure
`
Strategy Process Automation
Infrastructure Consideration
CI / CD Pipeline Reliable, repeatable
Example: Jenkins
Test Environments Easy to create and scale. Monitor for cost
Test Frameworks Can be integrated with build pipeline, e.g. GitLab
Or separate tool, e.g. Robot / TestNG
Quality Metrics Transparent, consistent
Example: SonarCube, Bug tracker with analytics

36. © 2018 VERACODE INC.36
Test Layer Consideration Example Tools
UI Match your UI language
E2E Include performance testing
Integration Focus on interactions
between microservice and
external services
Component Include performance testing
Unit Match your language
primitive
Automation: The Test Stack
Test Layer Consideration Example Tools
UI Match your UI language Protractor for Single Page Apps,
or Selenium, Cyprus, Jest
E2E Include performance testing Selenium, Protractor, Cyprus,
Jest
Integration Focus on interactions
between microservice and
external services
API: Rest Assured, Postman
Component Include performance testing Mockito for mocking framework
Unit Match your language
primitive
Junit, PyUnit
UI Unit tests: Karma, Jasmine
`
Strategy Process Automation

37. © 2018 VERACODE INC.37 © 2018 VERACODE INC.
Driving Quality
Across The
Organization

38. © 2018 VERACODE INC.38
Creating a Quality Culture
Break the
Silos
Support the
Team
Learn
Continuously

39. © 2018 VERACODE INC.39
Guilds: Experts Support Each Other
• Identify your leaders and
practice experts
• Hold regular ‘birds of a
feather’ meetings
• Share learnings, trends and
best practices constantly
• Encourage & reward
participation

40. © 2018 VERACODE INC.40
Continuous Learning

41. © 2018 VERACODE INC.41
What About Security Quality?
Specify
Security
Strategy
Security
Process
Security
Automation

42. © 2018 VERACODE INC.42
Strategy: Security Policy
Policy defines and supports your tolerance for risk.
• Requirements for remediation of vulnerable code
and components
• Standards for software licence usage
• Recommended libraries, frameworks, embedded
components

43. © 2018 VERACODE INC.43
Process: Security Maturity Model (SAMPLE)
Activity Base Beginner Intermediate Advanced Expert
Training
Secure Design
Security Code
Review
Security
Testing
Third Party
Activity Base Beginner Intermediate Advanced Expert
Training
Secure Design Security is not a
design
consideration
Security reqts are
generally defined
after
development has
started or
completed
Threat modeling
before major
components or
features
Security reqts are
defined before
major componen
ts or features
Threat modeling is
incorporated into
the story process
Security reqts are
defined as story
Acceptance
Criteria on
relevant stories
Security
Acceptance
Criteria defined
for all relevant
stories
Security Code
Review
Security
Testing
Third Party

44. © 2018 VERACODE INC.44
Security Automation
The best app security is
invisible to developers

45. © 2018 VERACODE INC.45
Recap: On This Webinar
1.
Quality and
Business
Success
2.
DevOps:
Promises &
Gaps
3.
Building a
Quality System
“Quality is never an accident; it is always the result of high
intention, sincere effort, intelligent direction and skillful execution”
– William A. Foster

46. © 2018 VERACODE INC.46

47. © 2018 VERACODE INC.47 © 2018 VERACODE INC.
Q & A
www.veracode.com

48. © 2018 VERACODE INC.48