While the DevOps methodology supports faster and more automated software production, it does not necessarily guarantee code quality unless specifically required for development teams. At the same time, producing reliable and secure software has become increasingly difficult for organizations. Applications are not only growing in size, they are also becoming more complex and intertwined across platforms, systems and devices. This complicated environment is forcing business executives, IT leaders and software developers to think and work differently to produce quality software at a faster pace – with the goal of increasing their competitive advantage in the digital economy. Join Maria Loughlin, VP of Engineering at CA Veracode as she discusses why software quality can no longer be a checkbox item, but rather a framework that spans throughout an organization. She will share examples of how enterprises have adopted the mentality to own the success of their code — and develop best practices within the DevOps framework to secure their application layer and produce high-quality software.

1. © 2018 VERACODE INC.1
Software Quality as a Competitive
Differentiator
Maria Loughlin, VP Engineering
@marialoughlin

2. © 2018 VERACODE INC.2
On This Webinar
1.
Quality and
Business Success
2.
DevOps:
Promises & Gaps
3.
Building a
Quality System

3. © 2018 VERACODE INC.3
Poll: Who’s Attending This Webinar?
• Quality Professional (Tester, QA Eng, SWET, Architect)
• Developer / DevOps / Release Engineer
• Operations Engineer
• Product Manager / UX Designer
• Engineering Manager
• Business Leader
• Other

4. © 2018 VERACODE INC.4
Revenue and Net Promoter Score
SOURCE: The Net Promoter System. Bain &
Company, Inc.
“On average, an industry’s Net
Promoter leader outgrew its
competitors by a factor greater
than two times.”

5. © 2018 VERACODE INC.5
High Quality, High Trust
SOURCE: https://cxi.today/2018-cx-trends/analytics-infographic-5-trends-shaping-cx-in-2018/

6. © 2018 VERACODE INC.6
Quality
Productivity
Predictability
Employee
happiness
Innovation

7. © 2018 VERACODE INC.7
Maria Loughlin
VP Engineering, Veracode
• Two decades of software engineering
leadership across a range of industries and
company profiles.
• Manages development and operations for the
FedRAMP instance of Veracode’s Application
Security products.
• Deep expertise in Secure SDLC and DevSecOps.

8. © 2018 VERACODE INC.8
Poll Results: Who’s Attending This Webinar?
• Quality Professional (Tester, QA Eng, SWET, Architect)
• Developer / DevOps / Release Engineer
• Operations Engineer
• Product Manager / UX Designer
• Engineering Manager
• Business Leader
• Other

9. © 2018 VERACODE INC.9 © 2018 VERACODE INC.
Part II: Quality
and DevOps

10. © 2018 VERACODE INC.10
DevOps Promise: Continuous Testing
Image: https://www.parasoft.com/solutions/continuous-testing

11. © 2018 VERACODE INC.11
DevOps Promise: Comprehensive Testing
Unit
Component
Integration
E2E
UI
Shift
Right
Shift
Left
Automation throughout the
stack
Automation throughout the
lifecycle

12. © 2018 VERACODE INC.12
Unit
ComponentIntegration
End to End
DevOps Reality: Inconsistent Testing
• Quality investment often
driven by delivery team,
independent of overall
strategy
• All sorts of tests with
almost equal priorities

13. © 2018 VERACODE INC.13
State of Software Quality
SOURCE: GitLab 2018 Global Developer Report, https://about.gitlab.com/developer-survey/2018/
Quality is often compromised for schedule

14. © 2018 VERACODE INC.14
SOURCE: GitLab 2018 Global Developer Report, https://about.gitlab.com/developer-survey/2018/
Testing causes
delays

15. © 2018 VERACODE INC.15
The Change Failure
Rate for high
performers is
5 times lower
than for low
performers
SOURCE: Puppet 2017 State of DevOps Report,
https://puppet.com/resources/whitepaper/2017-state-of-devops-report

16. © 2018 VERACODE INC.16
The Mean Time to
Repair (MTTR) for
high performers is
96 times
faster
than for low
performers
SOURCE: Puppet 2017 State of DevOps Report,
https://puppet.com/resources/whitepaper/2017-state-of-devops-report

17. © 2018 VERACODE INC.17
85% of
applications are
vulnerable35.9%
33.5%
85.1%
84.9%
First Scan
Latest Scan
High or
Very High
Severity
Any
Severity
Percent of Applications with Findings
Source: Veracode SOSS Volume 9
SOURCE: Veracode SOSS Volume 9, https://www.veracode.com/state-of-software-security-report
State of Software Security

18. © 2018 VERACODE INC.18
The percent of
applications
passing OWASP
Top 10 Policy
on first scan is
consistent over
time
23% 77%
13% 87%
32.3% 67.7%
38.6% Passed 61.4% Did Not Pass
30.2% 69.8%
2010
2013
2015
2016
2017
Percentage of Applications Passing OWASP on First Scan
Source: Veracode SOSS Volume 9
SOURCE: Veracode SOSS Volume 9

19. © 2018 VERACODE INC.19
What’s The Challenge?
1. Reinvested quality process
2. Unfocused quality efforts
3. Relentless pressure to deliver
features
4. Complexity of software –
more than ever before

20. © 2018 VERACODE INC.20
Challenge 1: Who’s responsible for Quality?
Dev Product
Tester
Designer Mgr
Dev Product
Tester
Designer Mgr
Monitoring
Analytics
SupportInfra-
structure
Dev Product
Tester
Designer Mgr
Waterfall Agile DevOps

21. © 2018 VERACODE INC.21
Challenge 2: Unfocused Quality Efforts
Quality can be subjective
and contextual.
“Quality is value to some
person.”
-Jerry Weinberg, Software quality expert

22. © 2018 VERACODE INC.22
Challenge 3: Relentless Pressure To Deliver Features

23. © 2018 VERACODE INC.23
Challenge 4: Software Is Increasingly Complex
Today’s software is
• Distributed
• Embedded in complex systems
• Autonomously learning and
evolving
• Deployed to untrusted
environments

24. © 2018 VERACODE INC.24

25. © 2018 VERACODE INC.25 © 2018 VERACODE INC.
Part 3:
Creating a
Quality System

26. © 2018 VERACODE INC.26
Creating a Quality System
Specify
CI/CD across organization with recommended tools
Drive
Test
Strategy
Test
Process
Test
Automation

27. © 2018 VERACODE INC.27
Strategy: Quality Goals
`
Strategy Process Automation
Customer-visible
quality
•Functional
•Great user
experience
•Consistent, reliable
•High performing
Technical quality
•Maintainable
•Scalable
•Secure

28. © 2018 VERACODE INC.28
Strategy: When and Where?
• Pre-production
– Test functionality, stability, security, customer satisfaction, compliance
• Production
– Test functionality, performance, resilience, stability
– Experiment to test new ideas
`
Strategy Process Automation

29. © 2018 VERACODE INC.29
Strategy: Who?
Unit
Component
Integration
E2E
UI Delivery team owns the tests
• Maturity of organization impacts
exact staffing
– Lower layers always developer
• Quality mindset always present
– QA architect and ever-present
voice of customer
`
Strategy Process Automation

30. © 2018 VERACODE INC.30
Process: Investment by Phase
• Inspect and adapt process
• Continuous production feedback
• Customer data
• Upper layers of test pyramid
• Quality dashboards
• Security testing
WALK
CRAWL
RUN
• Deployment automation, CI/CD
• Unit tests and mocking code
• Test infrastructure
`
Strategy Process Automation

31. © 2018 VERACODE INC.31
Process: Metrics
Internal View
• Test coverage
• Reopened issues
Customer View
• Escaped defects
• MTTR
• Service interruption
`
Strategy Process Automation

32. © 2018 VERACODE INC.32
Process: Definition of Done
Test investment
`
Strategy Process Automation

33. © 2018 VERACODE INC.33
Automation: The Test Stack
Unit
Component
Integration
E2E
UI
Protractor, Selenium, Cyprus, Jest
Performance test
Microservice integration
API: Rest Assured, Postman
Include Performance test
Junit, P:yUnit
UI Unit tests: Karma, Jasmine
`
Strategy Process Automation

34. © 2018 VERACODE INC.34
Automation: Infrastructure
• CI/CD pipeline
• Test environments
– LocalStack / Full function Stack
• Logging, monitoring, debugging, Sumologic, New Relic
• Code Coverage, SonarCube
• Story/Bug management
• Frameworks
– Robot / TestNG
• Mock framework, Mockito
`
Strategy Process Automation

35. © 2018 VERACODE INC.35
Automation: Test Data
• Supporting artifacts
– Mock code
– Test data
`
Strategy Process Automation

36. © 2018 VERACODE INC.36 © 2018 VERACODE INC.
Driving Quality
Across The
Organization

37. © 2018 VERACODE INC.37
ManagementSecurity DevOps Teams
Seed the Program
Get Commitment

38. © 2018 VERACODE INC.38
The Quality System: Culture
• Continuous learning, accountability
• QA chapter
• Customer focus

39. © 2018 VERACODE INC.39
Security Quality
• Integrate with SDLC
SCA
Define policy

40. © 2018 VERACODE INC.40
Creating a Quality System
Specify
CI/CD across organization with recommended tools
Drive
Test
Strategy
Test
Process
Test
Automation

41. © 2018 VERACODE INC.41

42. © 2018 VERACODE INC.42 © 2018 VERACODE INC.
Q & A
www.veracode.com