As organizations try to meet faster delivery schedules and improve collaboration between development and operations, DevOps has become a hot topic. So, where does testing fit into a DevOps strategy? By narrowly focusing on just Dev and Ops, the term DevOps itself ends up missing testing. Nikhil Kaul gives an overview of recent trends in DevOps, specifically changes that have taken place in the testing industry in the past year. Nikhil explores why testing is becoming more important than ever before. With Apple, Google, and Microsoft making it easier for Dev and Ops to build and deploy cross-platform apps, there is more to test—and less time in which to test. Testing can become a bottleneck to the entire DevOps process, especially when automated UI-based test suites start failing due to a layout or a backend change. Nikhil explores how testing can facilitate the transformation to DevOps, ways to build a scalable testing framework, and important reusability practices that can be applied—from development and test to operations—to reduce post-deployment debugging time.

1. DT4
Session
6/9/16 3:00 PM
Where Is Test in DevOps?
Presented by:
Nikhil Kaul
SmartBear Software
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.techwell.com/

2. Nikhil Kaul
SmartBear Software
Nikhil Kaul is product manager, testing products at SmartBear Software, the
leader in software quality tools for the connected world. Nikhil was previously at
Digité, an application lifecycle management solutions provider, where he gained
insight into the software development and testing market. Nikhil is very involved
in discussions taking place in development and testing communities. His most
recent thoughts can be found in Forbes and CIO.com. In his free time, Nikhil
enjoys running, hiking, and reading. Follow him on Twitter @kaulnikhil.

3. Where’s Test in DevOps?

4. 2
Quick Intro
Nikhil Kaul

5. 3
SmartBear’s technology stack

6. 4
On our plate today
1
Industry trends
2
What these trends mean for testers
3
Practices that can help facilitate the transformation

7. 5
The move away from Microsoft
Source: Gartner, Apple, Google, Microsoft, Nokia, Blackberry, a16z

8. 6
Another 1 billion to come online
Source: ITU, a16z

9. 7
The way apps are being built is changing
• Cross-platform dev becoming easier
— Microsoft buys Xamarin
— Windows 10
• Universal Windows Apps
• Write once & deploy it across platforms
— Apple making Swift open source

10. 8
Web continues to get more sophisticated
Approval for plugins
Closing APIs exposing
local machines
Technology giving an
access
Responsive UI
Web Components CSS implementation varies

11. 9
Web and mobile worlds colliding
‒ Cordova
‒ PhoneGap
‒ Open web tech
Mobile
‒ HTML
‒ JavaScript
‒ CSS
Web
React, React Native, Combined JSX, getUserMedia , Geolocation

12. 10
Browsers & app stores are becoming locked
Approval time for plugins and apps increasing
Closing APIs exposing
local machines
Technology giving an
access

13. 11
More to test in less time
Cross platform apps Changes in web &
mobile
High quality
Reusability critical

14. 12
Testing in a DevOps world involves deciding
Test ReportingTest ReductionTest DesignWhere to start

15. 13
Testing in a DevOps world involves deciding
Test ReportingTest ReductionTest DesignWhere to start

16. 14
Start small for quick wins
1
Latest
3
Beta
2
Previous
Browser App
80%

17. 15
Testing in a DevOps world involves deciding
Test ReportingTest ReductionTest DesignWhere to start

18. 16
Test pyramid strategy can come in handy
Service
Unit
UI

19. 17
Creating API tests helps prevent false defects
Web
Mobile
Tablet
1
2
3
4

20. 18
Reuse those API tests across SDLC
• API functional test
• API load test
• GUI functional test
• API security test
• Monitoring
Time reduced
by 80%

21. 19
The feedback from UI tests is more exhasutive
Browser: Chrome
End-to-end testing
x x x x
HTML5, Angular JS Network Service/API/Database

22. 20
Maintaining brittle tests can be a pain
Text
Duplication
Maintainability
Magic Strings

23. 21
Create one page object for each page
Home Page of Amazon
Result Search Page
Product Details Page
Confirmation Page

24. 22
Abstraction solves some of the problems
Page Object Class 1
Test
Page Object Class 2
Abstract control identification
Test steps
Abstract control identification
Abstract control actions
Abstract control actions

25. 23
The brittle code now looks like
Magic StringsLocating the control
Abstracting the set text action for the control
Using the abstracted set text action

26. 24
Testing in a DevOps world involves
Test ReportingTest ReductionTest DesignWhere to start

27. 25
Use PairWise testing to cut down tests
I(0,1)
B (0,1)
U(0,1)
App
B I U
1 1
1 0
0 1
0 0
B I U
1 0
0 0
1 1
0 1
B I U
1 1
1 0
0 1
0 0
B I U
1 1 1
1 0 0
0 1 0
0 0 1
1
2
3
Final Set

28. 26
Testing in a DevOps world involves
Test ReportingTest ReductionTest DesignWhere to start

29. 27
Numbers often don’t share the right picture
• 300 tests
• Each test has .5% failure rate
• Pass rate for each tests: 99.5%
• Pass rate for the suite: (99.5%)^300 = 22.23%

30. 28
Use statistics for measuring test stability
Sample Size (N)
• 50
P
• 29/50 = .58
1-p
• 21/50
Desired confidence (α)
• 95% confidence

31. 29
Takeaways
1
The role QA is becoming more important
2
Start small and build a modular test design
4
Reuse tests across software development lifecycle
3
API testing is key: Don’t forget about the GUI

32. Thank You!