-What are the blockers hurting organizations from modernizing their devOPS and testing approach to become agile -How to embed quality across the SDLC, step by step, pragmatically -Breaking down the dev-OPS wall by introducing production insight as current sprint activity

1. 5 Stages of Digital Quality Maturity
Amir Rozenberg, Director of Product
Chris Bartosic, IT Solutions Leader, Metlife

2. 5/9/2018 2© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Agenda
The challenges of modern quality practices
Implementing quality in DevOPS (CI/CD)
The MetLife journey
Q&A

3. 50% of CEOs expect their industries to be substantially
or unrecognizably transformed by digital.
Source: Gartner, 2016

4. Increased Complexity
IoT
Multi
Screen
Platforms
OTT
Streamline
Location
Injection
Acceleration Medical
Enrichment
Sensors
New models
Video AR
Chatbots
& Audio
Touch ID
Face ID
Iris Scan AUTH
Data EntryImage
Injection

5. The realities of the Dev team lead
Scope
CoverageTime

6. Current State of Digital
Quality
Outdated Quality Practices
•Repetitive manual testing, post sprint
•Outdated local lab
•Outdated bank of scripts and framework
•Partial reporting & lack of direction
Team Impact
•Risk-based approach, poor app behavior
•Expensive, latent feedback
•"Over the wall" attitude
•Frustration and fatigue
Business Impact
•User frustration, adoption, usage and churn
•Impact on top-level metrics: share value,
competitive leadership, brand etc.

7. Quality @ devOPS Principles
• Skills elevation
• Dev-QA-OPS: one team
• Dev agile process
• Everyone should work at the same
rhythm
• Real time decisions
• Real time bug fixing
• Developers test
• Automation AI:
• Green is Green
• 3 strikes out
• Smart retry
• Always on, scalable & stable lab
• Enabling framework
• Big data reporting
• Transparency & alignment

8. Confidential – for internal use only
8
Testing Approach @ MetLife – Today
TestDesign
Test
Strategy
Test Management Repository
Financials / Coverage / Quality / Estimates / Inputs / SLA’s & SLO’s
Require-
ments
Use
Cases
Inputs & Outputs
Capability
Dynamic
Inputs
Management
More Automated
TestOptimization
ata
Test
Data
Automation
Engines
Automation Lab
Mobile
Middleware
Accessibility
Cross Browser
UI – open source
Mainframe
IVR
Virtualization
Test Data Mgmt
Crowd
Scriptless
digitalfocus
Analytics
Defect
Management

9. Stage 1: Plan Quality @ DevOps as an Agile Process
5/9/2018 © 2015, Perfecto Mobile Ltd. All Rights Reserved.
Coverage
DailySprint
Smoke
Full
Coverage
before
Robustness
Robustness
Before
Coverage
Most of the market
• Sprint level execution
• Half-way to coverage
Execution Frequency

10. Stage 2: Automate, Reuse, Scale
Local Devices/Lab
APM/ Production
Monitoring
Use your Automation Framework
Execute tests every build
Connect the cloud to your IDE
Android Studio • IntelliJ • Eclipse • Visual Studio

11. Stage 2: Automate, Reuse, Scale
• Functional Testing
• Increase automation: OSS+, Framework, Reuse (one lab)
• Increase coverage: Wind Tunnel, Audio/Video, Localization, IVR,
sensors
• Parallel execution
• Non-Functional Testing
• Single user performance
• Accessibility
• Security
=
Multi-screen
End user conditions

12. Stage 3: Shift left & Prioritize
Stability, visibility & trust => Value
CoE
Dev

13. Stage 4: Continuous Quality Visibility

14. Stage 5: Implement across the entire SDLC
Use your Automation Framework
Execute tests every build
Connect the cloud to your IDE
Android Studio • IntelliJ • Eclipse • Visual Studio
Quality Visibility
One report, all platforms, see the trend
‘Test in Production’
Know it • Triage it • Fix it
Remote Debug to the Cloud
ADB • Chrome inspect

15. Confidential – for internal use only
People
Process
Technology
15
Transitioning from Test to “Q” Assurance
SaaS / iPaaS
Open source
Integrated into IT
Security
Less traditional vendors
Limitless devices
Siloed
Proprietary
Tightly coupled
On-prem Some integration
Less traditional vendors
Some cloud
Limited devices
Managed Service
Program Management
Engineering
Agile
Quality owned across SDLC
Design & OptimizationWaterfall
Hard gates
Static data review
Siloed
Onshore
Some iterative
Defect measurements
Quality reviews
Some offshore
Blurred ownership
Application SME
More technical
Some Automation
Some Analytical
Tester
Siloed
Non-technical
Manual
Traditional
Technicians
Information analysts
Program management
Architects
Developers
Business
Sr. Leadership

16. Confidential – for internal use only
1616
Test Engineering by Function
• Test Design and Technical Strategy, escalation support
• Primary bridge between QA and Architect group
StrategySupport/TechnicalLead
 Emphasis on shared services, workflow management, self-service capabilities for TDM
 Long drawn plan for increasing TechQA role in test data management
 Extension of tools strategy for digital platforms and tools
 Focus on end to end testing for digital programs with shift left adoption
 Intelligence driven testing inclusion
 Emerging data and environment practices leverage for DevOps testing
 Emphasis on continuous testing and reduced dependencies on UI testing and automation
 Focus on extending end to end testing capabilities to support Agile /DevOps
 Shift left & right adoption
 Big Data testing & automation
 Expedited initiatives on Scriptless automation and Services test automation, QA ecosystem
 Focus on opportunity areas including legacy platform, IVR and AI automation, test coverage

17. Confidential – for internal use only
17
Mobile Device Cloud
• Data centers located globally
• Private & public cloud available to MetLife
• Private: 40 Devices (phones, tablets, VDI’s)
• 26 Phone & tablets - Primarily US-supported (iPhone,
HTC, LG)
• 14 Virtual Desktops - All major O/S and Browser
Combinations
• Public: +200 Devices (monthly limit usage)
Available Devices
Count of
devices
Apple 12
Google 4
HTC 1
LG 1
Samsung 7
Sony 1
Grand Total 26

18. Confidential – for internal use only
18
To edit go to: Insert > Header and
Footer
Transformation In-flight
Increase in Hybrid Lab utilization, including desktop
• Single presentment platform for digital QA
• Cross-browser, mobile, accessibility, performance
• Dual execution labs supporting 24X7 operations
Improved leverage of automation and non-UI testing approaches, reduced cycle times
• 300 % increase in execution over 1 quarter
• Introduction of in-house and market tooling to address capability gaps
• “Automation First” embedded as part of Intake, +30% increase defects identified projected
Comprehensive functional and non-functional validations with “shift-left” approach
• End to end view of coverage at the onset of work
• Synthesized strategies between engineering and delivery “producer & consumer”
Early defect identification and improved customer experience
• Defect analytics pushing focus on systemic problem areas
• Blurring lines between hard “hand-off’s” and tighter requirements/testing integration
Continuous testing integrated into project lifecycle, reduced cycle time
• Integration into broader DevOps lifecycle and ecosystem
• Accelerated introduction of automated entrance/exit testing

19. PREP PROVE SCALE MAINTAIN
Phase 1
Plan Development
and Delivery
Phase 2
Build Foundation
(Framework)
Phase 3
P1 Smoke Test
Creation
Phase 4
P1 Smoke w/
Reporting
Per-Commit CI
(Few Platforms)
Running Nightly
(All Platforms)
Phase 5
P2 Suite w/
Reporting
Running in
scheduled CI
(Few Platforms)
Running Nightly
(All Platforms)
Phase 6
P3 Regression
w/Reporting
Running Nightly
(All Platforms)
Running Nightly
(All Conditions)
Phase 7
Script
Maintenance
Process
Established
Phase 8
Execution
Optimization
Increased
Automation
Maximize
Automation
Maximize
Coverage
Fast
Feedback
Increased
Coverage
Path to Success Timeline
1 Week 3 Weeks1 3 Weeks1 1 Week2 8-10 Weeks1,2,3 12 Weeks1 4 Weeks 4 Weeks
Embedded Across Dev-QA-Ops
New
1 Month 3-6 Months* 2 Months
Existing
2-3 Weeks
New
1 Month
Existing
1-3 Weeks

20. Pragmatic plan to achieve Digital Quality
Automate everything
Green is Green
Prioritize Executions
Report Early
Streamline processes

21. @Amir_Rozenberg@bartosik_chris