The document outlines Virgin Media's transformation in mobile testing to enhance quality, efficiency, and scalability through automation and cloud-based solutions. With a significant increase in mobile usage, the company has adopted a continuous quality approach, transitioning from manual to automated testing processes. Key highlights include achieving TMMi level 5 accreditation and a commitment to security and quality as they adapt to evolving technological demands and customer needs.

1. Continuous Quality in the
Digital Age
Bob Randall
Virgin Media
Gidi Pridor
Perfecto

2. A FEW HOUSEKEEPING ITEMS
1. Submit questions using
the Q&A panel.
2. Slides and recording
sent after the webinar.
3. Please complete survey
after the presentation.

3. Continuous Quality in the
Digital Age
Bob Randall
Virgin Media
Gidi Pridor
Perfecto

4. 4

5. 5
54% would
switch banks
based simply
their satisfaction
with the bank’s
mobile app
CNBC Apr 2015

6. WHO ARE VIRGIN MEDIA?
Double Play : 19%
• 4.2m fixed telco customers
• 3.1m Mobile customers
• 4.5m broadband customers • 4.9M TV & VOD customers
Internet & Data TV & VideoTelco & Mobile
1
Double Play : 19% Double Play : 19%

7. OUR TESTING SERVICE AT A GLANCE
UK
India
75 FTE 197FTE
…We’re 72% offshore, in India…Our testing service is now
4 years old
We’re responsible for the Test and Quality
across all IT projects delivered
Mobile Broadband
Digital TV Mobile Apps
Business Engineering
Corporate
Access
Transformation
TMMi Level 5
accreditation awarded
in 2015
Innovation programmes
delivered ensuring best in
class tooling and
methodology
We run 100k test cases per year
and find on average, 13k P1-P4 defects
One of only 9 organisations worldwide to
achieve this top tier accreditation
30
2

8. 42
69
102
150
2012 2013 2014 2015
93
93 projects
already
delivered as at
Q3 2015
ITProjectsperannum
We deliver 250% more projects now compared 2012
+70%
Aspiration for mobile test
automation
90% reduction in P1-P2s
in production
increase in the volume of
defects identified prior to ‘go-live’,
with only 1 P1 and 1 P2 since
2013 in production
55%
We have Transformed our Digital Quality 3

9. Managing Documents in VM Cloud
Our customers are Going Mobile
Watching TV
Controlling Set Top Boxes
Browsing Our Websites
Managing Their Accounts
Finding Virgin Wi-Fi Hot Spots
What Next?
4

10. Customer Behaviours are Driving Change
• Mobile browsing doubled in a 12
month period from January 2013
• Mobile conversions track 10%
greater than browse
• Conversions doubled within 18
months
• Growth of mobile is constant
• Smart devices are now the norm,
mobile first services are the default
• Desktop based orders will be the
minority by December this year
THE RISING IMPORTANCE OF MOBILE FOR VIRGIN MEDIA
5

11. Old Process, New Challenges
100% Manual Testing (hand-held
devices)
Devices were procured or borrowed
individually by each team
Devices were insecure, often
lost/missing and unavailable when
needed
No corporate standard for mobile
quality & test coverage
All testing had to be conducted on-site,
no option to offshore
Quality Efficiency
Overhead Control
• Limited coverage
• Poor oversight
• Slow cycle
• QA bottleneck
• Scalability
• Availability
• Logistics
• Cost
• Can’t Offshore
• Can’t keep up
• Security
• Governance
• Standardization
• Audit
• Reliability
6

12. TRANSFORMATION
We Had To Do More For Less, A Lot Less…
Transform the way we approached mobile testing
• Scale & Speed
• Reduce Costs
• Improve Quality
CQ
Continuous Quality
7

13. Key Requirements for Transformation
Robust
Automation
Sustainable Device Agnostic Unattended
Central Cloud
Platform
Remote testing -
offshore
Avoid device
mgmt.
Governance
Flexibility to
Scale
Add users Add use cases Update devices
Enterprise-
grade
Standard
Security SLA Reliability
Future
Readiness
Easily integrated Any environment
Any Framework ,
CI etc.
Investing in technology was the only realistic way.
SPEED
SCALE
EFFICIENCY
COVERAGE
8

14. Cloud Based Continuous Quality Lab
Proprietary &
Dev/TestEnvironments(+CI/CD)
9

15. Use cases
India
Netherlands
UK
10

16. How has it changed what we do?
We have embraced Continuous Quality as a practice
Mobile and
Responsive Web
Testing
• Centralized enterprise test lab
• Virtualised devices as a cloud service
• Maximized test coverage
• Accelerated using Automation
Mobile Application
Security Testing
• centralised Security Testing capability
Networks Virtualisation
and Testing
• Network virtualisation
11

17. 2012 2013 2014 - 2015
Transforming to Stay Ahead
Standardising Processes and Methodologies Enabling Efficiency Transformation & Innovation
Standardisation &
Industrialisation Delivery Quality Cost Effectiveness
HP LOAD RUNNER STANDARDISATION
HP QUALITY CENTRE STANDARDISATION
STANDARDISED TESTING ACROSS VIRGIN MEDIA
TRANSITIONED 239 SILOS INTO STRATEGIC TCOE
CENTRALISED AND STABILISED TEST ENVIRONMENTS
SV
REMOTE HFC MODEM ACCESS
DATA MASKING
MONITORING AND ANALYSIS
ENABLING OFFSHORE MOBILITY TESTING
TEST ENVIRONMENT VIRTUALISATION
ENABLING OFFSHORE TV TESTING
EARLY ENGAGEMENT – STATIC TESTING
STRATEGIC TEST AUTOMATION
TASKTOP SYNC: DEV AND TEST COLLABORATION
SAUCELABS DIGITAL CLOUD EXECUTION PLATFORM
TEST INNOVATION AND TRANSFORMATION PROGRAMME
TEST ENVIRONMENTS DASHBOARD
HP SAAS HOSTED APPLICATION LIFECYCLE MANAGEMENT
HP SAAS HOSTED PERFORMANCE TESTING SUITE
FORMALISED RISK BASED TESTING
NEW MEDIA PERFORMANCE TEST ENVIRONMENT BUILT
TMMI LEVEL 5 ACCREDITATION
QUALITY MANAGEMENT
NON FUNCTIONAL PROCESS IMPROVEMENTS
TASKTOP DATA: REAL TIME DATA REPORTING
NEOLOAD: CLOUD PERFORMANCE TESTING
Embracing technology and tooling
has been key to our success
SECURITY TESTING CAPABILITY
AGILE READY TEST CAPABILITIES
INTEGRATED MOBILE TEST AUTOMATION
12

18. Results
Perfecto’s tight integration with Selenium has helped us develop our strategic
mobile automation framework, which will be our primary mechanism for
automating all mobile application testing
Perfecto has enabled us to extend our Responsive Test Automation
platform to use real devices instead of emulators to get increased
confidence that we’re testing the real customer experience
Responsive
Testing
Framework
The on-premise capabilities of Perfecto have enabled us to be able to test
Connected Home type services from India,
On
premise
capabilities
By utilising Perfecto, we’ve been able to take a significant portion of our
mobile applications testing to India
Perfecto has enabled us to conduct 1st level application support through
our offshore application support team
13

19. Our current challenges
CHANGING DELIVERY MODELS – THE SHIFT TO BI-MODAL IT
• Transformation to a Digital Business is demanding a
change in IT delivery approaches
• The increasing need for agility is driving a need for Bi-
Modal IT, through embracing Agile delivery methods
for faster paced programmes
• The demands on our Testing service are changing
• Increased levels of Automation, broader technical skill
sets, DevOps and changing governance controls and
processes are needed
MODE 1 – WATERFALL
MODE 2 – AGILE / ITERATIVE
14

20. How Perfecto Mobile contributes to solving the
challenges ahead 15
Integration of Perfecto Mobile within our
new DevOps reference architecture for
Continuous Delivery operations
Deployment of Perfecto Local Link to
enable more flexible testing in scrum
team environments
Further deployment of Perfecto Local Link
to enable “Location Based” testing
Integrated automation through Selenium
scripting
Requirement
Management ALM Defect Management
Software
Configuration Management Build and Packaging
Automated Testing Data Analysis/Code
Quality
Continuous Integration Artifact Repository Continuous DeliveryOrchestrates
Virgin Media
Continuous Delivery
Framework

21. Quality Maturity Model
1
Initializing Definition Transitioning Integration Continuous Quality
2 3 4 5
Release
Frequency
Test
Coverage
Automation %
Your Test
Process
Your
Environment
2-3/year Quarterly Every 45-60 days Monthly When done
<30% coverage -50% coverage 50-90% coverage 90%+ coverage 100% coverage
+/- automation Regression
testing at end of sprint
<33% automation Some
automation with legacy
QA tools
-50% automation Up to 75% automation >75% automation
Agile development
Process +Waterfall
QA
Separate QA and Dev
ALM tools
Test cycles take less
than a week
Centralized
testing platform
Performance testing
Production monitoring
No performance testing
No monitoring in
production
Applying desktop testing
approach to mobile
Isolated test environments
(no lab)
Using dev tools for
automated testing
(Java, C#)
Automation framework
Performance testing
several times per year
Build process runs
automated tests (CI)
Optimized automation
framework
Managed lab
16

22. Future 17
• Perfecto Mobile is a key component in our strategy to implement CI and CD, we
see this capability evolving and maturing in 2016, alongside further automation
development
• Device types are proliferating – as new mobile and wearable technologies become
relevant we’ll use Perfecto Mobile to host and test against these new devices
• Security is going to be more and more important as we adopt mobile devices into
every aspect of our lives. We have a responsibility to ensure our customers and their
data is kept as secure as possible so we intend to use Perfecto Mobile to support our
device security strategy
• Virgin Media is a TMMI Level 5 accredited Test Centre of Excellence and we want
to exploit the good practice, technologies, processes and approaches we’ve
developed to support the wider Liberty Global family of companies

23. Summary 18
Change is the only constant
Transformation isn’t easy
Automate to keep up
Build Enterprise grade solutions

24. Any Questions?