Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
 
W5
Session	
  
4/15/2015	
  11:00	
  AM	
  
	
  
	
  
	
  
"Continuous Mobile Testing for
Critical Business Apps"
	
  
P...
Steven Winter
FIS Mobile
Steven Winter loves building teams that break big things! As the director of
quality for FIS Mobi...
4/8/15	
  
1	
  
CONTINUOUS MOBILE TESTING FOR
CRITICAL BUSINESS APPS
Steven Winter
Director of Quality, FIS Mobile
Speake...
4/8/15	
  
2	
  
•  World’s largest banking and
payments provider
•  Nation’s largest mobile banking
and payments provider...
4/8/15	
  
3	
  
MASS
MOBILE
ADOPTION
INCREASING
DEVICE
DIVERSITY
RAPID
CLIENT
GROWTH
CONSUMER
DRIVEN
INNOVATION
INCREASIN...
4/8/15	
  
4	
  
Reducing the Regression Tsunami	
  
Efficient Scripting Architecture	
  
Single, configurable scripts base...
4/8/15	
  
5	
  
What are the core requirements of a
new approach?	
  
ü  Vision, Goals, Metrics
ü  Robust Script Archit...
4/8/15	
  
6	
  
FIS Mobile CI System	
  
SVN JENKINS
Build
 Deploy
 Test
 Results
 Report
STEP 1
 STEP 2
 STEP 3
 STEP 4
...
4/8/15	
  
7	
  
What is the role of continuous integration
in mobile?	
  
Cease dependence on mass
inspection to achieve ...
4/8/15	
  
8	
  
What are the real benefits in speed and quality?	
  
•  250 hours of manual tests = just over 1 hour every...
4/8/15	
  
9	
  
Q & A
THANK YOU
Steven Winter
steven@fismobile.com
@winterweb
Upcoming SlideShare
Loading in …5
×

Continuous Mobile Testing for Critical Business Apps

418 views

Published on

Mobile testing presents a daunting challenge to software development shops and testers. QA/test teams must design test plans to account for multiple platforms, an enormous and ever-increasing number of devices, and frequent OS updates that often introduce dramatic changes. To add complexity, the increasingly consumer-centric demands of the field require new features and fixes—almost daily. Although test automation is often cited as the best—if not the only—possible solution for mobile, it has presented some big challenges. Tasked with compressing the testing process for FIS Mobile, the leading mobile financial transaction platform, Steven Winter and his team revamped their approach to quality by implementing precision mobile test automation, continuous integration, and on-site mobile labs. This discussion will overview how the team reduced potentially months of manual mobile testing to an overnight automation task. The objective of the discussion is to take back new ideas to identify the core requirements of continuous mobile testing and overcome hurdles to mobile quality.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Continuous Mobile Testing for Critical Business Apps

  1. 1.   W5 Session   4/15/2015  11:00  AM         "Continuous Mobile Testing for Critical Business Apps"   Presented by: Steven Winter FIS Mobile                 Brought  to  you  by:         340  Corporate  Way,  Suite  300,  Orange  Park,  FL  32073   888-­‐268-­‐8770  ·∙  904-­‐278-­‐0524  ·∙  sqeinfo@sqe.com  ·∙  www.sqe.com
  2. 2. Steven Winter FIS Mobile Steven Winter loves building teams that break big things! As the director of quality for FIS Mobile, Steven is responsible for the complete quality ecosystem for America’s leading provider of mobile financial services and the innovative power behind the Starbucks Card Mobile. With nineteen years of experience in quality, Steven has tested through a wide array of technologies from web, firmware, hardware, cameras, phones, desktop, data—and everything in between. He has continually pushed the testing edge and delivered high performance, world-class QA operations at Broderbund, First Data, Leapfrog, PureDigital/Flip, mFoundry, and now FIS Mobile.  
  3. 3. 4/8/15   1   CONTINUOUS MOBILE TESTING FOR CRITICAL BUSINESS APPS Steven Winter Director of Quality, FIS Mobile Speaker Bio   Director of Quality for FIS Mobile responsible for the complete quality ecosystem for America’s leading provider of mobile financial services, including over 1,200 banks with over 2,000 apps in market, and the innovative power behind the nations first and most successful mobile payments initiative, the Starbucks Card Mobile. With 19 years of experience in quality, Steven has successfully delivered high performance, world-class QA operations to a wide array of companies like Broderbund, First Data, Leapfrog, and PureDigital/Flip, mFoundry, and now FIS Mobile.   Steven Winter Director of Quality FIS Mobile
  4. 4. 4/8/15   2   •  World’s largest banking and payments provider •  Nation’s largest mobile banking and payments provider •  > 3000 Mobile Apps in market •  +1,600 banks, +29MM end users •  Most successful payments app •  Most successful banking app •  Award-winning mobile wallet   FIS Mobile   FIS Mobile earned the 2013 Best in Class Mobile Banking Vendor Overall from Javelin Strategy & Research, the group’s highest award for mobile banking vendors. AGENDA: KEY TAKEAWAYS •  Challenges of being an early leader in mobile quality •  Core requirements of a new approach •  Mobile test automation capabilities •  Role of continuous integration in mobile delivery •  Real benefits in speed and quality
  5. 5. 4/8/15   3   MASS MOBILE ADOPTION INCREASING DEVICE DIVERSITY RAPID CLIENT GROWTH CONSUMER DRIVEN INNOVATION INCREASING SECURITY EXPLOITS INCREASED TESTING COMPLEXITY NEED MORE DEVICES UNABLE TO FINISH TESTING IN 1 SPRINT THOUSANDS OF APPS TO SUPPORT IN MARKET SCALABILITY AND SPEED PRESSURE CAUSES EFFECTS What were our big challenges?   What are challenges of being an early leader in mobile quality?   OPERATIONAL TECHNICAL •  Immaturity of mobile test automation •  Deciding what to automate •  Regression “Tsunami” •  MDM – Mobile Device Management •  Staff •  ROI expectations •  Diverging delivery teams   •  Mobile testing is difficult •  Device compatibility matrix •  Lack of robust test tools •  Reliable test results •  Configurable script framework •  Technical debt  
  6. 6. 4/8/15   4   Reducing the Regression Tsunami   Efficient Scripting Architecture   Single, configurable scripts base used by all teams C ore Product Integrat ion an d Depl oy ment Hoste d Ser vices UNIT API DEVICE
  7. 7. 4/8/15   5   What are the core requirements of a new approach?   ü  Vision, Goals, Metrics ü  Robust Script Architecture ü  “Divide and Conquer” Approach ü  Real Devices ü  Need the Right Tools for the Job ü  Immediate Results to the Developers ü  Tight Control of Test Noise   How do we test?   Test Framework Tools/Platform Primary Usage Unit •  jUnit •  Fast, code-level test writing and execution •  Granular testing from unit test level upwards Server API •  Java •  Apache HTTP Client •  Web services testing of server API Client UI •  EggPlant •  TouchTest •  Calabash •  Selenium (SWAP) •  Requirements-based testing of UI flow •  Testing against real devices and emulators Performance •  CloudTest •  Jmeter/BlazeMeter/New Relic •  In-depth performance runs •  Targeted small-scale tests executed frequently Security •  Fortify •  IBM AppScan •  NowSecure •  ViaLabs (in-house tool) •  Static code analysis •  Dynamic server analysis •  External scans done by vendors •  Internal scans done by security tester
  8. 8. 4/8/15   6   FIS Mobile CI System   SVN JENKINS Build Deploy Test Results Report STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 What is the current state of mobile test automation capabilities?   Object Recognition On-device Performance Telemetry (RUM) Enhanced Mobile Performance Testing Maturing Test Platforms and Support of CI Device Clouds 1 3 5 2 4
  9. 9. 4/8/15   7   What is the role of continuous integration in mobile?   Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place. - W. Edwards Deming   “ ” What is the role of continuous integration in mobile?   •  Essential to compete in time-to-market and scalability •  Represents a huge paradigm shift •  Provides faster, more specific test results so developers can work smarter •  Identifies integration issues sooner •  Enables continual quality  
  10. 10. 4/8/15   8   What are the real benefits in speed and quality?   •  250 hours of manual tests = just over 1 hour every night •  Nightly Regression Tests instead of waiting until feature complete – 4 months to overnight •  25% reduction in time to market for production deployments •  Earlier bug fixes •  Declining field failures (aka “Escapes”) •  Lots of ancillary tools as a result of developing test and CI frameworks. One tool alone saved 40+ hours a week!   SUMMARY •  Mobile test technology is catching up •  Very real benefits from mobile CI •  We are freeing the humans!
  11. 11. 4/8/15   9   Q & A THANK YOU Steven Winter steven@fismobile.com @winterweb

×