Fast, Strong & Nimble Mobile Performance Testing


Published on

Sept. 18, 2012 webinar on Mobile Performance Testing. Mark Tomlinson (former LoadRunner PM) and Dan Bartow (SOASTA VP Product Management) provide overview of planning and executing mobile testing that measures for Fast (front-end) and Strong (back-end) mobile systems. Recording of event will be available here:

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I’ve been testing software for nearly 20 years – in many, various roles and capacities – but always keeping hands-on in the craft. I’ve worked on all types and sizes of tests – specifically around performance…testing for micro-latencies in the Windows kernel up to the first large-scale website test emulating more than 1,000,000 concurrent users on a website (in November 2000) I’ve worked at startups, mid-size companies – and large software vendors and big banks – as an employee and consultant my experience is that decisions about testing are made by individuals…who choose to think, or to not think about testing I’m an avid speed freak: motorcycles, bicycles, downhill skiing…anything that goes fast, I’m in!
  • 4 years ago – I started giving webinars with guidance about how to prepare for the “upcoming WAVE of MOBILE adoption” – I know that Dan likes to surf – and I’m sure when he was learning…there were lots of crashes in the waves. Well, the parallel with mobile testing is that if you didn’t start paddling hard 4 years ago…you’re probably crashing in the waves like Dan.
  • The right call: - the back-end systems will quickly exhaust memory resources because slower mobile transactions will reserve memory for longer periods of time - when heap reaches a specific % utilized of Physical memory – you swap…then you crash - to prevent this, you would need to segment slower traffic from faster traffic…via load balancing proactively, or adding application logical partitioning
  • Ron Patton outlines 16 different steps – none of them specifically including mobile Scott Barber lists 24 pages in a performance engineering strategy template – the word mobile doesn’t appear “ Testing Computer Software” from Kaner, Faulk and Nguyen – mentions “devices” generically, but not mobile We need to refresh and update our working templates, approaches and strategies – to include mobile
  • Since mobile is new, the test plans often are built by new engineers or testers – without historic experience. - the scope of activity is narrowed to just the device …not the back-end systems - the geographicall end-user location is often missed - the connection/disconnection scenarios are often missing Many times the mobile application developers are the only people involved in testing an application – (WHY!!!?!?!???) I’ve actually heard a customer say: “oh, we don’t need QA – this is a mobile application.”
  • Fast, Strong & Nimble Mobile Performance Testing

    1. 1. Presents:Webinar 1
    2. 2. Expert Perspectives on Mobile Performance TestingTODAY’S SOASTA PRESENTERSMark Tomlinson: West Evergreen Consulting, former LoadRunner PMDan Bartow: SOASTA VP Product ManagementModerator: Brad JohnsonAgenda:• Poll question• Embracing mobile performance testing – whats new and whats not• Building a performance test plan to address what really matters• Testing and measuring for "Fast" and "Strong" all at once for mobile and web• Implementing a solution that does it all with agility to match the new normalQuestions: 2
    3. 3. o Sheer Number of Devices (953M Smartphones)o Different Operating Systemso Scale of Global Customers (6B)o Dynamic Content (Video, Animation, …)o Emerging People to Machine Interfaces Mobile performance at scale can not be tested manually. 3
    4. 4. Mobile User Native Experience (mUX) Apps is affected by CPU app & device mUX is most Battery performance only affected by back Memory end load and network performanceWebbrowser Mobile Web Apps Appusers Web Server Load Server Cache App Balanc Server Web erMobile Mobile Web Shared web & Server Databasbrowser browser traffic mobile eusers mUX is affected by HTTP(S) infrastructure native app, device, back-end and network performance Fast AND Strong Hybrid Mobile Apps AppReal Web Serverdevices Load Cache Server App& large Balanc Server Web erscale Native Dedicated or Server Databasmobile App Web shared mobile infrastructure etraffic traffic 4
    5. 5. o Development teams test at a unit and component level (as always)o Test teams load test with web-based traffic (if load testing at all)o Ops teams monitor with services built pre-Smartphone o No one communicates, yet agile dev is all about communication! 5
    6. 6. 6
    7. 7. o 20 years in testingo Performance Guyo Small tests & Big testso Small & Big companieso …speed freak. 7
    8. 8. - Old applications migrated to new mobile formats – “transformation”- New Applications added-on to existing systems – “bolt-on”- Entire new business units created for mobile consumer experience- Device manufacturers are investing and expanding – alarmingly!!- “The number of Smartphone users worldwide is predicted to exceed 1 billion by 2014” (Parks Associates, 2010)- “Tablets to hit 100M shipments in 2012” (Source: ABI Research, 2012) “Every customer I’ve worked with in the last 18 months is prioritizing mobile performance testing.” -Mark Tomlinson 8
    9. 9. WHAT’S NEW? WHAT’S NOT!oDevices – lot’s of ‘em oThe risks of poor performanceoBattery Life & Heat oResponse Time & VolumesoGeographic Diversity oRoot-cause AnalysisoLatencies – insanely high oCapacity EstimationoVariable Processor Frequency oTest Planning oBugs…lot’s of ‘em 9
    10. 10. A real story about a customer situation:1.1,000 requests per second into the services infrastructure2.Systems were fine-tuned, low-latency (< 250 ms)3.Memory (in .NET CLR) was also optimized with perfection4.They introduced a new mobile experience to the end-user5.10% of the company started using the mobile app6.What happened next…??? YOU MAKE THE CALL… 10
    11. 11. Enhance existing plans for mobile: • Specific response-time goals for mobile • Connected/disconnected test scenarios • Mobile test lab setup & configuration • Matrix of devices to be tested/supported • Define end-user location and conditions • Monitors for device’s physical resources • Monitor client-side performance Let’s review an example test strategy… 11
    12. 12. Ensure that new mobile plans include: • Mobile Performance Objectives (time) • Defined scope for device types • Detailed explanation of end-user behavior • Inter-app test conditions • Carrier-specific test conditions • Failure and recovery conditions • Back-end system dependency and impactsLet’s review a new mobile test plan… 12
    13. 13. Dan Bartow 13
    14. 14. Measure what MattersGlobalWebbrowserusersGlobal Hybrid Mobile AppsMobile Any App Serverbrowser mix Web Load Serverusers Cache App of Balanc Web Server er traffi Native Dedicated or Server DatabasReal c at App Web shared mobile infrastructure edevices any traffic& large scalscalemobile e o Dev teams test early and continuously for FREE withtraffic CTLite o Test teams respond quickly with complete mobile coverage o Ops team validate and tune production infrastructure 14
    15. 15. Thanks & Q&A Next Webinar: Sept. 27, 10 a.m. PST “Automated Testing & Continuous Deployment for Mobile Apps in the Cloud” (SOASTA & CloudBees)Register at White Papers, Webinar Recordings, Case Studies - Knowledge Center Contact SOASTA: Contact Mark @mtomlins 866.344.8766 Follow us: Contact Dan @PerfDan 15