1.
Mobile Performance TestingPart 1: How to Performance Test Mobile Client Applications on aMobile Device

2.
Series of 3 Mobile Performance WebinarsPart 1: Clients applications – NowPart 2: The Server – Feb 21Part 3: The Network – March 14All at 10 AM EST

3.
Alan Trefzger Newman YangJan Princen

4.
Software Quality Improvement• XBOSoft – Software QA Consulting and Testing – Currently 100+ QA experts – San Francisco, Beijing, Oslo, Amsterdam

5.
Quality Life Cycle

6.
“Proven Results” “Our products are very complex. We expected a one year ramp up time, but XBO was able to contribute to our release cycles in just 6 months." - Development Vice President, Hyperion-Oracle"The results are very encouraging. Thanks a lot for all the effort put into thisproject. We really appreciate your work."- Manager, Quality Assurance, Autodesk "Everything youve done is impressive and we are quite pleased, well done! So keep up the good work." - QA Director, CA

7.
Agenda• Importance of Mobile Performance Testing• Types of Mobile Performance Testing• Mobile Performance Testing for Local (Android) Applications – Setting up a performance test – Using different tools to test• Case study – How fast can Angry Birds run• Evaluate the tools• Q&A

8.
Importance of Performance• Mobile internet traffic expected to pass desktop internet traffic in 2014.• By 2015, over half of all mobile subscribers are expected to be engaged in m-payments .• Amazon reported that the company calculated it lost 1% of sales for every extra tenth of a second required to load a page.• Google experiment: traffic and revenue fell by 20% when the pages with more results took an extra half second to load.• Akamai found that the number of users who abandon a page after 3 seconds was 57%. - Mobile Site Optimization, Sep 2011, Strangeloop

9.
Types of Mobile Performance 1. Client Application / device performance 2. Server performance 3. Network performance

10.
Server PerformanceFollowing are some of the challenges, variables, and issues covered in the Server Performance webinar.• Amount of data sent,• Number of communications between Server and Client/Browser• Tools• Optimization Strategies

11.
Network PerformanceFollowing are some of the challenges or variables covered in the Network Performance webinar.• Latency, jitter, packet loss,…• 2.5G, 3G, 4G LTE• CDNs

12.
Local Device and App Performance Why focus on the local device and App? • Discover performance based on different hardware/software configurations with the same application for a defined task. • Discover which platform meets a set performance objective with the least cost. • Determine which platform uses the least resources.

13.
Performance is important to almost every application or taskBuy a product Obtain a bank balance Look up a ticker symbol Kill a man with Download one page spear (WoW) of information Start an application (game)

14.
How Fast can Angry Birds Run?

15.
Performance Test StrategyPerformance test on a few devices, and extrapolate that it will work on those devices that are more powerful (more memory, higher CPU Hz, etc…, 5 criteria are defined later).Different than functional test. It may work functionally on a certain device, but not meet performance requirements. Example, for Android, you can test on a well chosen set of 7-15 devices and be confident that your application will work functionally on the majority of Android devices. However some of those devices may not meet performance requirements.

16.
Defining the testProduct Manager wants to find the lowest phone that satisfies the performance requirements.• Requirements: – Launch Angry Birds in 15 seconds – Throw an Angry Bird far away in 8 seconds – Play Angry Birds and play music at the same time – Play Angry Birds and at the same time delete 100 SMS’s in 5 secondsLowest solution that fits requirementsUsing tools and manual testing to make purchasing decision

17.
Steps for performance test1. Define task, requirements and test.2. Choose a sample set of devices.3. Choose one or more tools and run them.4. Compare to manual test.5. Use Quadrants’ DB of devices to eliminate those phones which do not meet performance requirements.6. Chose a mobile device based on criteria other than performance.

18.
Configuration of devices• 3 popular Android devices

19.
Tools• Quadrant – measures CPU, Memory, I/O, 2D and 3D.• Linpack - measures CPU floating point performance by single thread and multi thread.• Smartbench – Simple tool to get idea if the device is suited for example gaming.• Others: Jbenchmark, SPMark, Nenamark

20.
Quadrant ResultTotal Score averages the scores of the 5 specific criteria 3500 3000 2500 2000 Lenovo A1 1500 HTC Desire HD 1000 Motorola 500 ME860 0 Total Score CPU Memory I/O 2D 3D

21.
Quadrant Mobile DB

22.
Using Linpack 60 50 40 Lenovo A1 30 HTC Desire HD 20 Motorola ME 860 10 0 Single Thread Multi ThreadMFLOPS - Millions of Floating Point Operations per Second Lenovo A1 HTC Desire HD Motorola ME 860Single Thread 9.91 39.01 38.94(MFLOPS)Multi Thread 8.55 32.51 51.96(MFLOPS)

23.
Using Smartbench300025002000 Lenovo A11500 HTC Desire HD1000 Motorola ME860 500 0 Productivity GameMotorola ME860 >HTC Desire HD > Lenovo A1

24.
Manual Test Angry Bird

25.
Manual Test Result - Angry Birds Scenarios Required Lenovo A1 HTC Desire HD Motorola ME 8601. Launch Angry Bird 15 seconds 19 seconds 14 seconds 10 secondsV2.1.12. Throw the bird to a 8 seconds 8 seconds 6 seconds 5 secondsfar distance3. Play Angry Bird SlightV2.1.1 and play music at Continuous Continuous Continuous discontinuitythe same time4. Play Angry Birds anddelete 100 SMS 5 seconds 4 seconds 3 seconds 2 secondssimultaneously

26.
Results of Manual test• Lenovo A1 - didn’t meet requirements• HTC Desire HD - met requirements• Motorola ME 860 - met requirementsConclusion• Since the Lenovo A1 device didn’t meet the performance requirements, any device with a lesser Quadrant score than the Lenovo will be rejected.• Any device with an higher score than the HTC device will be accepted.

27.
Tying it all together1. Define task, requirements and test – Angry Birds, certain speed, multi apps2. Choose a sample set of devices – Lenovo, HTC, Motorola3. Choose a tool (0ne or more) – Quadrant, (Linpack, Smartbench)4. Compare to manual test – Find out which device conform to performance requirements5. Use Quadrants’ DB of devices to eliminate those phones which do not meet performance requirements.6. Chose a mobile device based on criteria other than performance.

28.
Thanks Questions and Answerswww.xbosoft.com services@xbosoft.com408-350-0508 28