Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mobile Network Performance Testing


Published on

Mobile Performance Testing consists of three parts:
Part 1 - Client Application performance
Part 2 - Server performance
Part 3 - Network performance

The slide deck we cover how to performance test the network as it applies to a mobile device.

■ Causes of network related problems
■ What tools and services are available
■ CDNs and other strategies to mitigate network cause problems

  • Be the first to comment

Mobile Network Performance Testing

  1. 1. Mobile Performance Testing: Part 3: Testing the Network
  2. 2. Mobile Performance:Part 3 – Testing the Network Alan Trefzger XBOSoft Senior Quality Consultant
  3. 3. Agenda• Introduction – 2013 State of Mobile Testing Survey• Testing the End-User experience• Network Problems• Solutions – What can we do? – Case Study: Analyzing mobile websites across different Networks. (and how to fix them) – CDN: Content Delivery Network.• Conclusion
  4. 4. Introduction
  5. 5. The Mobile Challenge• Mobile Internet usage is projected to pass desktop Internet usage in 2014• Mobile traffic is doubling every year from 2009 onward and continue until 2014 4,000,000 3,500,000 Mobile Traffic in 3,000,000 Terabytes per month 2,500,000 2,000,000 1,500,000 1,000,000 500,000 0 2009 2010 2011 2012 2013 2014
  6. 6. Mobile Performance• 60% of mobile failures are performance, non functional issues.• Google found that a .5-1 second increase in page load time resulted in a 20% decrease in traffic and revenue• Gold standard used to be 6 seconds, but now is 3 seconds• We expect the online experience to be as fast as our desktop experience, but it isn’t, and that is a problem.• Mobile is slower, but it doesn’t have to be.
  7. 7. 2013 State of Mobile Testing Survey by XBOSoft• Ratio of developers to testers: 2 to 1• Plans for 2013 will decrease testers relative to developers.• Only 41% do performance testing• 60% don’t consider the network when testing.• 64% don’t test in other countries where the network characteristics will be a lot different.• 62% have dedicated mobile QA teamsDownload the full report at
  8. 8. Mobile Performance TestingPart 1: Testing the Device• Tested a high price, mid-priced and cheap phone.• ‘Angry Birds’ app was used for the test.• Used the tool Quadrant to compare.Part 2: Testing the Server• Used Webpagetest to analyze performance on a mobile device.• Looked at ways to speed up performance.Part 3: Testing the Network• Used Shunra’s Mobile Performance Report to test and compare.
  9. 9. If your not considering the network before you start developing,you are setting yourself up for failure.
  10. 10. Testing the End-User Experience
  11. 11. Testing Goal• Test your application as the end-user experiences it.• End-to-End Testing: Testing the entire product.
  12. 12. Testing the End-User ExperienceApp Website Server Internet • Functional, security, usability testing, etc. • 59% stop here and declare ‘success’
  13. 13. Service VirtualizationApp Website Server Internet • Functional, security, usability testing, OS Phones Tablets etc. • 59% stop here and declare ‘success’20+ Android 3,000+ 100+ Apple 5 3 15Testing actual devices
  14. 14. Service Virtualization DB / mainframeApp Website Under Server construction Internet Service Virtualization – Intelligent Stubbing • May not be able to access company DB / MF • Parts of the product may not be finishedTesting actual devices
  15. 15. Service Virtualization 3rd party DB / mainframeApp Website Under Server construction Internet Composite Application • Access 3rd party web services • May cost to access • They don’t want you stress testing their serviceTesting actual devices
  16. 16. Service Virtualization Testing 1,000 3rd party DB / mainframe virtual usersApp Website Under Server construction Internet Service Virtualization • Need to stress test 100 – 10,000 users • Transport-level emulation of many users • GUI emulationTesting actual devices
  17. 17. Network Virtualization Testing 1,000 3rd party DB / mainframe virtual usersApp Website Network Under Server construction Internet Network Virtualization • Hardest to emulate • Most performance problems caused by networkTesting actual devices
  18. 18. What connection do you have? 3G Wi-Fi 2.5G Internet 4G Website ServerNetwork varies widely, very unpredictable
  19. 19. Problems of Mobile Networks(that Broadband doesn’t have)
  20. 20. Problems of Networks• Uncertainty – don’t know ahead of time what network you will be connected to.• Uncertainty – don’t know ahead of time what the quality of network will be.• Very hard to repeat network conditions for testing because everything is constantly changing.• Global differences in network speeds and reliability.• Customers don’t care where the problem is. You will get blamed for poor performance even if it is the network.
  21. 21. Problems of NetworksCan’t fine tune it later or solve it later.• Can’t upgrade bandwidth.• Can’t upgrade hw in mobile device.• Can’t put in an accelerator.• Can’t do many of the things that you could if it was a PC. The mobile Network is just shining a light on pre-existing problems.
  22. 22. Slow network speeds• Long, Slow TCP connections• Slower network speeds means strategies that worked for desktop/broadband connection, will not work for mobile. e.g. Serial connections can take very long time with a slow network.• TCP Time-outs – Increasing TCP timers is NOT the answer. – Changing the timers can be high risk and cause unexpected timing problems. – Can cause application timers to timeout. – Whenever you see TCP timeouts, you will have significant performance degradation.
  23. 23. Latency Latency - time it takes to go end-to-Internet end (or round trip). Latency is 2-10 times longer than broadband. Jitter – latency variation over time. Jitter can vary, be very large and will cause disconnects. • A change in latency from 2ms (broadband) to 400ms (3G network) can cause a page load to go from 1 second to 30 seconds
  24. 24. Connection Disruption• Sometimes when you are traveling and your cell phone connection is being passed from one cell tower to another, a disruption can be caused.• Almost no packet loss for broadband, but a truck going passing between you and the cell tower can cause packet loss and re-transmission.• Some think that bandwidth is the problem, but many times it is NOT, it is a latency problem.• Packet drops with retransmissions will be common.
  25. 25. Global Network Issues• Testing the same phones in USA, India, and China. – Phones, browsers, and websites that worked fine in the USA didn’t work in India.• Data charged is a big factor in the emerging markets. – What network providers’ charge is not necessarily based on the size of the download. – Amount customer was charged can vary greatly depending on the browser or phone.• Speed can vary greatly depending on phone, browser, or network
  26. 26. SolutionWhat Can We Do?
  27. 27. What can we do?Many say, “How can we change our customers perception? so we don’t get blamed.”Can’t do, we must change our reality.• Developing for mobile is different than desktop.• Developing for performance from the beginning.• Just because we haven’t experienced a big failure yet, doesn’t mean won’t and soon.
  28. 28. Plan ahead• Critical that you carefully plan a mobile service deployment.• The network can make a problem an order of magnitude worse.• What if your mobile service goes viral and your infrastructure can not support it, then all you have done is caused a large number of potential customers never to consider you again.• Use Service Virtualization to test your mobile service.
  29. 29. Service VirtualizationEmulating services that are hard to replicate during testing• The Network• Third party composites• Actual in-use internal database• Parts of application that are under construction• 10,000 usersNot just a stub, but emulates what is happening or what data is coming backService Virtualization = Intelligent StubbingAllows Test early / Test often
  30. 30. Network Virtualization Testing 1,000 3rd party DB / mainframe virtual usersApp Website Network Under Server construction Internet Network Virtualization • Most performance problems caused by network. • Hardest to emulate – must use a toolTesting actual devices
  31. 31. Tools• Network Link Conditioner - Apple in its developer kit for iOS6, good place to start – Simulate internet connection and bandwidth speeds – Simulate a 3G, DSL, Edge, very bad connection, etc…• Network Catcher – Shunra – can capture network profile and environment from almost anywhere in the world. – Gives details on why the problems happened. – Global library of networks from all over the world.
  32. 32. Tools• vCat – Shunra – emulates a network profile – LoadRunner is user virtualization a – LR is closely integrated with vCat – LR controls the entire test, including vCat, – Very repeatable – What if scenarios• Perfecto Mobile• Device Anywhere
  33. 33. Live vs. Emulated Network• Live, real Network – Cannot duplicate conditions – Too many variables – Actual conditions won’t be consistent• Emulated Network – Repeatable – Can vary conditions – What if scenarios
  34. 34. Network VirtualizationShunra offers a free service using their NetworkCatcher and vCat products to analyze mobile applications on the various networks.Mobile Application Performance Report• Analysis of your mobile application on various networks.• Recommendations on how to improve your mobile application.
  35. 35. Performance on Different Networks
  36. 36. Network CharacteristicsWiFi 4G fair 3G good 3G 2.5G
  37. 37. Component and Waterfall Waterfall ComponentsNumber of Requests bycomponent type
  38. 38. Performance Score• received a very low score of 54/100. Example: 8 of the 25 areas where performance improvement can be made.
  39. 39. What else can be done? CDNsCDN – Content Delivery NetworksGoal: To serve content to end users with cheapest cost or highest performance. Optimal situation is when these two goals are aligned.• This is achieved by optimizing delivery across local networks• Instead of accessing data from a central server, the CDN parses out the requests to the nearest server.
  40. 40. CDN – Content Delivery NetworksThe Problem: Data delivery can be slow depending on severalfactors1. Request Location2. Current server load3. Network Load London Example: Service to Wash DC a client in Wash DC can be a lot faster than service to Internet service to a client in London Dulles VA Website Server
  41. 41. CDN – Content Delivery NetworksThe Solution:• High Speed replication of content• Server load balancing• Service request routing• Web Caching London Wash DC Example: Service to a client in Wash DC Internet can be as fast as service to a client in London Dulles VA Website Server
  42. 42. Improvements to Watch ForBetter integration of CDNs to MobileTraffic Optimization: Wireless operators are managing content to provide a better overall mobile experience.• Fast dormancy• Bigger pipes• Offloading to other networks• Optimizing traffic
  43. 43. Characteristics of a Good mobile app/• Worrying about the end user experience ahead of deployment.• Proactive engineering• Fault tolerant• 3 second response• Can handle spikes in load• Performance standards are specified in advance.• Topology of my servers is planned and understood.
  44. 44. Issues to consider at beginning, but especially before deployment• Set performance service standards, e.g. 3 seconds response• What if your service goes viral.• Is it internal or external.• Paid/not paid.• Capacity planning.• Revenue generating.• Is your logo on it? Bad performance will hurt your brand.
  45. 45. Optimization Strategies(Covered in Part 2: Server Performance Testing)1. Reducing the number of HTTP Requests – Consolidating Resources – Embed resources in html for first-time use – HTML5 Web Storage – Uni-directional server updates2. Reduce Payloads: – Compress – Resize – Simplify
  46. 46. ConclusionMoore’s Law for computer power and storageNielsen’s Law for Internet bandwidthLong term looks good, but in the mean time.• Test early, test often.• Best way to solve mobile performance problems is to prevent them. Don’t wait until after deployment to start worrying about performance issues.• No network performance testing will lead to disaster!
  47. 47. Q&AFollow us on twitter: @XBOSoftAre you ready to test Mobile Performance?Let XBOSoft help you get started.