Continuous Testing for Optimal Mobile Peroformance - STPCon Spring 2014
•Download as PPTX, PDF•
0 likes•180 views
No description can be viewed at this moment
Recommended
Recommended
More Related Content
What's hot
What's hot (10)
Viewers also liked
Viewers also liked (10)
Similar to Continuous Testing for Optimal Mobile Peroformance - STPCon Spring 2014
Similar to Continuous Testing for Optimal Mobile Peroformance - STPCon Spring 2014 (20)
More from SOASTA
More from SOASTA (20)
Recently uploaded
Recently uploaded (20)
Continuous Testing for Optimal Mobile Peroformance - STPCon Spring 2014
- 1. 1© 2013 SOASTA. All rights reserved.
- 2. 2© 2013 SOASTA. All rights reserved. • Continuous Integration for mobile performance testing • The cloud to help test for mobile scale • Real devices to validate user experience with actual device performance • Access and distribution of the right information to the right people at the right time
- 3. 3© 2013 SOASTA. All rights reserved. Load tests run via Jenkins Daily (or more frequent) updates keep teams “performance focused” Identify bottlenecks & inform teams, just like unit and functional tests in Jenkins Suitable test environment * Manage with a new metric! …while functional automation runs on real devices capturing device-side perf metrics
- 4. 4© 2013 SOASTA. All rights reserved.
- 5. 5© 2013 SOASTA. All rights reserved. Does it work as designed? Does it perform under extreme circumstances? Raw Materials Real Drivers What happens in actual use? Assembly Line
- 6. 6© 2013 SOASTA. All rights reserved. Functional Testing Performance, Reliability, Scalability? Development Real Business Flow and Devices What happens in actual use? Build
- 7. 7© 2013 SOASTA. All rights reserved. Source Code Repository Devs Build Server Check out Test Results Check in Run Tests Unit Tests ☐ Pass ☐ Fail ✓ To QA or Users
- 8. 8© 2013 SOASTA. All rights reserved. http://www.soasta.com/knowledge-center/whitepapers/
- 9. 9© 2013 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds o Consistent environments o Developer – Tester Collaboration
- 10. 10© 2013 SOASTA. All rights reserved. o Device test automation o API/Web test automation o Version control o Build automation o Artifact management o Configuration automation o Test clouds o Device clouds
- 11. 11© 2013 SOASTA. All rights reserved. Test tool launches tests on the devices4 CI Server kicks off test compositions for updated apps 3 Updated App sent to CI Server, compiled and made testable 1 Devices poll test tool over Wifi or carrier network CI server polls version control for latest code JENKINS SERVER Test Tool Analytics Main Database Apps pushed to devices (!) 2 Test tool setup CI setup Mobile devices (anywhere) USB Hub (slave) Results sent to testers & developers 5 Version Control System
- 12. 12© 2013 SOASTA. All rights reserved.
- 13. 13© 2013 SOASTA. All rights reserved. Customers SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
- 14. 14© 2013 SOASTA. All rights reserved. “Record” Analytics Test Tool Databas e Main Perf Tester 1 SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
- 15. 15© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
- 16. 16© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
- 17. 17© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. Backend Metrics Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS)
- 18. 18© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Analytics Load GeneratorLoad Generator Virginia (IBM)
- 19. 19© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Analytics Load GeneratorLoad Generator Virginia (IBM) Federated Cloud
- 20. 20© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Analytics Load GeneratorLoad Generator Virginia (IBM) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. Federated Cloud Backend Metrics
- 21. 21© 2013 SOASTA. All rights reserved. Analytics Test Tool Databas e Main Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Analytics Load GeneratorLoad Generator Virginia (IBM) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Access for all Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) Federated Cloud Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. Backend Metrics
- 22. 22© 2013 SOASTA. All rights reserved.
- 23. 23© 2013 SOASTA. All rights reserved.
- 24. 24© 2013 SOASTA. All rights reserved. o Select your critical set o Determine who controls/manages o Consider private and public requirements o Agile teams like hands-on o Integrate to perf testing o Add a performance metric
- 25. 25© 2013 SOASTA. All rights reserved. o Manage a private device cloud installed behind the firewall o Provide real-time device inventory, status and availability o Look for online access for all your teams o Enable 24x7, secure, remote access to devices o Remotely view and interact with a selected mobile device and any mobile app • Use devices for manual testing and monitoring • Remote screen-share devices for collaboration on design or issues
- 26. 26© 2013 SOASTA. All rights reserved.
- 27. 27© 2013 SOASTA. All rights reserved. o Include real mobile devices and their “system” metrics o Test in the production environment o Test to 2-3x the expected peak traffic levels o Test across geographies o Test corner cases o Measure in real time http://www.soasta.com/knowledge-center/whitepapers/
- 28. 28© 2013 SOASTA. All rights reserved.
- 29. 29© 2013 SOASTA. All rights reserved.
- 30. 30© 2013 SOASTA. All rights reserved. o Pass / Fail is the first step o Facts, Jack. o Manage with a new…metric
- 31. 31© 2013 SOASTA. All rights reserved.
- 32. 32© 2013 SOASTA. All rights reserved. Load tests run via Jenkins Daily (or more frequent) updates keep teams “performance focused” Identify bottlenecks & inform teams, just like unit and functional tests in Jenkins Suitable test environment * Manage with a new metric! …while functional automation runs on real devices capturing device-side perf metrics
- 33. 33© 2013 SOASTA. All rights reserved. • Continuous Integration for mobile performance testing • The cloud to help test for mobile scale • Real devices to validate user experience with actual device performance • Access and distribution of the right information to the right people at the right time • Information? http://www.soasta.com/knowledge-center/whitepapers/ • Bjohnson@soasta.com - @bradjohnsonsv
Editor's Notes
- TouchTest has been integrated with popular CI frameworks like Jenkins and Bamboo, Making 100% autmomated regression testing using real devices a reality. We start with 3 core components: First: TouchTest. The SOASTA platform is deployed wherever needed. In a lab or hosted in an MSP or the cloud. Tests are captured and replayed on real devices, and all the work is done via any browser in a visual test envorinment Second, there is the Continous Integration system. We labeled Jenkins here, since it’s open source and we have excellent plugins, but this can be done with others like Bamboo. One point to make here is that if you a part of a separate QA team and Dev doesn’t use CI, this is still applicable. However you pick up builds to test, you plug in this CI solution. Third, you need some devices. Our solution uses off the shelf, unaltered iOS or Android devices over carrier or WiFi networks. This mean you can build labs inexpensively and quickly, and with TouchTest, any registered device, wherever it is, is part of your private device cloud. We can also ship a lab ready to run or host them for you, too. To finalize setting the stage, a CI server polls the source control system, looking for new builds. That’s just part of what they do. With TouchTest, our “agent” – just a web app – listens for the signal to run any automated tests. Here’s were the magic happens! Task 1: A new build hits the VCS signaling the CI server to compile and run the task that injects our library to “Make App Touch Testable” Task 2: This is one of the really hard parts. Some vendors have stated that mobile testing isn’t a good candidate for CI – this is one of the reasons…physically getting the Apps to the devices to test. We’ve automated this task, and every time the app is compiled, it’s pushed to any devices designated to receive it. There’s some addition magic here, to, we’ve built additional tasks to wake up sleeping devices and reboot them. This solves several other issues voices by customers who have used hosted device labs in the past. Task 3: Now the CI system calls TouchTest to kick of our tests – what we call compositions. Tastk 4: TouchTest contacts all the devices and runs the tests on the devices Task 5: Results including from the devices, including performance information like battery, CPU and memory, are sent to TouchTest. In the Jenkins UI – users see all the pass/fail results, and if the engineer wants to dig into the issue, we’ve embedded the TouchTest dashboard into Jenkins. Now lets take a quick look a TouchTest in action.