Mobile App Quality Roadmap for DevTest Teams
•Download as PPTX, PDF•
3 likes•1,665 views
Join Perfecto & CloudBees for a presentation on how to drive mobile app quality feedback in every build, on real devices. Watch a demo featuring the CloudBees Jenkins Workflow showcasing automated testing with Perfecto's Continuous Quality Lab.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Mobile App Quality Roadmap for DevTest Teams
Similar to Mobile App Quality Roadmap for DevTest Teams (20)
More from Perfecto by Perforce
More from Perfecto by Perforce (20)
Recently uploaded
Recently uploaded (20)
Mobile App Quality Roadmap for DevTest Teams
- 1. Mobile App Quality Roadmap for DevTest Teams Dec 2015 © 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 2. A few housekeeping items 1. Submit questions using the Q&A panel. 2. Slides and recording sent after the webinar. 3. Please complete survey after the presentation.
- 3. 12/16/2015 3© 2015, Perfecto Mobile Ltd. All Rights Reserved. The Common Agile Challenge Maturity Aspiration
- 4. Distress call! • Release late by 2.5 months (from committed baseline) • Low quality confidence • Coverage gap concerns • High defect rate • “Stabilization” exposes more issues • Low customer ratings after the delayed release • Is this “Agile” or “Fragile”???
- 5. Reality … Testing Slows Down Agile 1.0 Teams ... 12/16/2015 5© 2015, Perfecto Mobile Ltd. All Rights Reserved. “We are doing agile iterations for development and testing and it is going quite well. One of our greatest challenges is dealing with the stuff we cannot finish within the iteration – currently its one big pile at the end” VP R&D for a 1000- person group “Let’s leave the serious testing for the release testing phase”
- 6. Why? A limited amount of feedback comes inside the development cycle (days) 0 20 40 60 80 100 120 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 done done done Late testing Slower velocity Higher cost Test Execution
- 7. Agile – Take 2 7
- 8. In Real Agile – Specify Acceptance Tests FIRST to DRIVE Design/Development Elaborate Requirements Test Design Technical Design Coding / Unit Testing Test Execution + Fixing Done Apply ATDD Thinking Test scenarios guide code design ATDD* = Build Quality Into Design to prevent defects rather than just finding them http://www.agilesparks.com/test-first-reading-list Test Automation *ATDD: Acceptance Test Driven Design Define acceptance tests before developers begin coding Start earlier Done is Done
- 9. Testers create more detailed test examples Developers use them to guide technical design/specification Team pulls a few stories
- 10. Story is pulled into development Testers focus on detailed test plans, automation preparation, data preparation. Developers code and do developer/unit/small testing
- 11. Developer verifies the story passes the team “ready for story testing” criteria: • Code is “in the build” • The acceptance tests for this story pass in the dev env • Good code coverage using automated unit tests • Sanity test passes Result: Teams using ATDD Thinking get much higher, automated, test coverage(some above 90%) than the typical test-last teams. Both due to discipline as well as due to developers awareness to the acceptance tests Developer marks story as dev done, ready for testing Development done means the acceptance tests already pass
- 12. Tester pulls story into testing • Tester finds significantly fewer defects • Product Owner is happier with how the story works & finds fewer surprises. • Both as a result of deeper alignment achieved in the specification workshops with ATDD scenarios. Developers meanwhile pull more stories into coding
- 13. UAT Regression Performance Security Functional Progression Exploratory Achieve Agile Testing Flow at the Story AND Feature/Epic/Iteration level 1. Add Feature/Epic level testing 2. left-shift more and more testing through automation, enabling teams using environments/tools/knowhow and more. The 3-level Test Strategy Pyramid - Story+Feature/Epic/Iteration+Release Across platforms and user conditions
- 14. Automation Ownership Poll Who’s responsible for automation in your world? 1. Automation COE separate team/department 2. Automation COE engineers assigned to the Agile Teams 3. Test Engineers/Testers on Agile Teams with support by Automation COE experts 4. Developers/Testers on the Agile Team 5. Other 14
- 15. Continuous Integration/Delivery Needs an Automation Engine Complex Delivery Pipelines Delivery of App and Config Robust and Highly Available Results Report SOURCE CODE CONTROL SYSTEM Code Commit </> Development Production Continuous Integration/Delivery Commit Build Test Stage Deploy
- 16. Development Production Continuous Integration/Delivery Commit Build Test Stage Deploy Jenkins – #1 Automation Platform for CI/CD “Jenkins is for building, testing and delivering software projects continuously.” 10 Years of Open Source Innovation http://www.jenkins-ci.org
- 17. • Pipeline-As-Code • Version control, Code review, etc. • Entire flow is one concise Groovy script using Workflow DSL • For loops, try-finally, fork-join, etc • Stages throttle concurrency of builds • Allocate build nodes and workspaces • As many as you want, when you want • Human input/approval integrated into flow Multi-Stage CI/CD: Enter Jenkins Workflow
- 18. Jenkins Workflow: Powers Multi-Stage CI/CD Pipelines Development Deploy/Publish Commit Build Regression Release ? ? Branching Looping Restarts Checkpoints Manual Input Conditionals Unit Test Smoke Test Jenkins Workflow Continuous Feedback
- 19. CloudBees Jenkins Workflow Stage View Jenkins Workflow Stage View SOURCE CODE CONTROL SYSTEM Code Commit </> Development Production Jenkins Workflow
- 21. Perfecto CQ Lab 12/16/2015 21© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 22. Easily Applied for Browsers … Mobile Create Challenges … UI Integration Unit Testing
- 23. Build Process 12/16/2015 23© 2015, Perfecto Mobile Ltd. All Rights Reserved. 5min 32s 37min 29s1 min 48S6 min 12s1 min 3S
- 24. •Connect WIFI •Use iPhone •Skip that OS update •Connect to WIF •Check Facebook •Login your app •Watch YouTube videos •Answer the phone •Clear emails •Return to your app – check status •Accept LinkedIn invite •Use navigation app •…
- 25. UI Integration Unit Testing Performance Security Mobile Requires Shifting the Plan … Target Devices Target Env.OS OS versions Form factors CPU & Memory Manufacturers Wearables IOT Networks Orientation Location Notifications Events Device events Edge tests
- 26. The Team and Project 12/16/2015 26© 2015, Perfecto Mobile Ltd. All Rights Reserved. Test Type Target Devices Time Verification Tests Verify the app can be installed on the devices and basic flow works All Few minutes Smoke Tests Test the “happy path” and validate the version ready for regression (10%) 4 Up to 20 minutes Regression Tests Full functional tests • 2 clean devices + 6 WT scenarios • Happy path 8 Up to 5 hours Edge tests Full functional tests • 4 different WT scenarios • Extreme parameters • Load with n Virtual users 16 Up to 12 hours * Wind Tunnel devices
- 27. WT results 12/16/2015 27© 2015, Perfecto Mobile Ltd. All Rights Reserved. Android Galaxy 4.1Samsung Galaxy 6 5.1
- 29. CloudBees Jenkins Platform Editions Enterprise Edition Cloud/Container Support Platinum Support Gold Support
- 30. Recap • Minimize post-sprint testing • ATDD is a Sprint 0 Automation enabler • Manage the test workflow • Continuously prioritize and refactor the test suite • CD building blocks: • ATDD to better achieve definition of done • Automation that works • Continuous optimization for fast (relevant) feedback 12/16/2015 30© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 31. Thank You © 2015,Perfecto Mobile Ltd. All Rights Reserved.
Editor's Notes
- Submit questions using the Q&A panel. We try to answer all questions during and after the presentation. Our experts do not have access to the chat interface Participate in the polls Your participation is critical to a relevant, meaningful discussion. Watch your email for a copy of the recording and slides Today’s presentation will be recorded and sent to you. Complete the survey following the presentation Your feedback helps us provide educational tools like this webinar.
- http://www.flickr.com/photos/andreasnilsson1976/1307180278/sizes/l/in/photostream/ slipping release due to long time finalizing quality.
- Continuous Delivery needs to be managed by a robust automation engine. The Automation engine must orchestrate the full CD pipeline including execution of each stage of the pipeline and the best of breed DevOps tooling used within each stage. Developers can simply focus on creating code. When they check in code to the Source Code Control System, the CD automation engine will see that check-in, pull the code out, run the application compile and build process, execute and monitor unit tests and functional tests, possibly deploy to a staging environment and then run user acceptance tests and then, if desired, the CD automation engine can even automatically deploy to production. (Automatic deployment to production is optional as not every organization is ready for that. However, Continuous Delivery means that there will always be production-ready code available to push to production when appropriate.) A enterprise-ready CD automation engine will need to have three characteristics: 1) It needs to be able to create and manage complex, real-world CD Pipelines. These are pipelines that may not be just linear sequential steps. There may be parallel branches for parallel testing, there may be loops and restarts, and there may be a need for human-intervention within the pipelines. The automation engine needs to support all of these. 2) Continuous Delivery often is applied to not just the application code but the actual environment that the application runs in as well. This means defining the application environment configuration (OS parameters, etc) “as code” and managing that “configuration-as-code” through the CD Pipeline along with the Application code. This helps insure that the application will perform exactly in production as it did in Dev and Test. 3) The last characteristic of an enterprise CD automation engine is that it needs to have the capabilities expected of enterprise software including the ability to run in an HA, or highly available, configuration. Since the CD Pipelines are touching production systems and are automating the path to production, the CD Automation engine needs to be HA. So, how can we do CD?
- Jenkins is open source software that was originally developed over 10 years ago.
- Here are some really cool features in Jenkins Workflow. SAs talk here about coolness of Workflow.
- As we said, CD involves creating Pipelines for application code. This means that for every code “check-in” by every developer, the new code needs to go through the Pipeline that will take it from Dev to Prod. So, the CD Platform needs to manage the execution of many of these Pipelines. And the pipelines can be complex. They may have decision points or branches. They may need to be paused or restarted. Modeling the real-world process of application delivery requires that these pipelines be robust, flexible and powerful.
- The CloudBees Jenkins Platform also provides a dashboard that shows the current state of the pipeline activity, such as this example, which is critical to providing a full understanding of how things are performing.
- CloudBees Jenkins Platform Enterprise Edition: For Enterprise DevOps or Infrastructure teams that want to: Run Jenkins operations @scale to support a large number of developers and/or teams Optimize the performance and efficiency of the CD platform Monitor usage and share resources across their Jenkins implementation Have access to Jenkins support experts for quick problem resolution Ships with All Advanced Enterprise Features for both: CloudBees Jenkins Enterprise CloudBees Jenkins Operations Center CloudBees Jenkins Platform Team Edition: For small Dev teams or organizations with a limited number of developers that want to: Improve developer productivity by leveraging advanced developer features for Jenkins Easily get started with continuous integration (CI) and continuous delivery (CD) Have access to Jenkins Support experts for problem resolution Ships with a starter-pack of Enterprise Features focused on Developer Productivity: Developer Productivity Improved developer feedback loop and eliminate downtime with Validated Merge for Git and GitHub Pull Request Team Management Organize teams with folders Recoverability from failures using backup Security Roles-based access control Isolate team-sensitive slaves Analytics Build and performance dashboards Support Module Faster diagnosis for production issues