Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Wind Tunnel Testing with Personas
Similar to Wind Tunnel Testing with Personas (20)
More from Lizzy Guido (she/her)
More from Lizzy Guido (she/her) (16)
Recently uploaded
Recently uploaded (20)
Wind Tunnel Testing with Personas
- 1. Wind Tunnel
- 2. Agenda • Digital Challenges • Overview of Wind Tunnel • Personas • What is a Persona? • Existing and Custom Personas • Adding Wind Tunnel capabilities to your test • Persona • Points of Interest • Rotation • Sample Test using Wind Tunnel • Eclipse/Java/Selenium • BB&T U app and Perfecto sample website • Viewing results © 2016, Perfecto Mobile Ltd. All Rights Reserved. 2
- 3. Digital Reality Challenges Digital Reality Challenges Users are expecting apps to work in the mobile moment (regardless of conditions) Failure to satisfy is costly © 2016, Perfecto Mobile Ltd. All Rights Reserved. 3
- 4. Mobile Experience Quality Challenge The Mobile Experience Quality Challenge 1) Building common user environment condition profiles into test suites 2) Assembling multiple solutions to execute tests across profiles 3) Seamlessly integrating user condition testing within Continuous integration pipelines 4) Analyzing app performance blind spots 5) How to set up a lab and these various conditions? © 2016, Perfecto Mobile Ltd. All Rights Reserved. 4
- 5. Example of Changes – In Person vs. Online © 2016, Perfecto Mobile Ltd. All Rights Reserved. 5
- 6. User Centric Quality Lab with End User Conditions Using a Banking/Financial App 1 2 Network Quality Cellular, Wi-Fi 3 Varied Device Conditions 4 Multiple devices Specific profiles 5 Location 6 Resource Conflicts 7 Interacting with sensors Peak time - Millions of transactions 8 Simple Functional Testing Is Not Enough© 2016, Perfecto Mobile Ltd. All Rights Reserved. 6
- 7. Overview of Perfecto Wind Tunnel Wind Tunnel Profile Selector Usage Patterns 1. Login 2. Search 3. Select 4. Payment 5. Logout Unchanged Test Flows Example: User Journey on Banking/Financial App Existing Flows Analyze & Optimize • Lives in NY • iPhone 6S • Verizon Customer • Constant WhatsApp msg Georgia 12 3 4 PersonaPoints of Interest Wind Tunnel Testing Wind Tunnel Report © 2016, Perfecto Mobile Ltd. All Rights Reserved. 7
- 8. The Personas • The Perfecto Wind Tunnel personas define the test environment • A Persona is defined by a set of capabilities • There are 5 available pre-defined personas • Personas can be created and edited • Georgia • 47 year old high income, corporate road warrior living in Manhattan traveling to San Fran & London • Traits • Network: Constantly connected, frequent network switches across various WiFi and cellular networks • Brands of choice: Apple & Verizon • Device(s): iPhone 6S, Smartwatch & iPad Pro • App loyalty: Intense social user, WhatsApp, Mobile Banking and PO & Expense Report approvals Location Carrier & Brands Device coverage Background apps © 2016, Perfecto Mobile Ltd. All Rights Reserved. 8
- 9. 5 Pre-Defined Personas 9© 2016, Perfecto Mobile Ltd. All Rights Reserved.
- 10. Wind Tunnel – Languages & Frameworks 10© 2016, Perfecto Mobile Ltd. All Rights Reserved.
- 11. Using & Tailoring Personas • Set a persona using the windTunnelPersona capability • Tailor an existing persona by defining new capabilities capabilities.setCapability("windTunnelPersona", "Georgia"); capabilities.setCapability("windTunnelPersona", "Georgia"); //Add location either by an address or lat,long coordinates capabilities.setCapability("windTunnelLocation", "New York City,NY"); //Set orientation to landscape and portrait capabilities.setCapability("windTunnelOrientation", "portrait"); //list the apps running in the background capabilities.setCapability("windTunnelBackgroundRunningApps", "Candy Crush Saga"); © 2016, Perfecto Mobile Ltd. All Rights Reserved. 11
- 12. Persona JSON file • Example persona JSON file { "settings":{ "windTunnelLocationAddress":"New York, NY", "windTunnelBackgroundRunningApps":"United,Twitter,Outlook,Marriott,BofA", "windTunnelVNetwork":"4G LTE Poor", "windTunnelOrientation":"landscape" }, "properties":{ "description":"Corporate Road Warrior, Age 47", "name":"Georgia", "image":"PUBLIC:Personas/Georgia.png"}, "device":{ "model":"iPhone-6S||iPad Mini 4" } } © 2016, Perfecto Mobile Ltd. All Rights Reserved. 12
- 13. Rotation Snippet 13© 2016, Perfecto Mobile Ltd. All Rights Reserved. Map<String, Object> params = new HashMap<>(); params.put("state", "landscape"); params.put("method", "device"); driver.executeScript("mobile:device:rotate", params);
- 14. Points of Interest •The focus points that are part of your Wind Tunnel execution and reports •The app issues that are being tested in real user conditions Examples: • Purchase transaction that takes too long • CPU consumption of the running Facebook app © 2016, Perfecto Mobile Ltd. All Rights Reserved. 14
- 15. Adding Points of Interest to the test •Java Example • Following code adds a Point of Interest that will add an indication of a successful page load. • Will be selectable in the wind tunnel report ... // Wind Tunnel: Add success point of interest to the Wind Tunnel report WindTunnelUtils.pointOfInterest(driver, "Point Of Interest 1: Page load","Success"); © 2016, Perfecto Mobile Ltd. All Rights Reserved. 15
- 16. Single Test Report The Single Test Report timeline includes: • Points of interest • Wind Tunnel events • Checkpoints (validation) © 2016, Perfecto Mobile Ltd. All Rights Reserved. 16
- 17. Example Script • Create a new Perfecto Lab Project in Eclipse • Add sample java code to your project • Run the test • Review the run results • The Perfecto report ("C:testreport.pdf") contains a link to the new Wind Tunnel report © 2016, Perfecto Mobile Ltd. All Rights Reserved. 17
- 18. Demo
- 19. References • Complete Wind Tunnel Guide https://community.perfectomobile.com/series/24692 • Defining Capabilities https://community.perfectomobile.com/series/20208/posts/945255 • Network Conditions https://community.perfectomobile.com/posts/1110522 • Sample Script https://community.perfectomobile.com/series/25427/posts/1114150 © 2016, Perfecto Mobile Ltd. All Rights Reserved. 19
- 20. Next Steps…Get Certified! Become a Perfecto Specialist in Selenium RemoteWebDriver and Appium! HOW? • Log into www.perfectomobileacademy.com and register. • Click on Self-Paced eLearning and select the Selenium RemoteWebDriver and Appium course. • Take the FINAL EXAM for Specialist Certification in Selenium RWD and Appium (SRA) Good Luck! © 2016, Perfecto Mobile Ltd. All Rights Reserved. 20
- 21. Thank You
Editor's Notes
- Focus on Banking/Financial Use Eclipse Project Named TrainingProjectExample RemoteWebDriverTest.java – run as java test Credentials in code: To run the sample remember to add your credentials and Lab URL String host = “myLab.perfectomobile.com"; capabilities.setCapability("user", “myUser"); capabilities.setCapability("password", “MyPassword"); Selenium and Perfecto Tree Uses BB&T U application – tried to login with username, password and validates error message for unsuccessful login Also opens device browser, goes to perfecto sample page, logs in and validates successful login
- The challenge Customers using a mobile application are themselves mobile. They enter an elevator and switch from WiFi to Cellular throughout the day. Calls and messages pop up, applications run in the background, location changes as users are on the go. A standard functional test is not enough, the testing needs to consider the environment that affects the application.
- Different users at different points of the day go through different experiences (user flows of the app) in different environments. For example, a travelling business person who is on and off planes and is exposed to poor reception; and a gamer will have a lot of CPU consuming apps running in the backgrounds. Different devices are configured differently, in terms of permissions, access, the number of background apps. For example, there may be apps using location. There is a lot happening outside the control of my app.
- In person interactions with this example Financial Institution ( B of A) has declined 23% from 2009 to present and online mobile transactions have increased 94%. Online/Mobile Banking is growing exponentially!
- Let’s dig into Quality. Mobile quality differs dramatically from web. Specifically it is the user conditions. Therefore Quality teams must look beyond testing function toward testing functions across all user conditions. The objective is to set defining and controlling user conditions as a key required capability. This can be expressed as a day in the life of how you use your favorite app. Think about traveling from point a to b and various network, device, device resources consumed and potentially conflicting apps that are also running on your device. App performance is influenced by a variety of factors, such as degraded network conditions, frequent network transitions back and forth between cellular and WiFi, on board sensor or resource conflicts along with a non-stop stream interruptions – SMSs, notifications, system pop-ups and phone calls. These factors impact app performance. Mobile app quality programs must account for these factors during testing to consistently deliver high-quality end user experience.
- Meet real user experiences and bring the real user into the lab. Test conditions, unplanned events, environment changes and expand the test coverage and predictability for production via the continuous quality lab. ** Adding Wind Tunnel to an existing script will effortlessly expand your test coverage to include real user experience. Testing what is going on outside the control of the application and not only what is inside the application. This includes the application server, the carrier, and the device OS. For example, testing what happens when an incoming call event occurs during a transaction in a bank application. The solution Wind Tunnel offers the following abilities to extend coverage to real user conditions. Pre-defined Personas that simulate network, location and other properties Points of interest that identify crucial application flows Timer reporting ability to track performance of the application All these abilities are easy to add to the test and visible in a new reporting interface designed to help understanding and drill down of issues.
- Defining the real user environment is a challenge. Perfecto has created the concept of Persona in which a type of user is defined. Each persona contains a series of properties that together simulate a real end user. By setting a persona all its pre-defined properties are set. It is possible to use the Perfecto provided profiles and also to create custom profiles. There are five different available personas ready to use (with unique capabilities). For example, Georgia, a 47 year old high income, corporate road warrior living in Manhattan, traveling to San Fran & London. Her traits include: Network: Constantly connected, frequent network switches across various WiFi and cellular networks Brand of choice: Apple & Verizon App loyalty: Intense social user, WhatsApp, Mobile Banking and PO & Expense Report approval. Georgia’s list of background apps allows testing your app, considering the resources consumed by the other apps constantly running on Georgia’s device, such as high CPU, network consumption, camera, etc. These traits are translated into capabilities, creating Georgia's persona that can be easily incorporated into your test.
- There are five different available personas ready to use. See above image and detailed table for their unique capabilities. Existing tests can be executed across the set of personas. The personas can be parameterized and run utilizing common continuous integration servers, such as Jenkins or test execution managers such as TestNG.
- Perfecto agnostically supports a wide variety of languages and test frameworks
- To set a persona use the windTunnelPersona capability. Example - java To tailor an existing persona with specific values, define new capability values to override those that have already been defined in the persona.
- What is a persona? Continued… A persona is a set of capabilities stored in the format of a JSON file located in the Perfecto lab repository. The pre-defined persona JSON files are located in PUBLIC:Personas folder (for example: PUBLIC:Personas/Georgia.json). When you set values of elements common to a persona definition, you override the persona’s original corresponding value (e.g. setLocation or mobile:location:set override the persona location). Notice Georgia has more than one device, therefore, unless you have overridden that value, one of her devices is randomly selected for the test execution. The network profile can be more easily defined using the constants listed in the Wind Tunnel post listing the Network Conditions for Wind Tunnel. https://community.perfectomobile.com/posts/1110522 Don’t forget to override this value by null, if you’re using a cloud with no network, or your test will fail. If you do not need any specific persona attributes, yet you do wish to add the Wind Tunnel mechanism to your test, you can avoid adding a persona altogether, use other Wind Tunnel commands, and still generate the report. A reference persona with mostly empty fields is automatically added to your capabilities, with some outline image appearing in the Wind Tunnel report. When using a reference persona remember to: Set the device, or your test will fail. Use the vitals start/stop command to ensure the generated Wind Tunnel report includes the device vitals.
- Note that while the view of the device is still in portrait, the display of the application is in landscape. The report will show the device in the correct orientation.
- The Wind Tunnel Point of interest (POI) allows mobile teams to pin point real user specific issues around performance, functionality and more. By using POI users can get a detailed dashboard with view into the actual error for a specific transaction. For example, a purchase transaction that takes too long (see the Wind Tunnel’s Timer functionality), or the CPU consumption of the running Facebook app (in the vitals’ CPU graph). This allows identifying app quality issues in specific environment under specific conditions before app delivery to production, assuring end-user satisfaction and eliminating redundant long debugging cycles.
- Points of interest are the points in the script that the user may want to log, in order to highlight in the Wind Tunnel report. For example, loading a new page. The user can also use the point of interest to indicate a logical failure such as incorrect data showing in the app. ** For details and code snippets see the Adding Wind Tunnel to your test post in the Wind Tunnel guide. https://community.perfectomobile.com/series/24692/posts/1048049
- Wind Tunnel introduces a new interactive test report enabling real time analysis. Using a persona in your test will create a Single Test Report. Download the Perfecto report from your test, using the downloadReport method, and click on the link to open the Single Test report. This report is opened in a browser. Alternatively, you can open the Report from the link that appears on the report page in the Reporting tab of the CQ Lab interface. You can also retrieve the report URL via: String reportURL = (String) (driver.getCapabilities().getCapability(WindTunnelUtils.WIND_TUNNEL_REPORT_URL_CAPABILITY)); Understanding the results: When reading the Single Test Report, first take a look at the points of interest reported from the test and identify any failures (commands and timers). Take a look at the previous command, the app & OS vitals (graph), the device screenshots and video. (When selecting a specific point in the Single Test Report timeline, the device video is synced to that time.) ** For details, the Wind Tunnel story, and code snippets, see the Viewing results post in the Wind Tunnel guide. https://community.perfectomobile.com/series/24692/posts/1048048
- Notes: next slide we’ll run the sample test. Focus on Banking/Financial Use Eclipse Project Named TrainingProjectExample RemoteWebDriverTest.java – run as java test Credentials in code: To run the sample remember to add your credentials and Lab URL String host = “myLab.perfectomobile.com"; capabilities.setCapability("user", “myUser"); capabilities.setCapability("password", “MyPassword"); Selenium and Perfecto Tree Uses BB&T U application – tried to login with username, password and validates error message for unsuccessful login Also opens device browser, goes to perfecto sample page, logs in and validates successful login
- First – show basic test in Eclipse Project Named TrainingProjectExample RemoteWebDriverTest.java – run as java test Then show Lufthansa video with callouts
- Additional references: Measure the time it takes for an application to open, using the new Timeout parameter in the Application start command. https://community.perfectomobile.com/posts/1008846 Measure application responsiveness with user experience timing. https://community.perfectomobile.com/series/23468