Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to SQADAYS 21 Moscow 2017
Similar to SQADAYS 21 Moscow 2017 (20)
More from Adam Sandman
More from Adam Sandman (20)
Recently uploaded
Recently uploaded (20)
SQADAYS 21 Moscow 2017
- 1. Software quality assurance days XXI International Conference of Software Quality Assurance sqadays.com Moscow. May 26–27 , 2017 Denis Markovtsev Inflectra Corporation. Moscow, Russia Automated UI testing in uncontrollably improving environment
- 2. 2Automated UI testing in uncontrollably improving environment tools for developers, testers and planners
- 3. 3Automated UI testing in uncontrollably improving environment solution for automated UI testing Desktop. Web. Mobile.
- 4. 4Automated UI testing in uncontrollably improving environment
- 5. 5Automated UI testing in uncontrollably improving environment Real problem – share experience
- 6. 6Automated UI testing in uncontrollably improving environment Real problem Demo example Why?
- 7. 7Automated UI testing in uncontrollably improving environment Outlook plugin for Office 365
- 8. 8Automated UI testing in uncontrollably improving environment Works in desktop Outlook and
- 9. 9Automated UI testing in uncontrollably improving environment Why test UI? Is it unavoidable?
- 10. 10Automated UI testing in uncontrollably improving environment Environment changes
- 11. 11Automated UI testing in uncontrollably improving environment Plugin load error
- 12. 12Automated UI testing in uncontrollably improving environment Plugin display issues
- 13. 13Automated UI testing in uncontrollably improving environment Environment City Client Type Result Browser New York paid works Browser Hong Kong trial not working Outlook Moscow trial works Outlook London paid not working Depends on client type and geographical location
- 14. 14Automated UI testing in uncontrollably improving environment Matrix controls us
- 15. 15Automated UI testing in uncontrollably improving environment Environment is constantly improving and we can not control it
- 16. 16Automated UI testing in uncontrollably improving environment Speed of reaction is important
- 17. 17Automated UI testing in uncontrollably improving environment Automated UI tests are required
- 18. 18Automated UI testing in uncontrollably improving environment
- 19. 19Automated UI testing in uncontrollably improving environment Outlook plugin - web application, even on desktop
- 20. 20Automated UI testing in uncontrollably improving environment //span[@class='badge ng-binding']
- 21. 21Automated UI testing in uncontrollably improving environment Locators Desktop: embedded IE XPATH Browser: xpath_to_frame XPATH
- 22. 22Automated UI testing in uncontrollably improving environment Same tests for Outlook and browsers
- 23. 23Automated UI testing in uncontrollably improving environment Email upload Email client launch Authorization Email opening Plugin activation Test Finalization
- 24. 24Automated UI testing in uncontrollably improving environment Microsoft Active Accessibility UI Automation Managed Java Access Bridge
- 25. 25Automated UI testing in uncontrollably improving environment Few details
- 26. 26Automated UI testing in uncontrollably improving environment SeS('Mail_Folders') .DoClickNode(userName + ";Inbox"); JavaScript Object Locators Properties Actions
- 27. 27Automated UI testing in uncontrollably improving environment “No programming”
- 28. 28Automated UI testing in uncontrollably improving environment Testing project results
- 29. 29Automated UI testing in uncontrollably improving environment Other projects
- 30. 30Automated UI testing in uncontrollably improving environment Regression test coverage
- 31. 31Automated UI testing in uncontrollably improving environment Monitoring
- 32. 32Automated UI testing in uncontrollably improving environment Single tool One language Unified approach Reporting
- 33. 33Automated UI testing in uncontrollably improving environment Get Rapise https://www.inflectra.com/SQADAYS21.aspx GitHub https://github.com/Inflectra/office365-outlook- plugin-ui-testing Email denis@inflectra.com
Editor's Notes
- Good afternoon. Thank you for coming. My name is Denis. It happened that more than 15 years I am working on tools for developers and testers. At different times it were compilers, and IDEs, tools for load and functional UI testing.
- Now I work at Inflectra. Inflectra is one of leading vendors of ALM and functional testing solutions..
- At present moment I am focused on UI test automation. At Inflectra we develop Rapise – it is an IDE and execution environment for automated testing of web, desktop and mobile applications. The product has success on international markets, but until today was not presented in Russia. We think that it is time to close the gap and see the outcome.
- Demonstrating the tool is informative but a bit boring. For this purpose people invented webinars which you can listen relaxing in a comfortable chair.
- Therefore I’ll talk about a real problem that Rapise helped to solve. Experience that I want to share today is pretty universal and you can use it in other projects applying other tools, even free ones. We frequently help clients to solve complex tasks in testing process setup. As a rule our clients are not online stores or vendors of mass market software solutions. In many cases they are big companies using software to support their internal business processes. These are solutions for financial sector, ERP, healthcare, e.g. blood banks. As a rule such clients are pretty closed and do not hurry up to give access to their systems. And they have high standards for software quality. Frequently they ask for help because their software solutions are complex, implemented using different technologies, frameworks, programming languages and custom components. Therefore our work resembles remote surgery operations. It is really complex for several reasons.- Not always we can work with and see a live system. - We even do not know what metrics were achieved by a client in a testing project. In many cases it is a binary function: client satisfied, not satisfied. - Almost always we sign NDA which forbids us to disclose information about client and it’s application for demo purposes. Thus today I’ll speak about a real problem, but using a demo example which we prepared specially for this conference. This will enable you not only learn information but also put hands on demo example yourself. Our example consists of public application, testing tool and test sources. All of these can be downloaded, installed and executed.
- We frequently help clients to solve complex tasks in testing process setup. As a rule our clients are not online stores or vendors of mass market software solutions. In many cases they are big companies using software to support their internal business processes. These are solutions for financial sector, ERP, healthcare, e.g. blood banks. As a rule such clients are pretty closed and do not hurry up to give access to their systems. And they have high standards for software quality. Frequently they ask for help because their software solutions are complex, implemented using different technologies, frameworks, programming languages and custom components. Therefore our work resembles remote surgery operations. It is really complex for several reasons.- Not always we can work with and see a live system. - We even do not know what metrics were achieved by a client in a testing project. In many cases it is a binary function: client satisfied, not satisfied. - Almost always we sign NDA which forbids us to disclose information about client and it’s application for demo purposes. Thus today I’ll speak about a real problem, but using a demo example which we prepared specially for this conference. This will enable you not only learn information but also put hands on demo example yourself. Our example consists of public application, testing tool and test sources. All of these can be downloaded, installed and executed.
- Let’s return to the problem. We will test Outlook plugin for Office 365. Outlook is a mail client from Micrsoft, Office 365 is a cloud based version of Microsoft Office. The plugin is named TextMiner, it scans email body for signatures and automatically extracts information about name, address, job title, company name, etc.
- Outlook plugin is a web application that can work in a browser and in desktop version of Outlook on Windows and Mac.
- Why develop automated UI tests in this case? We are familiar with the test pyramid and know that this type of testing is the most complex and expensive. Is there a way to save resources? Is there an alternative? One can test backend with unit tests. Integration tests can throw email body into backend using CURL. This is simple. The question is what to do with the frontend, with user interface? Though user interface is pretty simple there is one “But”.
- The plugin works in external environment which may change any time. What worked a minute ago may just stop working. Consequences of unexpected and unpredicted updates from Microsoft side can be different. Let’s list a few of them.
- Developer’s nightmare – receive a bug report looking like: “nothing works”. The reason of this can be changed API or authorization and license checking mechanism.
- Plugin may be displayed incorrectly. Such situation may occur when default frame size is changed or because of issues with mailbox data access.
- Moreover such problems may depend on type of a client and it’s geographical location. Reasons of such behavior are hidden in cloud nature of such a service as Office 365. Different accounts are attached to different versions/builds of the application. Probably updates are applied to trial accounts first and in selected regions. For example users from Europe may experience issues with plugin loading and at the same time clients from North America may be just fine. Also accounts created at different times may be attached to different builds of Office 365 server side components.
- So we have conditions when we do not control situation 100%. We are in the cloud and it makes decisions for us. In the case of desktop applications we at least have an opportunity to follow update schedule of the operating system and timely check our application for compatibility on Virtual Machines with specific configuration. In a cloud we do not have an opportunity to control version of the software we are working with. Such situation is not always the case. For example, in commercial cloud systems supporting manufacturing and sales where software failure leads to tangible financial losses, vendors warn about updates beforehand and give a chance to adapt. Unfortunately Office 365 is not the case. Modern tendency to issue releases frequently makes situation even worse. Though all developers know situation when fixing a bug in one module leads to appearance of others in other parts of the system. Because other modules accounted for incorrect behavior of the module in hand.
- It is a paradox. Office 365 team makes a better system with each update, but it leads to problems for plugin vendors.
- Speed of reaction and ability to extinguish the fire is very important in such a case. And it is very desirable to detect a problem before users.
- Thus a minimal set of automated UI tests is required. It must check operability of the plugin in desktop versions of Outlook and different browsers. Also the test coverage must be executed constantly in monitoring mode, for different types of accounts (paid, trial), in different geographical zones, to detect problems as early as possible and react appropriately. The question to experts from the auditory. Would you develop automated UI tests in such a case? If not, what is the alternative? If yes, what tools would you use? [interactive session goes here, approximately 3-5 minutes]
- Our client decided to use Rapise. I will not show the source code of the tests and the tool in details. We just do not have time for this. At the end of the talk I’ll show how to get Rapise for free and download tests. To run the tests you will need Office 365 account (you can easily get a trial one) and install TextMiner into it.
- So, let’s proceed to the test coverage. I remind you that plugin is a web application, even in desktop version of Outlook it is loaded into embedded browser.
- Therefore UI element locator does not depend on the container. In all cases it is the same XPATH for a given element.
- To get UI element on desktop one needs to attach to embedded browser in Outlook (it is Internet Explorer on Windows), and then use XPATH locator. For a regular browser use XPATH of the frame, where the plugin is loaded, and then XPATH of the element. This enabled the customer to build the test system in such a way that test recorded for browser is suitable to run on desktop without any changes.
- This way the customer saved time and resources required for development and support of tests. Additional economy was achieved with ability to record user actions, locators are calculated automatically in this case (though some postprocessing may be required).
- Just a few auxiliary functions were implemented separately for desktop and browsers. For example, navigation to a particular email in a mailbox. Email loading into the system was implemented using SOAP client built into Rapise.
- One of solved tasks was automation of Outlook for Windows. That part of it one use to open an email in a mailbox and launch the plugin. Windows offers several technologies to automate testing of desktop applications. It is Microsoft Active Accessibility. Technology available on Windows since the end of last century. It was invented to help people with disabilities to use computer. - UI Automation – successor of MSAA, but developed with the idea of automated testing in mind. Maximum freedom is achieved when testing .NET (managed) applications. In this case it is possible to inject into application process and get access to all UI objects and classes. Java Access Bridge – component from Oracle, that makes possible Java testing on Windows. Rapise supports all of the listed technologies. So we had a choice. Outlook is not a Java or .NET application, so the choice was obvious – use UI Automation.
- Now a few details for those who want to try the demo framework or figure out how the tool is implemented.
- Rapise works with objects. Objects have locators, properties and actions. Locators are used for automatic search for elements in an application. Properties and actions allow a tester to ignore low level details and get necessary information about an object. For example, it’s coordinates, text, screenshot. Or interact with an object: expand a node in a tree, choose an item in a list, push a button or enter a text. This approach is implemented for all types of applications: desktop, mobile and web. In the slide heading you see the code generated by Rapise during test recording. The only thing done manually is parameterization of email address. Here ‘Mail_Folders’ – object identifier, locator is bound to it and stored separately. Type of the object is Tree so it has DoClickNode action – it makes possible to click on a specific node in a tree given node path..
- Frequently clients ask that they could use non-programmers for creation and modification of tests. It happens for several reasons. It can be people coming from manual testing. Or analytics, managers, who are experts in a domain (e.g. launching atmospheric probes) but do not write code. In such cases the team may consist of testers-programmers who deal with locators, user action emulation (for them software is mostly a set of forms and elements) and people who implement test cases and work at a higher level of abstraction. For testers of second type Rapise offers capability to create tests using spreadsheets. Examples of such spreadsheets you see on the screen. The first example is calling a scenario (MyLogin) implemented by a programmer. The second example is filling a form (registering a new book in a library).
- It is always interesting to learn from testing project results. Was it possible to reach goals or use gained experience in other projects? So, most significant results from the real project. Not prioritized by importance.
- Analogous to described approach was applied to other applications. A client tested ERP system Dynamics AX. It is a desktop application. Big set of building blocks. One of the blocks was embedded browser – Chromium Embedded Framework. We managed to attach to it as a real Chrome browser and testers worked with its contents as web application, using XPATH.
- Additional benefit. The test coverage, created by the customer, is used for regression testing. This way testers got kudos from developers.
- I think it is the main result. With the help of developed test coverage it became possible to monitor plugin operability. It was fruitful and allowed to catch a series of issues related to updates from Microsoft side. Also the customer caught some patterns in external environment behavior. It enabled customer support service to better interact with users. Example of a pattern – temporary glitches – sometimes plugin activation button just disappear and users can not launch it.
- The test coverage is implemented using single tool. All code of the tests is written in JavaScript. The language which is widespread and well-known. Unified approach is used to work with UI elements on desktop and in browsers.. Configured constant execution of tests, gathering and analysis of telemetry, report building. Such test coverage is easy to support and enhance. Even if tester team is replaced In general there are a lot of ways to reach the final goal. In this testing project our customer could apply other tools. The difference is how fast and qualitatively they could achieve desired result. We consider that Rapise did well in the project. Additional argument played for Rapise was that it is a part of a product family including tools for test planning and remote execution, bug and requirements management. This brought additional benefits to the project. Anyway which tool to use – is your choice.
- Thank you for attention. As I promised here is the link to the page where you can get Rapise for free. The link will be active for two weeks. And link to GitHub repository with demo framework source code. If you want to write me directly – use the email.We still have time for questions and answers. Please go ahead.