CodeFest 2013. François Reynaud. — Tools for mobile automation are nothing special
•Download as PPTX, PDF•
2 likes•1,062 views
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to CodeFest 2013. François Reynaud. — Tools for mobile automation are nothing special
Similar to CodeFest 2013. François Reynaud. — Tools for mobile automation are nothing special (20)
More from CodeFest
More from CodeFest (20)
Recently uploaded
Recently uploaded (20)
CodeFest 2013. François Reynaud. — Tools for mobile automation are nothing special
- 1. IOS UI Automation François Reynaud Test engineer, eBay
- 2. Agenda • Introduction • Selenium • Native • Hybrid • Real device • Scaling François Reynaud- EBAY INTERNATIONAL 2
- 3. Introduction •EUQE •Selenium committer •History •Current focus François Reynaud- EBAY INTERNATIONAL 3
- 4. Selenium • What selenium is about – Browser automation tool – Clean API, draft for the W3C • Why Selenium for mobile testing ? – API works for mobile – API stability makes development easy – Reusability François Reynaud- EBAY INTERNATIONAL 4
- 5. Goal Cut down regression time for the eBay IOS products. •4 Native apps •Hybrid apps and mobile web •reuse existing tools if possible •open source ( = no eBay specific features ) François Reynaud- EBAY INTERNATIONAL 5
- 6. The project Status : version 0.6 Visible on github •https://github.com/ios-driver/ios-driver Documentation •http://ios-driver.github.com/ios-driver/ Getting help •IRC : #ios-driver on freenode •Google group : ios-driver François Reynaud- EBAY INTERNATIONAL 6
- 7. The team François Reynaud- EBAY INTERNATIONAL 7 Luke Inman Semerau IPhone driver Graham Abell NativeDriver fork François Reynaud Grid2, TWIN
- 8. Technical choices - Selenium – No modification of the application – Language / test framework shouldn’t matter – A stable API / recognized is the key François Reynaud- EBAY INTERNATIONAL 8
- 9. Native : Instruments François Reynaud- EBAY INTERNATIONAL 9
- 10. Instruments – integration capabilities François Reynaud- EBAY INTERNATIONAL 10
- 11. Design François Reynaud- EBAY INTERNATIONAL 11 Client Server WebDriver API ios-driver Java C# Ruby Python
- 12. Writing tests François Reynaud- EBAY INTERNATIONAL 12
- 13. Challenge 1 - content François Reynaud- EBAY INTERNATIONAL 13
- 14. Challenge 1 - content François Reynaud- EBAY INTERNATIONAL 14
- 15. Challenge 1 - content François Reynaud- EBAY INTERNATIONAL 15
- 16. Challenge 2 - inspector François Reynaud- EBAY INTERNATIONAL 16
- 17. Hybrid - UIWebView 17François Reynaud- EBAY INTERNATIONAL
- 18. WebKit Remote Debug Protocol 18François Reynaud- EBAY INTERNATIONAL
- 19. integration 19François Reynaud- EBAY INTERNATIONAL
- 20. Safari 20François Reynaud- EBAY INTERNATIONAL
- 21. Real device Not very different : – Instruments works on real device – WKRDP works on real device Limitation : – Cannot change global settings – Need to handle app signing François Reynaud- EBAY INTERNATIONAL 21
- 22. Scaling using Selenium Grid François Reynaud- EBAY INTERNATIONAL 22 Client serverHubClient CI server server
Editor's Notes
- Me:Joined eBay 8 years ago.1) part of the EUQE =European Quality engineering team. Small team :About 15 people testing everything that impacts the 12 european sites.Large scope :12 web sites, windows desktop application ( Turbo Lister ) and Mobile-> no in house tools. Use open source if possible, and create and open source all tools created.2) Selenium committer, project lead for Selenium Grid.3) History :Web test : done with seleniumWindows desktop test : done with TWIN ( open sourced : https://code.google.com/p/twin/ ) Scalability / architecture : done with selenium grid ( part of the selenium project )4) Current focusMobile : current focus. I’m focusing on IOS. DominikDary focuses on Android.Goal is to integrate with what we already have
- Selenium is at the core of all our automation.What is selenium :a browser automation tool ( not a test framework )Clean design that allowTo use any language / any test framework to write the testsRun the tests in parallel against all major browsersX-browser , super stable API : tests written today for firefox will run tomorrow on the next chrome versionBecoming a W3C standard – no going away any time soon. Why selenium for Mobile :selenium’s strength is its API that describes user interactions.Those users interactions can be applied directly to mobile. A subset of it, some interaction do not exist in the mobile world : mouse move, mouse overMore commands can be added using the vendor extension section of the selenium protocolHaving this stable API allows to write tests while working on the driver. When bugs are fixed, and features added, there is no need to refactor the tests already written. Reusability :By following the same design for mobile, we can significantly reduce the effort it takes to write tests. The same design and abstraction can be applied to mobile, and all the mocks and helper classes can be reused. For eBay, starting mobile automation in a selenium envt means 50% of the work is already done ( create users, items, DB access )
- Business goal :1) eBay releases 4 native apps for IOSThose apps have to be tested by the EU team already testing Web and windows desktop.The goal of our automation is NOT to automate everything, but rather to :Get rid of the repetitive tasks Free resources to have more time for exploratory testingA typical task we don’t do with any of our automation is layout validation. We only validate functionalities.2) Mobile web is also a big focus. Long term plan :eBay isn’t a framework company. We can’t afford dedicating resources on framework maintenance.So we reuse what we can, and open source to have the testing community help with the maintenance.This has worked perfectly so far.
- Version 0.6Main functionality are working and tests can be writtenbut there is still a lot of polishing to doVisible on github. Contributors welcome. The project is in C, Java and Javascript.Best way to get interactive help : IRC
- Open source project. Does belong to a single company. Several experienced people already :2 selenium committersCurrent maintainer of the NativeDriver community fork
- I’ve been a user and contributor for Selenium, and I believe the selenium approach to testing is the correct one :1) The application under tests shouldn’t have to be modified to be tested2) A browser automation / mobile automation framework shouldn’t force a user to choose a testing framework , or testing methodology.It’s not the role of the mobile framework to decide vsTestNG or Junit in the java world.It’s “ to decide is BDD should be used or not.The key is integration. Most likely users will already have something in place when they start mobile automation.3) Rewriting all the tests each time the implementation of the framework changes isn’t an option.
- First part I worked on was Native app.We are in an IOS world, so looking at Apple tools is a good start.For Native apps, Apple provides the Instruments tools - great tools - does all the heavy lifting : takes the application, and allow to inspect it and generate user interaction - integrated in the apple tool suite - language is JS : coming from a Web world,it makes it easy.Some drawback - integration with non apple suite ( anything not part of the Apple UI really ) isn’t easy - browsing the native object tree isn’t pleasant. A nicer API is needed.
- Thankfully, instruments comes with 2 handy features :Instruments can be started without the UIInstruments scripts can communicates with other processes using performTask.That allows to build a framework that leverages all the good features of Instruments :No modification of the appApple support, so new SDK recognized when launchedAnd can improve the not so good parts of it, making it as easy to use as Selenium
- Green boxes are what a tester will see. That code isn’t part of ios-driver, it’s taken as is from the Selenium project.The part ios-driver is about id the server side, the part wrapping instruments, and controlling the device / simulator ( setup, teardown of apps )Allow remote executionMap the instruments command to the selenium ones.
- Writing a ios mobile test becomes easy :No particular setup needed, just a mac with Xcode in the path and ios-server runningAny test Framework can be used.It uses plain selenium client. Example is in Java, but all client bindings will work.
- Main challenge for multi language apps is content.There is no ID like for web testing, and all selectors are heavily relying on content.The screenshot is an example of the International mountain app, the l10n example from apple.The app functionality is the same for all locale, but you need the content to target the elements.
- Fortunately the content is available in the app.Each piece of content is generated from a key ( assuming the app content isn’t hardcoded ).Ios-driver uses that and allow tests to pass the content key as a selector, translating it on the fly based on the locale.
- -That is what the selector becomes.- It’s not as easy as using an ID for a regular webpage, but it’s close enoughIt only help validating the functionality of the app, it assumes the translation in the app files are correct.To validate the translation itself, a different mechanism should be used.
- The 2nd challenge is the lack of inspector.When working on a webpage, I always need firebug on the chrome console opened to see what happens. We need something similar for ios.+ demo here if time.https://dl.dropbox.com/u/24687868/presentations/inspector_record.mov
- When all that is done, the native part of the ios apps can be automated.This is only half if what needs to be done to have a fully working IOS solution.Hybrid apps ( native apps with some html content cannot be properly automated using native API )Mobile web : same issue ( mobile safari is an hybrid app where 99% of the work happens in the webviews )
- Since IOS 6.0, there is a solution to that problem : we can use the WKRDP ( webkit remote debug protocol )If your device ( simulator, or USB connected real device ) has a webview, the webview can be access ( read + write ) using the debug protocol.Demo : https://dl.dropbox.com/u/24687868/presentations/wkrdp.mov
- Instruments and WKRDP are completely separate, and can be run in parallel. So what’s left isto build a 2nd driver in ios-driver that implement all the selenium command but for webviews this time.Allows to switch from native to webview, this is done with the switch window selenium APIThat allows :- native clicks and keypress for webview ( Demo if time here )Good web selector ( by id, css selector ) on hybrid pages. navigation between frames / windows.
- With hybrid support, safari support comes for free.Demo multi window, demo type.https://dl.dropbox.com/u/24687868/presentations/SafariNative.mov
- Most of development and tests are using the simulator, because it’s easier and cheaper.However, everything also works on real device.Working with real device has some challenges though :Fixed SDK versionFixed deviceFixed locale / languageNo geolocation changesExtra work managing certificatesSlow
- - UI tests are slow- UI IOS tests require real device etcRunning them in parallel and managing the infrastructure is even more critical than for web automation.Using Selenium grid 2 will make a huge difference for tester.https://dl.dropbox.com/u/24687868/demo.mov