Appium Better Practices
Lessons learned the hard way
Matt Isaacs
Gilt
What is Gilt
Short story
Luxe brands up to 60% off
Inventory changes daily.
- Sales last 36 hours.
Sales start at noon ET.
- Bonus sale...
Gilt Mobile
Goals
Understand how users use their devices.
Understand the capabilities of each device.
Understand our users and how the...
A brief tour
How we roll
By hand, with rigorous QA.
Engineers focus on app features + performance + fixes.
QA crew in Jordan takes over...
Seems to be working
"Perfect App !!! Well done"
"Awesome app! Love it!"
"OMG! Amazing!"
"Your app is great and keeps getti...
Or is it..
"Since your update this app runs slower than ever"
"How do you get rid of this. It is like a incurable disease....
It gets worse
Mobile First. So hot right now
Revenue from Mobile at ~40% and growing.
Future of Gilt is in our hands.
Increasingly invol...
Informal Process
Doesn't scale!
Leads to questions:
- Who's doing the build tonight?
- What's in this build?
- Which build...
What does it mean?
Potential for wasted QA time is high.
Expensive resource being used improperly.
Tough for outsiders to ...
How can this be?
Tools have not kept up.
Apple tools lean in favour of the IDE.
xcodebuild test issues.
UIAutomation is pa...
Time for change
Road to automating
Change gradually.
CI - Nightly builds with Jenkins.
Internal distribution. Dogfooding!
Dust off old uni...
What's the big deal with UI tests
Special time saving powers for QA.
Sanity tests are time consuming.
STR can also be time...
Appium
Why?
Philosophy works for us.
Test the app you release
Language/Framework independent.
Uses popular standard API - Seleniu...
Sounds great but...
Sometimes tests work. Sometimes they don't.
Reliability issues widespread enough to prompt development...
The setup at Gilt
Basic set of sanity tests.
- New user registration.
- Existing user sign on.
- Product purchase flow.
- ...
Some lessons learned.
Accessibility
Not just good for your user base.
UIAutomation leverages accessibility to drive your app.
Accessibility = Te...
The special case
Table cells are containers and so need the aforementioned special care.
UIAccessibilityContainer might no...
Product Detail Page Example
PDP Navbar title AccessibilityLabel = Product Name (which is what it should be)
How can a test...
Complex actions.
What if we want to search for unavailable products?
For each sale:
- Wait for the product list and invent...
Patterns
Action - Locator has a couple issues that are more promininent on mobile.
- No state. What page am I on?
- No con...
Final nugget of sage wisdom
Be wary of "clever" UI hacks. Expect testing + accessibility misery.
Pass throughs and overlay...
Appium needs you
Use it.
File those radar issues for Instruments and UIAutomation.
Contribute to Appium! - https://github....
Special Thanks
Sauce for hosting.
Appium community.
Everyone that joined and is still with us.
Thank you
Matt Isaacs
Gilt
http://gilt.com (http://gilt.com)
@gilttech (http://twitter.com/gilttech)
@haveahennessy (http:...
Upcoming SlideShare
Loading in …5
×

Automating Mobile Testing at Gilt with Appium

4,886 views

Published on

See how Gilt Groupe uses Appium to automate their mobile testing. Today, mobile purchases make up more than 40% of Gilt's sales.

Published in: Technology

Automating Mobile Testing at Gilt with Appium

  1. 1. Appium Better Practices Lessons learned the hard way Matt Isaacs Gilt
  2. 2. What is Gilt
  3. 3. Short story Luxe brands up to 60% off Inventory changes daily. - Sales last 36 hours. Sales start at noon ET. - Bonus sales at 9PM, Sundays and Wednesdays.
  4. 4. Gilt Mobile
  5. 5. Goals Understand how users use their devices. Understand the capabilities of each device. Understand our users and how the want to interact with Gilt. Put these all together to create experiences that surprise and delight our users.
  6. 6. A brief tour
  7. 7. How we roll By hand, with rigorous QA. Engineers focus on app features + performance + fixes. QA crew in Jordan takes over at the end of the day. Repeat.
  8. 8. Seems to be working "Perfect App !!! Well done" "Awesome app! Love it!" "OMG! Amazing!" "Your app is great and keeps getting better." "I LOOOOOOOVE YOU GUYS AND YOUR APP!"
  9. 9. Or is it.. "Since your update this app runs slower than ever" "How do you get rid of this. It is like a incurable disease." "Never using again. Sister and mom and friends have removed also. Horrible" Some issues aren't Gilt Mobile's fault. But still..
  10. 10. It gets worse
  11. 11. Mobile First. So hot right now Revenue from Mobile at ~40% and growing. Future of Gilt is in our hands. Increasingly involved with other teams at Gilt. Negative experience becomes more costly.
  12. 12. Informal Process Doesn't scale! Leads to questions: - Who's doing the build tonight? - What's in this build? - Which build must QA test? Unit tests are run at the discretion of today's hand roller. Surprise! Tests never get run.
  13. 13. What does it mean? Potential for wasted QA time is high. Expensive resource being used improperly. Tough for outsiders to work with Gilt Mobile. More shocking: We're not the only ones.
  14. 14. How can this be? Tools have not kept up. Apple tools lean in favour of the IDE. xcodebuild test issues. UIAutomation is part of intsruments. XCode Bots is nice, but not quite ready. Our way is easier.
  15. 15. Time for change
  16. 16. Road to automating Change gradually. CI - Nightly builds with Jenkins. Internal distribution. Dogfooding! Dust off old unit testing efforts. UI Testing
  17. 17. What's the big deal with UI tests Special time saving powers for QA. Sanity tests are time consuming. STR can also be time consumig. UI tests are QA enablers.
  18. 18. Appium
  19. 19. Why? Philosophy works for us. Test the app you release Language/Framework independent. Uses popular standard API - Selenium WebDriver. Thriving open source community. Support for multiple mobile platforms.
  20. 20. Sounds great but... Sometimes tests work. Sometimes they don't. Reliability issues widespread enough to prompt development of Flaky tool. UIAutomation is the root of a lot of the chaos. - "Instruments unexpectedly quit" - Random delays when searching tableviews. Sometimes Apple breaks it altogether - Xcode 5.0.1. Expect some misery on iOS + XCode major releases. One test at a time. Sauce has you covered.
  21. 21. The setup at Gilt Basic set of sanity tests. - New user registration. - Existing user sign on. - Product purchase flow. - Product Waitlist flow. Scala + ScalaTest. Allows us to leverage existing Selenium infrastructure. Why reinvent the wheel? Still a little too brittle.
  22. 22. Some lessons learned.
  23. 23. Accessibility Not just good for your user base. UIAutomation leverages accessibility to drive your app. Accessibility = Testability. Setting AccessibilityLabel and isAccessibilityElement only takes you so far. Container views are troublesome. If marked as accessible, their contents will be masked. UIAccessibilityContainer has you covered.
  24. 24. The special case Table cells are containers and so need the aforementioned special care. UIAccessibilityContainer might not be necessary. Set the accessibility label of the cell, unless you like surprises.
  25. 25. Product Detail Page Example PDP Navbar title AccessibilityLabel = Product Name (which is what it should be) How can a test identify this page? Dynamic Accessibility labels? UIAccessibilityIdentification is for you. "Using an identifier allows you to avoid inappropriately setting or accessing an element’s accessibility label."
  26. 26. Complex actions. What if we want to search for unavailable products? For each sale: - Wait for the product list and inventory to load, - Search elements for "Sold Out" or "Reserved" - Do some string processing to determine which of the products for a given cell is unavailable. Or we could just use our API. Complex navigations and searches with many moving parts can be susceptible to reliability issues.
  27. 27. Patterns Action - Locator has a couple issues that are more promininent on mobile. - No state. What page am I on? - No convetion: When do I check page? Before or after? - Found ourselves fine tuning delays. Page Objects formalize some conventions on top of Actions and Locators. - All actions return page objects - No assertions in page object actions. Never reuse page objects. Element IDs sometimes change. Some page actions dont always return the same page. Use page source to identify page. Since we dont have the luxury of navigating by URL. - Restart the driver - Recovery calls
  28. 28. Final nugget of sage wisdom Be wary of "clever" UI hacks. Expect testing + accessibility misery. Pass throughs and overlays: - Underlying views may be visible by you, but can't be acted upon because they are "hidden". - Tests become an error prone jumble of "mobile: tap" calls.
  29. 29. Appium needs you Use it. File those radar issues for Instruments and UIAutomation. Contribute to Appium! - https://github.com/appium IRC: #appium on freenode
  30. 30. Special Thanks Sauce for hosting. Appium community. Everyone that joined and is still with us.
  31. 31. Thank you Matt Isaacs Gilt http://gilt.com (http://gilt.com) @gilttech (http://twitter.com/gilttech) @haveahennessy (http://twitter.com/haveahennessy)

×