Gilt developed their mobile app using Appium for automated testing. They learned that accessibility is important for both users and tests. Page object patterns help organize tests, and tests should check page state before actions. Complex user flows are better tested through the API than UI automation. Clever UI hacks often cause testing and accessibility issues. The community contributes greatly to Appium's continued improvement.

1. Appium Better Practices
Lessons learned the hard way
Matt Isaacs
Gilt

2. What is Gilt

3. Short story
Luxe brands up to 60% off
Inventory changes daily.
- Sales last 36 hours.
Sales start at noon ET.
- Bonus sales at 9PM, Sundays and Wednesdays.

4. Gilt Mobile

5. Goals
Understand how users use their devices.
Understand the capabilities of each device.
Understand our users and how the want to interact with Gilt.
Put these all together to create experiences that surprise and delight our users.

6. A brief tour

7. How we roll
By hand, with rigorous QA.
Engineers focus on app features + performance + fixes.
QA crew in Jordan takes over at the end of the day.
Repeat.

8. Seems to be working
"Perfect App !!! Well done"
"Awesome app! Love it!"
"OMG! Amazing!"
"Your app is great and keeps getting better."
"I LOOOOOOOVE YOU GUYS AND YOUR APP!"

9. Or is it..
"Since your update this app runs slower than ever"
"How do you get rid of this. It is like a incurable disease."
"Never using again. Sister and mom and friends have removed also. Horrible"
Some issues aren't Gilt Mobile's fault. But still..

10. It gets worse

11. Mobile First. So hot right now
Revenue from Mobile at ~40% and growing.
Future of Gilt is in our hands.
Increasingly involved with other teams at Gilt.
Negative experience becomes more costly.

12. Informal Process
Doesn't scale!
Leads to questions:
- Who's doing the build tonight?
- What's in this build?
- Which build must QA test?
Unit tests are run at the discretion of today's hand roller.
Surprise! Tests never get run.

13. What does it mean?
Potential for wasted QA time is high.
Expensive resource being used improperly.
Tough for outsiders to work with Gilt Mobile.
More shocking: We're not the only ones.

14. How can this be?
Tools have not kept up.
Apple tools lean in favour of the IDE.
xcodebuild test issues.
UIAutomation is part of intsruments.
XCode Bots is nice, but not quite ready.
Our way is easier.

15. Time for change

16. Road to automating
Change gradually.
CI - Nightly builds with Jenkins.
Internal distribution. Dogfooding!
Dust off old unit testing efforts.
UI Testing

17. What's the big deal with UI tests
Special time saving powers for QA.
Sanity tests are time consuming.
STR can also be time consumig.
UI tests are QA enablers.

18. Appium

19. Why?
Philosophy works for us.
Test the app you release
Language/Framework independent.
Uses popular standard API - Selenium WebDriver.
Thriving open source community.
Support for multiple mobile platforms.

20. Sounds great but...
Sometimes tests work. Sometimes they don't.
Reliability issues widespread enough to prompt development of Flaky tool.
UIAutomation is the root of a lot of the chaos.
- "Instruments unexpectedly quit"
- Random delays when searching tableviews.
Sometimes Apple breaks it altogether - Xcode 5.0.1.
Expect some misery on iOS + XCode major releases.
One test at a time. Sauce has you covered.

21. The setup at Gilt
Basic set of sanity tests.
- New user registration.
- Existing user sign on.
- Product purchase flow.
- Product Waitlist flow.
Scala + ScalaTest. Allows us to leverage existing Selenium infrastructure. Why reinvent
the wheel?
Still a little too brittle.

22. Some lessons learned.

23. Accessibility
Not just good for your user base.
UIAutomation leverages accessibility to drive your app.
Accessibility = Testability.
Setting AccessibilityLabel and isAccessibilityElement only takes you so far.
Container views are troublesome. If marked as accessible, their contents will be
masked. UIAccessibilityContainer has you covered.

24. The special case
Table cells are containers and so need the aforementioned special care.
UIAccessibilityContainer might not be necessary.
Set the accessibility label of the cell, unless you like surprises.

25. Product Detail Page Example
PDP Navbar title AccessibilityLabel = Product Name (which is what it should be)
How can a test identify this page?
Dynamic Accessibility labels? UIAccessibilityIdentification is for you.
"Using an identifier allows you to avoid inappropriately setting or accessing an
element’s accessibility label."

26. Complex actions.
What if we want to search for unavailable products?
For each sale:
- Wait for the product list and inventory to load,
- Search elements for "Sold Out" or "Reserved"
- Do some string processing to determine which of the products for a given cell is
unavailable.
Or we could just use our API.
Complex navigations and searches with many moving parts can be susceptible to
reliability issues.

27. Patterns
Action - Locator has a couple issues that are more promininent on mobile.
- No state. What page am I on?
- No convetion: When do I check page? Before or after?
- Found ourselves fine tuning delays.
Page Objects formalize some conventions on top of Actions and Locators.
- All actions return page objects
- No assertions in page object actions.
Never reuse page objects. Element IDs sometimes change.
Some page actions dont always return the same page. Use page source to identify
page.
Since we dont have the luxury of navigating by URL.
- Restart the driver
- Recovery calls

28. Final nugget of sage wisdom
Be wary of "clever" UI hacks. Expect testing + accessibility misery.
Pass throughs and overlays:
- Underlying views may be visible by you, but can't be acted upon because they are
"hidden".
- Tests become an error prone jumble of "mobile: tap" calls.

29. Appium needs you
Use it.
File those radar issues for Instruments and UIAutomation.
Contribute to Appium! - https://github.com/appium
IRC: #appium on freenode

30. Special Thanks
Sauce for hosting.
Appium community.
Everyone that joined and is still with us.

31. Thank you
Matt Isaacs
Gilt
http://gilt.com (http://gilt.com)
@gilttech (http://twitter.com/gilttech)
@haveahennessy (http://twitter.com/haveahennessy)