Successfully reported this slideshow.
Your SlideShare is downloading. ×

Rock solid UI Test Droidcon Berlin 2021

Upcoming SlideShare
Jessebrackenburymasccc2014
Jessebrackenburymasccc2014
Loading in …3
×

Check these out next

1 of 68 Ad
1 of 68 Ad

Rock solid UI Test Droidcon Berlin 2021

We Android developers use the "flakiness" word more than the "stability" word while talking about Espresso and Instrumentation tests. We cannot truly trust flaky tests. Instrumentation testing will continue to be part of our lives even with Jetpack Compose therefore, we need stability.

Running all UI tests before merging any code is possible, and it saves us a lot with 400+ integration tests. Unfortunately, stability is not something that comes out of the box. A certain level of architecture, proper thread usage, and test setup with helper rules are required. There is an uncountable number of things that affect UI tests.

Good thing, there are explanations and solutions for most of the flakiness. In this talk, you will learn more about best practices to increase test stability, how to utilise Test-Driven Development with UI testing, and how this will help you find bugs earlier.

Sinan Kozak

We Android developers use the "flakiness" word more than the "stability" word while talking about Espresso and Instrumentation tests. We cannot truly trust flaky tests. Instrumentation testing will continue to be part of our lives even with Jetpack Compose therefore, we need stability.

Running all UI tests before merging any code is possible, and it saves us a lot with 400+ integration tests. Unfortunately, stability is not something that comes out of the box. A certain level of architecture, proper thread usage, and test setup with helper rules are required. There is an uncountable number of things that affect UI tests.

Good thing, there are explanations and solutions for most of the flakiness. In this talk, you will learn more about best practices to increase test stability, how to utilise Test-Driven Development with UI testing, and how this will help you find bugs earlier.

Sinan Kozak

Advertisement
Advertisement

More Related Content

Slideshows for you (19)

Advertisement

Rock solid UI Test Droidcon Berlin 2021

  1. 1. •
  2. 2. • •
  3. 3. • • •
  4. 4. o o • • • o o o
  5. 5. • • •
  6. 6. • • •
  7. 7. • • • •
  8. 8. • •

Editor's Notes

  • Every one is frustrated byy flaky tests
  • Who stop using android test because off flakiness
  • 800 tests run on every commit under 20m

    PR wont merge without succesful AndroidTest

    18 Different features has their Android test in their module

    We dont depend on UI test. Only most important features are tests end to end

    Almost all main usacases covered
  • User interface (UI) testing lets you ensure that your app meets its functional requirements and achieves a high standard of quality such that it is more likely to be successfully adopted by users.



  • One approach to UI testing is to simply have a human tester perform a set of user operations on the target app and verify that it is behaving correctly
    Time-consuming, tedious, and error-prone



  • The automated approach allows you to run your tests quickly and reliably in a repeatable manner.

    androidTest folder

    Creates apk to run on device
  • UI interaction
    End to End flow of feature
    Device resources
  • Pure java kotlin classes can be tested on jvm
    They are faster
    We should aim to have business logics in class we can test
  • In reality we wont be able to test everything on jvm
    User interface
    Device resource usage
    End to end verification from user persfective
  • Who stop using android test because off flakiness
  • There is no out of box solution

    Device control. We want to focus on WHAT to test
    Delegate device control as much as possible

    Threading issues
    Framework issues
  • APIs to write AndroidTests.
    Dont uses your classes
    Use app from user persfective
  • Espresso focuses on how to find and how check views

    Espresso is aware of multiple things
  • Espresso is low APIs focus on HOW to test
  • As developer we want to focus on WHAT to test

    Espresso is API around actions and espresso is has HOW to find views or HOW to check conditions

  • 10 minutes
  • How we can reduce learning curve
    Kakao is a library around Espresso APIs.
  • The goal of the Kakao is to provide better readability and discoverability for Espresso testing.

    Kakao doesn’t aim to change the behaviour of Espresso.

    By default, Kakao enforces the same mentality of robot pattern. Kakao expects a Screen class. And in Screen class, you can define common UI elements or actions using Kotlin DSL. 

  • Kakao has solutions for common UI element
    Easy to discover
    Type safety about actions and check
    Easy to read tests


    But this is not about stability
  • Lets talk about most common usecases

    1st animation
  • Because Espresso isn't aware of any other asynchronous operations, including those running on a background thread, Espresso can't provide its synchronization guarantees in those situations. In order to make Espresso aware of your app's long-running operations, you must register each one as an idling resource.
  • For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.


    It would be great if we can have one idling resource for threading solution we have.

    And actually we have some
  • 20 minutes
  • For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
  • For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
  • There is not a clean solution
    We can use similar provider solution as ExecutorProvider
    It is not possible to prevent libraries to use Default dispatcher

    SetDefault vs setMain

    With rx all of background operations covered

    It feels like we are not goiing forward but keep craeting new problems
  • If you remember the usecases for espresso maybe we can find a different solution.
    Robost solution

  • If you remember the usecases for espresso maybe we can find a different solution.
    Robost solution

  • If we cannot control concurency framework, we can track background operations directly
  • For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
    Create an interceptor and register idling resources with the information you have. In interceptor you can track current network operations.

  • After the network operation, you set the data to live data.

    Everything will be good, right?
    No
  • We set a countiing executor for arch component.

    DataBinding will use value of livedata and there wont be a flakiness, right?
    No
  • There is no easy way to keep track of when data binding applies changes to screen.
    There is only hasPendingIntent information

    Simple solution is not using databindiing

    But if yyou have databindiing, yyou need to regiister and idliing resource and check all the views.

    All theeee views
  • Network, livedata and databinding wont be a problem.

    We can use ListAdapter and submit a new list and everything will allright?

    You probably already understand the theme of this flow.

    No
  • We started a data flow network layer until data reaches to screen, we needed 4 solution.

    We will definitely need moree solutions

    That is why it is nice to have a single place to share solutions

    30 minutes
  • There are different options to run UI tests. 
    You can create your own test farm in the company with real devices. 
    You can use Github actions and run an emulator on Github action. 
    You can use other build frameworks like Jenkins and set up your own emulator on CI. 
    All of those options will need maintenance and they will be bound to your investment of money and time. 
  • Run on google infrastructure
    All of the devices on Firebase Test Lab have special adjustments
    We prefer to use virtual devices as there is generally no queue for them

    They are also active in Firebase slack.
    You can ask questions or report issues. And from my own experience, they help individual cases.
  • One cool feature of Firebase test lab is sharding

    Split tests into buckets and run them on different devices to parallelize execution

    Cost will be same

    Duration will depend on how small the test buckets are

  • Creating test matrices that should finish under desired time is a difficult problem.
    Luckily we don’t do that setup by hand.
  • We need gradle integration for Flank
    And Fladle is gradle wrapper
  • We have pipeline
    Cover more usecases at scout

    Good improvements coming to android testing


  • Espresso is great library.
    I always feel that espresso is just starting point like any other coffees.

    They provided base element so we can add milk or water and use it

×