Dont do it in android test automation

DON’T DO IT IN ANDROID
TEST AUTOMATION
ALEKSEI TIURIN, VK

WHO I AM?
LEAD AUTOMATION ENGINEER AT VK
•vk.com/alex_tiurin
•github.com/alex-tiurin
•linkedin.com/in/aleksei-tiurin
2

PREVIOUSLY ON HEISENBUG
РЕШАЕМ ПРОБЛЕМЫ ESPRESSO АВТОТЕСТОВ ANDROID В РЕАЛЬНОМ МИРЕ
4

WHAT ARE WE GONNA TALK ABOUT?
TYPES OF MISTAKES
5

TYPES OF MISTAKES
▸ Test data usage
6

TYPES OF MISTAKES
▸ Test data usage
▸ Test ﬂow
7

TYPES OF MISTAKES
▸ Test data usage
▸ Test ﬂow
▸ Test framework architecture
8

TYPES OF MISTAKES
▸ Test data usage
▸ Test ﬂow
▸ Test framework architecture
▸ Reporting
9

TEST CLASS SKELETON
class SomeTestClass {
...
}
13

TEST CLASS SKELETON
companion object {
@BeforeClass @JvmStatic
fun executeBeforeAllTests(){}
}
...
}
14

TEST CLASS SKELETON
companion object {
}
@Before
fun executeBeforeEachTest(){}
...
}
15

TEST CLASS SKELETON
companion object {
} 
@Before
@Test
fun assertSmthTest(){}
}
16

TEST CLASS SKELETON
companion object {
} 
@get:Rule
val activityRule = ActivityTestRule(MainActivity::class.java)
@Before
@Test
fun assertSmthTest(){}
}
17

TEST DATA USAGE
MARTY, STOP USING TEST DATA EVERYWHERE!!!
19

TEST DATA IS USED EVERYWHERE
object TestData {
val chatId = "..."
val documentId = "..."
}
20

object TestData {
val chatId = "..."
}
// Somewhere in user steps
fun sendDocInChat(user: User){
addDocToUser(user, TestData.documentId)
sendDoc(TestData.chatId,
TestData.documentId)
}
21

object TestData {
val chatId = "..."
}
}
// Somewhere in test class
@Test fun someTest(){
sendDocInChat(user)
}
22

WHAT’S A PROBLEM
23

WHAT’S A PROBLEM
▸ Lost a control on test data
24

WHAT’S A PROBLEM
▸ Unexpected test application condition
25

WHAT’S A PROBLEM
▸ Unexpected test application condition
▸ The same data can be used by several tests at the same time
26

TEST SUITES
TEST CLASSES
TEST STEPSASSERTIONS
TEST ADAPTERS
WHERE TEST DATA SHOULD BE USED? 27

TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
28

Д. БУЗДИН - КАК ПОСТРОИТЬ СВОЙ ФРЕЙМВОРК ДЛЯ АВТОТЕСТОВ?
29

SOLUTION
▸ Restrict the scope of test data deﬁnition to test class / test suite
TEST DATA USAGE 30

object TestData {
val chatId = "..."
}
}
31

RESTRICT THE SCOPE OF TEST DATA DEFINITION TO TEST CLASS / TEST SUITE
object TestData {
val chatId = "..."
}
fun sendDocInChat(user: User, docId: String, chatId: String){
addDocToUser(user, docId)
sendDoc(chatId, docId)
}
32

object TestData {
val chatId = "..."
}
}
33RESTRICT THE SCOPE OF TEST DATA DEFINITION TO TEST CLASS / TEST SUITE

object TestData {
val chatId = "..."
}
}
sendDocInChat(user, documentId, chatId)
}
34RESTRICT THE SCOPE OF TEST DATA DEFINITION TO TEST CLASS / TEST SUITE

SOLUTION
TEST DATA USAGE 35

SOLUTION
▸ Store read-only data as project constants
TEST DATA USAGE 36

STORE READ-ONLY DATA AS PROJECT CONSTANTS
object TestData {
// editable value, should be calculated somehow
val chatId = TestDataManager.createChat()
}
}
37

STORE READ-ONLY DATA AS PROJECT CONSTANTS
object TestData {
// editable value, should be calculated somehow
val chatId = TestDataManager.createChat()
val documentId = TestData.documentId // read-only value
}
}
38

MISTAKE
TEST DATA IS DEFINED AFTER ACTIVITY LAUNCH
39

MISTAKE
@get:Rule
val activityRule =
ActivityTestRule(MainActivity::class.java)
@Before
fun setUp(){
Api.addChatMessage(contact.id, "text")
}
40

TEST CLASS METHODS ORDER
@get:Rule
val activityRule =
@Before
fun setUp(){
}
REASON: @BEFORE EXECUTES AFTER ACTIVITY LAUNCHED
41

@get:Rule
val activityRule =
@Before
fun setUp(){
}
ActivityLaunched
42

@get:Rule
val activityRule =
@Before
fun setUp(){
}
@Before
43
ActivityLaunched

@get:Rule
val activityRule =
@Before
fun setUp(){
}
@Test
44
@Before
ActivityLaunched

@get:Rule
val activityRule =
@Before
fun setUp(){
}
45

WHEN TEST DATA SHOULD BE DEFINED 46

WHEN TEST DATA SHOULD BE DEFINED
DEFINE
TEST DATA
47

LAUNCH
ACTIVITY
DEFINE
TEST DATA
48

LAUNCH
ACTIVITY
DEFINE
TEST DATA @TEST 1
49

LAUNCH
ACTIVITY
DEFINE
TEST DATA @TEST 1
@TEST 2
50

@BeforeClass
@get:Rule
val activityRule =
companion object {
fun setUpTestData(){
}
}
SOLUTION #1: SET UP DATA IN @BEFORECLASS
51

@BeforeClass
@get:Rule
val activityRule =
companion object {
}
}
52

@BeforeClass
@get:Rule
val activityRule =
companion object {
}
}
ActivityLaunched
@Test
53

@BeforeClass
@get:Rule
val activityRule =
companion object {
}
}
ActivityLaunched
@Test
54

@get:Rule
val activityRule =
ActivityTestRule(MainActivity::class.java,
initialTouchMode = false,
launchActivity = false)
@Before
fun setUp(){
activityRule.launchActivity(Intent())
}
SOLUTION #2: LAUNCH ACTIVITY = FALSE
55

@get:Rule
val activityRule =
@Before
fun setUp(){
}
56

@BeforeClass
@get:Rule
val activityRule =
@Before
fun setUp(){
}
57

@BeforeClass
@Before
@get:Rule
val activityRule =
@Before
fun setUp(){
}
58

@BeforeClass
@Before
@get:Rule
val activityRule =
@Before
fun setUp(){
}
59

@BeforeClass
@Before
ActivityLaunched
@get:Rule
val activityRule =
@Before
fun setUp(){
}
60

@BeforeClass
@Before
ActivityLaunched
@Test
@get:Rule
val activityRule =
@Before
fun setUp(){
}
61

SOLUTION #3: CUSTOM SETUP RULE
class SetUpTearDownRule : TestWatcher() {
val setUps = mutableListOf<Condition>()
val tearDowns = mutableListOf<Condition>()
}
62

}
63

}
64

private val setUps = mutableListOf<Condition>()
inner class Condition(
val counter: Int,
val key: String,
val actions: () -> Unit
)
}
65

@get:Rule
val setupRule = SetUpTearDownRule()
.addSetUp {
}
...
}
66

@get:Rule
.addSetUp {
}
...
}
67

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
@BeforeClassSOLUTION #3: CUSTOM SETUP RULE
68

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
@BeforeClass
SetUps.actions
69

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
@BeforeClass
SetUps.actions
ActivityLaunched
70

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
@BeforeClass
SetUps.actions
ActivityLaunched
@Before
71

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
@BeforeClass
SetUps.actions
ActivityLaunched
@Before
@Test
72

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
73

74
SOLUTION #3
DOESN’T WORK
WTF?

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
SOLUTION #3 DOESN’T WORK ALWAYS! WHY?
75

@get:Rule
.addSetUp {
}
@get:Rule
val activityRule =
...
}
SOLUTION #3 DOESN’T WORK ALWAYS! WHY?
76

@get:Rule
val setupRule = SetUpTearDownRule().addSetUp{...}
@get:Rule
@get:Rule
val ruleChain = RuleChain.outerRule(setupRule)
.around(activityRule)
ORDER RULES WITH RULECHAIN
77

@get:Rule
@get:Rule
@get:Rule
78

@get:Rule
79

80
MARTY, ARE YOU REALLY
TELLING ME YOUR PLAN
IS BASED ON
“BACK TO THE FUTURE”?

81
MARTY, ARE YOU REALLY
TELLING ME YOUR PLAN
IS BASED ON
“RULE CHAIN” IN 2020?

@get:Rule
82

@get:Rule
@get:Rule
val ruleSequence = RuleSequence(setupRule, activityRule)
REPLACE RULE CHAIN WITH RULE SEQUENCE
83

ADVANCES OF RULE SEQUENCE
84
▸ Friendly interface

85
▸ Better work with test classes inheritance

86
▸ Better work with test classes inheritance
▸ Three lists of prioritised rules (add, addFirst, addLast)

SOLUTIONS
@BeforeClass
SetUps.actions
@Before
ActivityLaunched
@Test
87

SOLUTIONS
@BeforeClass
SetUps.actions
@Before
ActivityLaunched
@Test
88

SOLUTIONS
▸ Set up test data in @BeforeClass
SetUps.actions
@Before
@BeforeClass
89
ActivityLaunched
@Test

SOLUTIONS
▸ LaunchActivity = false. Set up in @Before
@BeforeClass
SetUps.actions
@Before@Before
90
ActivityLaunched
@Test

SOLUTIONS
▸ Custom SetUpRule + RuleSequence
@BeforeClass
SetUps.actions
@Before
91
ActivityLaunched
@Test

SOLUTIONS
▸ Custom SetUpRule + RuleSequence (best way)
@BeforeClass
SetUps.actions
@Before
92
ActivityLaunched
@Test

MISTAKE
TEST DATA IS DEFINED IN @BEFORECLASS OR @BEFORE METHOD FOR ALL @TEST
93

TEST DATA IS DEFINED IN @BEFORECLASS FOR ALL @TEST
companion object {
Api.clearChatMessages(contact.id)
Api.addChatMessage(contact.id, longMessage)
Api.addChatMessage(contact.id, simpleMessage)
Api.addChatMessage(contact.id, specialCharsMessage)
}
}
}
94

TEST DATA IS DEFINED IN @BEFORE METHOD FOR ALL @TEST
@Before
Api.clearChatMessages(contact.id)
Api.addChatMessage(contact.id, longMessage)
Api.addChatMessage(contact.id, simpleMessage)
Api.addChatMessage(contact.id, specialCharsMessage)
}
}
95

SIMPLE TEXT ASSERTION BECOMES HARDER
@Test
fun assertSimpleMessage() {
ChatPage(contact)
.assertMessageDisplayed(simpleMessage.text)
}
96

SIMPLE TEXT ASSERTION BECOMES HARDER
@Test
ChatPage(contact)
}
97

TEST DATA DEFINITION FOR EACH TEST
SOLUTION: CUSTOM SETUP RULE + @SETUP & @TEARDOWN ANNOTATIONS
.addSetUp { Api.clearChatMessages(contact.id) }
.addSetUp("ADD_SIMPLE") { Api.addChatMessage(contact.id, simpleMessage) }
.addSetUp("ADD_LONG") { Api.addChatMessage(contact.id, longMessage) }
}
100

}
101

}
102

}
103

}
104

}
105

@SetUp("ADD_SIMPLE")
@Test
ChatPage(contact).assertMessageDisplayed(simpleMessage.text)
}
}
106

SIMPLE TEXT ASSERTION BECOMES EASY
@Test
ChatPage(contact)
}
107

...
@Test
fun assertBothMessagesDisplayed() {}
}
108

...
@SetUp("ADD_SIMPLE", "ADD_LONG")
@Test
fun assertBothMessagesDisplayed() {}
}
109

...
@SetUp("ADD_SIMPLE", "ADD_LONG")
@Test
fun assertBothMessagesDisplayed() {
ChatPage(contact)
.assertMessageDisplayed(longMessage.text)
}
110

MISTAKE
TEST DATA IS RELATED WITH BUILD.SERIAL
Tests
runner
server
113

...
val user = TestData.getUser(Build.Serial)
}
114

Tests
runner
server
115

USER 1
Tests
runner
server
116

USER 1 USER 2 USER 3 USER 4 USER 5
Tests
runner
server
117

USER 1 USER 2 USER 3 USER 4 USER 5
Tests
runner
server
118

USER 1 USER 2 USER 3 USER 4 USER 5 USER 6
Tests
runner
server
119

https://openstf.io/
120

IF WE WANNA ADD OR CHANGE A DEVICE
121

▸ Support changing in test code
122

▸ Support changing in test code
▸ Conﬁgure test user to default test conditions
123

Tests
runner
server
124

Tests
runner
server
Test
data
server
125

Tests
runner
server
Test
data
server
126

ANY
USER
Tests
runner
server
Test
data
server
ANY
USER
ANY
USER
ANY
USER
ANY
USER
ANY
USER
127

WHAT TEST DATA SERVER PROVIDES TO US
Test
data
server
128

▸ Pool of test users and any other required objects
Test
data
server
129

▸ Data management via rest api
Test
data
server
130

▸ Data management via rest api
▸ Guarantee of test data uniqueness for separate test
Test
data
server
131

▸ Kotlin
▸ Spring Boot (https://spring.io/projects/spring-boot)
▸ JetBrains Exposed (https://github.com/JetBrains/Exposed)
WHAT TEST DATA SERVER FORMULA
Test
data
server
132

SOLUTION: TEST DATA SERVER
TEST DATA IS RELATED WITH BUILD.SERIAL 133

▸ No need to support devices

▸ Single point to control all test data

▸ Multiple test runs at the same time (CI tasks)

▸ Multiple test runs at the same time (CI tasks)
▸ It could be used by other test frameworks (iOS, Web, api tests)

TEST DATA USAGE
▸ Test data is used everywhere: 
-> Strict data definition to test suite or test case 
-> Store read-only data as constants
138

TEST DATA USAGE
▸ Test data is defined after activity launch 
-> Use custom SetUpTearDown rule + RuleSequence
139

TEST DATA USAGE
▸ Test data is defined in @BeforeClass or @Before method for all @Test
methods 
-> Use SetUpTearDown rule + SetUp and TearDown annotations
140

TEST DATA USAGE
methods 
▸ Test user is related with device Build.Serial 
-> Use TestData server
141

COOL TEST
@Test
fun goodTest(){
FriendsListPage().openChat("Ross Geller")
ChatPage().clearHistory()
.sendMessage("test message")
}
150

IF I WANNA SEND ANOTHER MESSAGE
IS IT A GOOD IDEA?
@Test
fun sendMessageWithLink(){
FriendsListPage().openChat("Ross Geller”)
.sendMessage("message with link vk.com")
}
151

IF I WANNA SEND ANOTHER MESSAGE
ACTUALLY IT ISN’T A GOOD IDEA!
@Test
}
152

MISTAKE
LONG TEST SCENARIO
153

MISTAKE
LONG TEST SCENARIO
▸ It’s a ﬂaky way
154

MISTAKE
LONG TEST SCENARIO
▸ It takes a lot of time
155

MISTAKE
LONG TEST SCENARIO
▸ It takes a lot of time
▸ Code duplication
156

STOP WRITING LONG TESTS
MARTY, LOOK AT THE SHORT TEST!
157

LONG TEST SCENARIO
SOLUTION: SHORT TEST
@Test
}
158

LONG TEST SCENARIO
@Test
ChatPage().sendMessage("message with link vk.com")
}
159

LONG TEST SCENARIO
class ChatTest {
@Test
fun sendMessageWithLink() {
}
}
160

SHORT TEST: LAUNCH TARGET ACTIVITY
class ChatTest {
@get:Rule
val activityRule = ActivityTestRule(MainActivity::class.java, false, false)
}
161

class ChatTest {
@get:Rule
val activityRule = ActivityTestRule(MainActivity::class.java, false, false)
}
162

class ChatTest {
@get:Rule
val activityRule = ActivityTestRule(ChatActivity::class.java, false, false)
}
SHORT TEST: LAUNCH TARGET ACTIVITY 163

class ChatTest {
@get:Rule
@Before
fun prepareData() {
MessageRepository.clearChatMessages(contact.id)
val intent = Intent()
intent.putExtra("INTENT_CONTACT_ID_EXTRA_NAME", contact.id)
activityRule.launchActivity(intent)
}
}
164

class ChatTest {
@get:Rule
@Before
fun prepareData() {
}
}
165

class ChatTest {
@get:Rule
@Before
fun prepareData() {
}
}
166

class ChatTest {
@get:Rule
@Before
fun prepareData() {
}
}
167

class ChatTest {
@get:Rule
@Before
fun prepareData() {
}
}
168

class ChatTest {
@get:Rule
@Before
fun prepareData() {
}
@Test
}
}
169

class ChatTest {
@get:Rule
@Before
fun prepareData() {
}
@Test
fun sendMessageWithLink() {...}
}
170

class ChatTest {
@get:Rule
.addSetUp {
}
@Test
}
171

class ChatTest {
@get:Rule
.addSetUp {
}
@Test
}
172

class ChatTest {
.addSetUp {
}
@get:Rule
@Test
}
173

SHORT TEST: LAUNCH TARGET ACTIVITY 175

LONG TEST SCENARIO
class ChatTest {
@Test
}
}
178

BLACKLIST CASE
WHAT CAN I DO WITH FRAGMENTS?
179

ANDROID FRAGMENTS 182
SettingsActivity

SettingsActivity
SettingsListFragment

SettingsActivity
BlacklistFragmentSettingsListFragment

SAME ACTIVITY BUT DIFFERENT FRAGMENTS 185

BLACKLIST DATA PREPARATION
class BlacklistTests {
...
@get:Rule
}
186

...
@get:Rule
@Before
fun openFragment(){
activityRule.runOnUiThread {
SettingsFragmentNavigator.go(BlacklistFragment::class.java)
}
}
}
187

...
@get:Rule
@Before
fun openFragment(){
}
}
}
188

...
@get:Rule
@Before
fun openFragment(){
}
}
}
189

...
@get:Rule
.addLast(SetUpTearDownRule().addSetUp {
}
})
}
190

val activityRule = ActivityTestRule(SettingsActivity::class.java)
val setUpTearDownRule = SetUpTearDownRule()
.addSetUp {
ContactsRepositoty.clearBlacklist()
ContactsRepositoty.addToBlacklist(contact.id)
}
@get:Rule
}
198

.addSetUp {
}
@get:Rule
}
199

.addSetUp {
}
@get:Rule
}
200

SOLUTION: USE INTERNAL NAVIGATION MECHANISM
FRAGMENTS NAVIGATION PROBLEM 205
}

LOGIN
WHAT CAN I DO WITH LOGIN PROBLEM?
206

LOGIN
▸ Use authorisation via UI
207

LOGIN
▸ Use authorisation via UI
▸ Use internal authorisation mechanism
208

MISTAKE
USE AUTHORISATION VIA UI
209

MISTAKE
210

USE INTERNAL AUTHORISATION MECHANISM
AccountManager()
.login(
user.login,
user.password
)
211

INTERNAL AUTHORISATION MECHANISM
class ChatTest {
...
.addSetUp {
}
@Test
fun sendMessageWithLink() {... }
}
212

class ChatTest {
...
.addSetUp {
...
}
@Test
}
213

class ChatTest {
…
.addSetUp {
AccountManager(targetContext).login(user.login, user.password)
...
}
@Test
}
214

SOLUTION: USE INTERNAL AUTHORISATION MECHANISM
LOGIN PROBLEM 215
AccountManager(targetContext).login(user.login, user.password)

TEST FLOW
I HAD THE LONGEST TEST EVER BUT NOW …
216

TEST FLOW
▸ Long test scenario
217

TEST FLOW
▸ Long test scenario  
-> Use short tests by opening target activity
218

TEST FLOW
▸ I navigate my tests to the Target Screen by UI actions  
219

TEST FLOW
-> Use internal navigation mechanism
220

TEST FLOW
▸ I navigate my tests to the Target Screen by UI actions 
▸ Login by UI actions  
221

TEST FLOW
-> Do it by internal authorisation mechanism
222

TEST FLOW
223
I HAD THE LONGEST TEST EVER BUT NOW I HAVE THE SHORTEST ONE

TEST FRAMEWORK ARCHITECTURE 224

TEST FRAMEWORK ARCHITECTURE
MY FIRST TEST FRAMEWORK
225

TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
227

TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
228

TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
STEPS
229

TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
APP UI
ELEMENTS
STEPS
230

TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
APP UI
ELEMENTS
STEPS
PAGE OBJECT CLASS
231

MY FIRST TEST FRAMEWORK
232

MY CURRENT TEST FRAMEWORK
233

REPORTING
IT WILL LOOKS LIKE THIS
235

REPORTING
WHY IT HAPPENS?
▸ Support hell
239

REPORTING
WHY IT HAPPENS?
▸ Support hell
▸ Integration with CI hell
240

REPORTING
SOLUTION: USE ALREADY MADE LIBS & TOOLS
241

REPORTING
▸ Allure android
242

REPORTING
▸ Allure android
▸ Test runner report
243

REPORTING
▸ Allure android
▸ Another already made report
244

TYPES OF MISTAKES WE’VE DISCOVERED
BACK TO THE PAST 245

BACK TO THE PAST
▸ Test data usage
246

TEST DATA USAGE
-> Use custom SetUpTearDown rule
methods 
▸ Test user is related with device Build.Serial 
-> Use TestData server
247

BACK TO THE PAST
▸ Test data usage
▸ Test ﬂow
248

TEST FLOW
249
I HAD THE LONGEST TEST EVER BUT NOW I HAVE THE SHORTEST ONE

BACK TO THE PAST
▸ Test data usage
▸ Test ﬂow
▸ Framework architecture
250

251SOLUTION: USE COMMON PRACTICES AND ADAPT IT
TEST SUITES
TEST CLASSES
TEST DATA
TEST ADAPTERS
APP UI
ELEMENTS
STEPS
PAGE OBJECT CLASS

BACK TO THE PAST
▸ Test data usage
▸ Test ﬂow
▸ Framework architecture
▸ Reporting
252

REPORTING
▸ Allure android
▸ Another already made report
253

GitHub:
1) Примеры кода
github.com/alex-tiurin/espresso-guide
2) Библиотека c Espresso DSL
github.com/alex-tiurin/espresso-page-object
Как подключить себе в проект?
repositories { jcenter() }
dependencies {
//espresso-page-object
androidTestImplementation 'com.atiurin.espresso:espressopageobject:0.1.16'
}

Dont do it in android test automation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Dont do it in android test automation

Similar to Dont do it in android test automation (20)

Recently uploaded

Recently uploaded (20)

Dont do it in android test automation