The document describes KKBOX's efforts to develop a smarter monkey testing tool called APE using machine learning. APE uses the TensorFlow Object Detection API to train a model on screenshots labeled with UI elements. It then deploys the trained model on Android and iOS to automatically detect and interact with elements during testing. Some key findings included that training with GPU is faster, active learning helps labeling, F1 scores measure model performance, minimizing labels improves learning, and grayscale training works well. The vision is that APE can help with localization testing by detecting languages and aid design patterns like page objects through element detection. In summary, the document discusses how KKBOX is leveraging machine learning for more intelligent automated testing.

1. Smarter Than Monkey, Meet APE
yihsuanchen@kkbox.com

2. Agenda
● What’s Monkey ?
● What’s APE ?
● What We Learned ?
○ Recommendation
○ Limitation
○ Vision

3. Before Starting

4. Yihsuan Chen
KKBOX
Software Engineer in Test

5. What’s Monkey ?

6. Monkey Test
● Fuzz testing
○ For stability
○ Crash, OOM, ANR
● Advantage
○ Fast
○ Low-cost

7. Monkey Test In KKBOX
● 2016 - 2019
○ 291 / 310 of closed / open issues
● Daily Crash Free Rate
○ 96% in 2016
○ 99.9% in 2019
● Development
○ 5 workdays

8. git@github.com:dino-su/android-monkey-quick-start.git
Dino Su @KKBOX
Android Monkey Test @TestCorner#14

9. ?

10. Limitation
● Hard to reproduce
○ Random script
○ Random location
○ Random event
All things are random!
● Minor issue
A Little Stupid

11. APE
With Object Detection

13. What We Want
🔲 Continuous Integration
🔲 Incremental Datasets
🔲 Speed
🔲 Well Integrity Lifecycle
🔲 Cross-Platform

14. TensorFlow Object Detection API
README.md

17. Setup
● Training
○ Local or Cloud
● TensorFlow
○ CPU or GPU

20. Training Inputs
● Screenshots
● Annotation tool
● Transfer your training dataset to TFRecord file format

23. Training Configs
● Pre-trained model Zoo
● Sample configs
e.g.
● ssd_mobilenet_v2_quantized
● ssd_mobilenet_v2_quantized_*.config

25. Training
● Config, saving place, training steps
$ python object_detection/model_main.py
--pipeline_config_path= ${CONFIG}
--model_dir=${MODEL_SAVING_PLACE}
--num_train_steps= ${TRAINING_STEPS}
--sample_1_of_n_eval_examples=1
--alsologtostderr

26. Tensorboard
● To inspect the progress of training
$ tensorboard --logdir= ${MODEL_SAVING_PLACE}

29. TensorFlow Lite
● Convert checkpoint to TFLite
model.ckpt-20000
$ python object_detection/export_tflite_ssd_graph.py ...
tflite/
detect.tflite

33. git@github.com:tensorflow/examples.git /lite/examples/object_detection/

34. TensorFlow Lite
● Deploy TFLite & labelmap.txt into your project
Android app/src/main/assets/labelmap.txt
app/src/main/assets/detect.tflite
iOS ObjectDetection/Model/labelmap.txt
ObjectDetection/Model/detect.tflite

35. Live Demo
TFLite Object Detector

36. How We Test
● Loop, in a specific event count
○ Screenshot
○ Feed the screenshot to TFLite Object Detection API
○ Get labels’ information (label name, coordinate, confidence)
○ Do actions (click, scroll, keyevent)

37. Live Demo
KKBOX APE

38. A Quick Recap

40. What We Want
✅ Continuous Integration
✅ Incremental Datasets
✅ Speed
🔲 Well Integrity Lifecycle
🔲 Cross-Platform

41. What We Learned ?

42. About ENV

44. About ENV
● Between TensorFlow-CPU & TensorFlow-GPU
tensorflow (Intel® Core™ i7-7700 Processor)
4h 11 min / 25000 steps
tensorflow-gpu (GeForce® GTX 1060 WINDFORCE OC 3G)
1h 12 min / 25000 steps

45. Training with GPU is highly recommended.

46. About Annotation Tool

48. Live Demo
VoTT Active Learning

49. Active Learning is very helpful for labeling.

50. About Measure

53. Deploy To Freeze Model
Convert checkpoint to freeze model (README.md)

54. Inference (Single Image)

55. Inference For Single Image
Detection
● Get a list of labels
● Make an assertion of specific condition

56. Fail
assertTrue('play button' in labels)
Case 1: Necessary

57. Fail
assertEqual(len(labels), 0)
Case 2: Not-Allowed

58. Inference For Single Image
● Advantage
○ Fast
○ Effective
It works, but not that smart.

59. Inference (F1 Measure)

60. F1 Measure
● With a general formula
○ Detection
○ Boxes information of ground truth
○ Confidence score
○ Precision / Recall
○ F1 Score
● F1 Score (Performance indicators)

62. Label F1 Criteria F1 Score F1 Diff
back button 0 0.946 0.946
cancel button 0 0.969 0.969
check box 0 1.0 1.0
cover 0 0.955 0.955
pause button 0 0.882 0.882
play button 0 0.876 0.876
text button 0 0.949 0.949
overall 0 0.939 0.939
F1 Score - 1st Model

63. Label F1 Criteria F1 Score F1 Diff
back button 0.946 0.948 0.02
cancel button 0.969 0.979 0.1
check box 1.0 1.0 0
cover 0.955 0.998 0.043
pause button 0.882 1.0 0.118
play button 0.876 1.0 0.124
text button 0.949 0.949 0
overall 0.939 0.982 0.043
F1 Score - 2nd Model

64. Label F1 Criteria F1 Score F1 Diff
back button 0.948 0.939 -0.009
cancel button 0.979 0.981 0.002
check box 1.0 1.0 0
cover 0.998 0.98 -0.018
pause button 1.0 0.955 -0.045
play button 1.0 1.0 0
text button 0.949 0.956 0.007
overall 0.982 0.973 -0.009
F1 Score - 3rd Model

65. Do performance measurement ASAP.

66. About Label

67. Before
● Labeling guideline
○ Each elements -> Different labels

68. go off air buttoncancel button

69. Before After
allow button
cancel button
confirm button
delete button
deny button
download button
go off air button
ok button

71. Why I'm here ?

72. After
● Labeling guideline
○ Each functional groups -> Different labels

73. Before After
allow button ok button
cancel button cancel button
confirm button ok button
delete button ok button
deny button cancel button
download button ok button
go off air button ok button
ok button ok button

74. ok button

75. Well, not bad

76. Keep minimizing labels
Model is more learnable than we thought.

77. We Lost All Annotations Of All Images

78. Before After
allow button text button
cancel button text button
confirm button text button
delete button text button
deny button text button
download button text button
go off air button text button
ok button text button

79. OMG, it’s really not that bad

80. Keep minimizing labels
Model is “really” more learnable than we thought.

81. About Color

83. About Color
● Color is not that important to Testing
○ Button color
○ Text color
○ Background color
● Take a long shot, we converted the training into grayscale

84. About Color
● Set model config to read image by garyscale
model {
...
image_resizer {
fixed_shape_resizer {
...
convert_to_grayscale : true
}
}
}

85. 15000 steps20000 steps

88. Grayscale training is a good choice.

89. A Quick Recap

91. What We Want
✅ Continuous Integration
✅ Incremental Datasets
✅ Speed
✅ Well Integrity Lifecycle
🔲 Cross-Platform

92. Limitation

93. About Resolution
● Training dataset
○ Android Phone (Portrait mode)
○ 1080p 16:9
● POC
○ Android Tablet (Landscape mode)
○ iOS iPhone

95. 1136 x 6401920 x 1080 1536 x 2048
HTC Nexus 9iPhone SESamsung A7

96. 300 x 300300 x 300 300 x 300
HTC Nexus 9iPhone SESamsung A7

97. 200x200x 200x
HTC Nexus 9iPhone SESamsung A7

98. We are working on it and will update you soon.

99. Vision

100. Vision
● Conclusion
ML is not that far away from us.
● For Mobile
○ Libraries is easy to use
○ Detect speed of TensorFlow Lite is fast
○ As above, and also precise

101. About Localization Testing

103. Localization Testing
● Optical Character Recognition System
● Flow
○ Set language
○ Test & Detect
○ Assert specific language string
登入 KKBOX
Login to KKBOX
ログイン

104. About Page Object

106. iPlayground@youtu.be/iKhC2oCgF4M
Vivian Liu @KKBOX
Design Patterns in XCUITest @iPlayground 2018

107. Page Object
class LoginPage {
init() {
# assertion of this page
expectExist(element: LoginButton)
}
func login() {
type(element: accountTextField)
type(element: passwordTextField)
click(element: loginButton)
}
}

108. Page Object
● Assertion with specific condition
○ isDisplay or isExist
■ On screen ?
■ On hierarchy ?

109. Page Object
● Assertion with object detection
● Truly exist and display on the screen
PlayerPage
assertTrue('play button' in labels)
LoginPage
assertTrue('login button' in labels)

110. Summary

111. APE
● TensorFlow Object Detection API
● Inference
● KKBOX APE with TensorFlow Lite

112. Recommended
● About Env
○ Ubuntu, with Nvidia GPU
● About Annotation
○ VoTT
● Optimization
○ Minimizing labels
○ Grayscale training

113. Vision
● What we thought
○ It’s helpful for Localization Testing
○ It’s helpful for Design Pattern
ML is not that far away from us.

114. Welcome to the ML world !
Let's do some brainstorming !

116. facebook.com/TestingWithKK

117. Thanks