Recommended
More Related Content
What's hot
What's hot (20)
Similar to Test Automation Improvement by Machine Learning Jasst'21 Tokyo
Similar to Test Automation Improvement by Machine Learning Jasst'21 Tokyo (20)
More from Sadaaki Emura
More from Sadaaki Emura (14)
Recently uploaded
Recently uploaded (20)
Test Automation Improvement by Machine Learning Jasst'21 Tokyo
- 1. Test Automation Improvement by Machine Learning March 15th , 2021 Emura Sadaaki Leisure Product Dept. Rakuten, Inc. JaSST’21Tokyo
- 2. 2 This presentation is • Test automation has daily operation and issue • Issue is resolved by system with machine learning
- 3. 3 Agenda 1 2 3 4 5 Background Test automation operation issue Investigate issue Improve operation Conclusion
- 4. 4 Agenda 1 2 3 4 5 Background Test automation operation issue Investigate issue Improve operation Conclusion
- 5. 5 Organization Developer group QA group Manual test team Test automation team Me : 11 services Product manager engineer https://www.irasutoya.com/
- 7. 7 Test automation responsibility 1. Test automation covers regression 2. Update regression script and create new script during project 3. Do regression test in test environment everyday
- 8. 8 Test automation covers regression Product Project scope Regression Cover by manual test Cover by test automation When project start. • Manual test focus on project scope • Test automation focus on all regression Responsibility 1
- 9. 9 ① ② Update regression script and create new script during project Area ① Change specification Script modify to cover last specification Area ② New function Create new script to add to existing regression Product Project scope Regression Cover by manual test Cover by test automation Responsibility 2
- 10. 10 Regression test in test environment everyday All regression test is scheduled everyday Objective Find bug Find environment issue Find release operation rehearsal issue Find non-functional bug : Responsibility 3
- 11. 11 Test automation responsibility many operations 1. Test automation covers regression 2. Update regression script and create new script during project 3. Do regression test in test environment everyday
- 12. 12 Agenda 1 2 3 4 5 Background Test automation operation issue Investigate issue Improve operation Conclusion
- 13. 13 What is test automation operation? 1. Check test automation results 2. Investigate test result of test automation failed 3. Handle test automation results All regression test is scheduled everyday operation Responsibility 3
- 14. 14 Check test automation results • 20 Jenkins servers are active • Around 2,000 test exist • Check summary in dashboard • If test failed, get test result from dashboard dashboard Operation 1
- 15. 15 Investigate test result of test automation failed ~ 1. Application bug 2. Test automation script bug 3. Environment issue 4. Temporary unstable Investigate failed reason by test result Operation 2 Failed reason category
- 16. 16 Handle test automation results 1. Application bug 2. Test automation script bug 3. Environment issue 4. Temporary unstable Investigate failed reason by test result Fix issue Operation 3
- 17. 17 Fix application bug Failed reason issue = “Application bug” 1. Report bug to engineers 2. Fix bug by engineers 3. Check application bug fixed Bug report Bug fix Operation 3
- 18. 18 Fix test automation script bug Failed reason issue = “Test automation script bug” 1. Check specification 2. Fix bug by myself 3. Check test automation work well Operation 3
- 19. 19 Fix issue after environment is ready Failed reason issue = “Environment issue” 1. During maintenance , stop test automation 2. Retry test automation after maintenance is finished 3. Manage scheduler Operation 3
- 20. 20 Fix issue to retry test automation Failed reason issue = “Temporary unstable” 1. Retry test automation Website is busy Non productive & take much time! Operation 3
- 21. 21 Test automation operation issue Temporary unstable issue operation is not productive and take much time Test automation has some operations
- 22. 22 Agenda 1 2 3 4 5 Background Test automation operation issue Investigate issue Improve operation Conclusion
- 23. 23 Investigate issue in detail Verify this fact and Consider improvement Temporary unstable issue operation is not productive and take much time Hypothesis
- 24. 24 Verify current issue trend ・・・ Jenkins server dashboard 1. Collect test results from Jenkins 2. classify failed reason category Test result management system ① ②
- 25. 25 classify failed reason category Failed reason When failed, investigate reason by test result and screenshots, movie. Finally classify failed reason category. 〇 Application Bug 〇Test automation script Bug 〇 Environment issue 〇Temporary unstable
- 26. 26 Summary failed reason category 0.5 4.9 20.4 74.1 Application Bug Test automation Bug Environment issue Temporary unstable 1. Application bug 2. Test automation script bug 3. Environment issue 4. Temporary unstable ※2020/1 ~ 2020/10 summary Unit = percentage Reason category Reason ratio
- 27. 27 Operation time needed for “temporary unstable” issue 0.5 4.9 20.4 74.1 Application Bug Test automation Bug Environment issue Temporary unstable ※2020/1 ~ 2020/10 summary Unit = percentage 1. Investigate failed result 2. Run test automation again (retry) 3. Check result again Example. 1 operation = 2 mins Daily operation = 300 Monthly operation = 112 hours Operation to fix issue Reason ratio
- 28. 28 Agenda 1 2 3 4 5 Background Test automation operation issue Investigate issue Improve operation Conclusion
- 29. 29 Operation time needed for “temporary unstable” issue 0.5 4.9 20.4 74.1 Application Bug Test automation Bug Environment issue Temporary unstable ※2020/1 ~ 2020/10 summary Unit = percentage 1. Investigate failed result 2. Run test automation again (retry) 3. Check result again Example. 1 operation = 2 mins Daily operation = 300 Monthly operation = 112 hours Operation to fix issue Reason ratio
- 30. 30 Improve operation idea “Auto healing system” ・・・ Jenkins server 1. Use classified data as training data 2. predict failed reason by machine learning 3. Retry test if there is a temporary unstable issue ① ② Auto healing system Training data ③
- 31. 31 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes
- 32. 32 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Step-1Test executes and fails Test result output • text error message by test tool • Error page screenshot
- 33. 33 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Use data • Test result • Training data (previous classified result data) UseTechnology (machine learning) • Tesseract-OCR • Deep Neural Network with Keras Step-2 Predict failed reason
- 34. 34 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Input test report Extract message from screenshot Text base prediction Image base prediction Cannot predict reason Output prediction Feedback
- 35. 35 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Input test report Extract message from screenshot Text base prediction Image base prediction Cannot predict reason Output prediction Feedback text error message Screen shot
- 36. 36 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Input test report Extract message from screenshot Text base prediction Image base prediction Cannot predict reason Output prediction Feedback Invalid operation (不正な操作が行われました。)
- 37. 37 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Input test report Extract message from screenshot Text base prediction Image base prediction Cannot predict reason Output prediction Feedback 不正な操作が行われました。 Temporary unstable Training data [Error] Cannot found XXXXpath predict
- 38. 38 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Input test report Extract message from screenshot Text base prediction Image base prediction Cannot predict reason Output prediction Feedback Temporary unstable Training data predict
- 39. 39 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes Input test report Extract message from screenshot Text base prediction Image base prediction Cannot predict reason Output prediction Feedback Training data Prediction result
- 40. 40 Auto healing system flow Test Result is Failure Predict reason Temporary unstable Retry Yes • When predict reason is temporary unstable, retry test • Otherwise do nothing Step-3 check predict result and decide to retry
- 41. 41 Auto healing system brings … 1. Application bug 2. Test automation script bug 3. Environment issue 4. Temporary unstable bug report & wait for fix fix script & retry recover environment & retry retry Category Operation This operation is done in auto
- 42. 42 Auto healing system brings … 1. Retry operation is dismissed, then human cost is ZERO 2. Test result keep stable 3. Make good use of test environment resources
- 43. 43 Auto healing system brings … 0% 20% 40% 60% 80% 100% Jan-20 Feb-20 Mar-20 Apr-20 May-20 Jun-20 Jul-20 auto healing manual operation Example. 1 operation = 2 mins Daily operation = 300 Monthly operation = 112 hours In July 78% failed jobs is healed in auto Reduction = 88 hours All manual operation human cost Auto healing human cost 78% Ratio of way to fix issue
- 44. 44 Agenda 1 2 3 4 5 Background Test automation operation issue Investigate issue Improve operation Conclusion
- 45. 45 Conclusion 1. Some of test automation operation are non-productive activity 2. These operation can be reduced with machine learning 3. There is still room for improvement auto healing system
Editor's Notes
- Hello! Thank you for joining my presentation. I’m Emura Sadaaki from Rakuten. Today, I give you a presentation about “Test Automation Improvement by Machine Learning” Before start presentation, let me introduce myself.
- Go back to presentation. Today , this presentation is 2 topics. 1st topic is “Test automation has daily operation and issue” 2nd topic is “This issue is resolved by system with machine learning” Let’s get started
- Agenda is here. 1st : Our QA team background 2nd : What is Test automation operation in our team and this issue 3rd : investigate this operation issue in detail 4th : improve operation to resolve this issue Finally : conclusion
- Let go 1st agenda “Background”
- Our organization is this. Our dept. have mainly 2 type group. Left side is developer group to create product. This group has product manager and engineer. There are 11 developer group in our Dept. Right side is QA group to check product quality when every application release. This group has manual test team and test automation team. I belong to test automation team. This QA group do test for 11 developer group services.
- 11 services is here.
- I explain test automation team responsibility in QA group. Test automation team has 3 responsibility. 1st : test automation covers regression . 2nd : update regression script and add new script during project. 3rd : do regression test in test environment everyday. I explain each responsibility in detail one by one.
- 1st responsibility is test automation covers regression. This image show all function in product . Blue area is regression area. Test automation cover this blue area. Let’s say “one new project start”. In this time, Red area appears. This is new specification. And it is new project scope. Manual test focus on this project scope , it’s red area. Automation test cover regression during this project, it’s blue area.
- 2nd responsibility is “update regression script and create new script during project”. As I said in 1st responsibility, test automation focus on regression area. Area No.1, it is regression and this specification is changed to satisfy new specification. So test automation basically does not work well so to support last specification , we should modify script . Other side, Area No.2 , this function is totally new . Test automation does not cover this area yet. So to support this new function, we should create new script then add to existing regression. Finally we expand regression area
- 3rd responsibility is Regression test in test environment everyday. Not only testing during project, All test automation regression is scheduled everyday then check quality every time . This main objective is finding bug. others is finding environment issue, release operation rehearsal issue, non-functional bug (in Japanese 非機能テスト) Etc..
- As above, Test automation responsibility are - cover regression - maintain script - run regression test everyday To keep these responsibility , many operations come up!
- In 1st agenda I introduced our QA team organization and test automation responsibility. In next agenda, I explain test automation operation and issue during this operation.
- 3rd responsibility is “All regression test is scheduled everyday.” In this regression test, we have 3 operations 1st check test automation results 2nd investigate test result if test automation failed 3rd handle test automation results I explain 3 operation one by one
- 1st Operation . Check test automation results. Our team check if all regression test work well or not everyday. Currently we have around 20 Jenkins servers. In these Jenkins, around 2,000 regression test exist and run everyday. To check this so many test status, we have dashboard right image, this provide the test status summary in each service. Ex. How many test is success , how many test is failed . If test failed, we get the test result from this dashboard.
- 2nd operation is investigate test result of test automation failed. We get test result from dashboard, like this image. This report has information Where test is failed What test data is used Failed page Screenshot Etc. From test result , we investigate why test is failed. Failed reason is classified into mainly 4 category. 1st application bug 2nd test automation script bug 3rd environment issue 4th temporary unstable
- 3rd operation , we handle test automation results. After investigate failed test, to be test success , we fix issue Let me explain how to fix issue one by one
- 1st type “Application bug” In this case, test automation does not work till developer fix application bug. So 1st we report bug to engineers Engineers get this bug report, fix application bug then notice QA team . Finally run test automation again then check application bug fixed
- 2nd type “Test automation script bug” In this case, it’s our side issue. So check specification again, fix test automation script bug by myself. Finally run test automation again then check test automation work well.
- 3rd type “Environment issue” This type example is “maintenance mode” Developer set this mode to do some operation. During this maintenance, test environment does not work. So during this maintenance , we stop test automation. Communicate developer, get information maintenance is finished, retry test automation .then check “ test automation work well “. If we know this maintenance schedule in advance, manage test automation scheduler .
- Final 4th type “temporary unstable” This situation is test environment does not work “in short period”. For example. Network connection difficulty . Application cause timeout. Web server cause busy In this case, fixing solution is very simple only “retry test automation “ Although we take much time to investigate this type issue, final operation is “RETRY” So this temporary unstable issue is very non productive and take much time.
- To run All regression test everyday, Test automation has some operations. In these operations, temporary unstable issue is not productive and take much time.
- 3rd agenda investigate this test automation issue in detail.
- During test automation operation, temporary unstable issue operation is not productive and take much time. But this “much time” is hypothesis at that time. 1st , we need to verify this fact (take much time) . if true, consider improvement method to resolve it.
- To verify this fact, we collect all test result and find out issue trend. So build this system. 1st we collect “all test results from 20 Jenkins servers”, and stock this information in DB automatically. We check these test result, then classify all test results by failed reason category in dashboard.
- This is classify operation web tool . We investigate why test was failed, then classify failed reason category. choose failed reason from mainly 4 category.
- This is the summary failed reason category. This circle is reason ratio Red is “application bug”. it’s 0.5 % Blue is “Test automation script bug” . It’s about 5% Yellow is “Environment issue“. It’s about 20 % Green is “Temporary unstable”. It’s three quarters ! This type issue occupy most of all
- As I said , when this temporary unstable issue is happened, operation to fix issue is this step. 1st Investigate failed result 2nd retry test automation Finally check result. Lets calculate operation time. For example this 1 operation take 2 mins daily operation 300 In this condition, we take about 100 hours per month. It’s huge operation cost!
- Next agenda is “improve operation”
- By investigating issue, we found out “temporary unstable” type issue cause huge operation cost. Also this operation is non-productive . We decided to improve this operation to reduce human cost.
- This is system to improve operations. Called “Auto healing system” 1st we use classified data during investigate issue trend as training data (in Japanese 教師データ) 2nd This system predict failed reason by machine learning with this training data. Finally , If there is a temporary unstable issue, retry test automation in Jenkins
- This is flow chart
- 1st step : test executes and failed. In this time, we get test results , mainly 2 things - text error message - Error page screenshot
- 2nd Step : predict failed reason To predict it, Use test result and training data. This system use machine learning technology. Mainly 2 tech. -Tessact OCR -Deep Neural Network with Keras-library
- Prediction step detail is here
- As I said , we use test results “Test error message and Screenshot” as system input ,
- We extract text error message from screenshot. Example. Invalid operation,不正な操作が行われました in Japanese from screenshot
- Using both text error messages. 1st prediction is done with training data.
- If 1st prediction is difficult, 2nd prediction start , image base prediction.
- After prediction, this prediction result feedback to training data
- 3rd step check predict result , decide to retry test automation When Prediction is “temporary unstable”, retry test automation. Otherwise do nothing.
- Auto healing do “Prediction” and “RED AREA” - “Retry test automation” in auto.
- Auto healing system benefit is here 1st benefit : Retry operation is dismissed . Then human cost is ZEO. 2nd benefit : This auto healing run every time after test automation run. Some failed test is fixed soon in auto. So test results keep stable 3rd benefit : Retry operation is done only when failed test might be fixed . That’s why we can make good use of test environment resources.
- This is actual output This graph mean 100 % is all failed test automation. RED is fixed by manual . BLUE is fixed by auto healing, this fix operation is ZERO human operation Pick up July case. For example,monthly operation take about 112 hours. 78% is fixed by auto healing. So ,about 88 hours human cost is reduct! It’s very huge outcome.
- Conclusion
- I explained test automation responsibility. To keep this responsibility, we have many test automation operations, and some of them are non productive activity. I introduce Auto healing system with Machine learning. It can reduce these operation. We can reduce non productive cost, but this auto healing is not perfect yet. For example, prediction accuracy is not high, feedback is manual operation. That why there is still room for improvement auto healing system. So we want to improve this system more and more.
- That it from me. Thank you for attending my presentation!