Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
MACHINE LEARNING
SMART ANALYSISAI IN TESTING
MACHINE LEARNING
kNN and GradientBoosting
CATEGORIZE AND GROUP
FAILS
LEARNING SET
java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but
found [false]
...
BUT THERE IS A LITTLE
THING
java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but...
java lang assertionError invalid upc service navigation
link redirection expected true found false
org testng assert fail ...
REAL-TIME SEARCH
• OPEN SOURCED
• FULL TEXT SEARCH
• ~REAL-TIME
• SCALABLE
AssertionError Invalid Service expected found
AssertionError expected
Invalid
AssertionError Invalid found
expected found
...
TF - IDF
Жил-был в лесу заяц. Все зайцы были как зайцы: летом серые,
зимой белые. А этот и зимой и летом был одним цветом....
AssertionError
Invalid
Serviceexpected
found
N* exception
M* expected
HOW IT DONE
ELASTIC
Service-
analyzer
REPORT PORTAL
API
INDEX STRUCTURE
"mappings": map[string]interface{}{
"log": map[string]interface{}{
"properties": map[string]interface{}{
"...
ANALYZER ANATOMY"analyzer": map[string]interface{}{
"character_filter":
"tokenizer":
"token_filters":
}
BOOL QUERY
return map[string]interface{}{
"size": 10,
"query": map[string]interface{}{
"bool": map[string]interface{}{
"mu...
MUST NOT"must_not": map[string]interface{}{
"wildcard": map[string]interface{}{
"issue_type": "TI*",
},
},
MUST
"must": []interface{}{
map[string]interface{}{
"term": map[string]interface{}{
"log_level": 40000,
},
},
map[string]i...
SHOULD
"should": map[string]interface{}{
"term": map[string]interface{}{
"launch_name": map[string]interface{}{
"value": l...
CHALLENGES
• IF NOBODY CHECK RESULT – IT DEGRADATE
• LEARNING ALWAYS NEEDS TRAINER
• IF MACHINE ERROR APPEARS - ERROR INFL...
OPEN SOURCED
GPL V3.0
GITHUB.COM/ REPORTPORTAL
REPORT PORTAL
REAL-TIME INTEGRATION
FEW HOURS
NO REFACTORING REQUIRED!
TestNG / JUnit
Xunit, NUnit2/3
VSTest
SpecFlow
jBehave
ScalaTest,...
WHO IS THIS GUY?
Report Portal – Product Owner.
10 years with EPAM.
Development background.
Leading initiatives developmen...
OPEN COMMUNITY
ReportPortal.io
GITHUB.COM/ REPORTPORTAL
QA Fest 2017. Дмитрий Гуменюк. ReportPortal.io - Как заставить Машинное Обучение разбирать результаты ваших автоматизирова...
Upcoming SlideShare
Loading in …5
×

QA Fest 2017. Дмитрий Гуменюк. ReportPortal.io - Как заставить Машинное Обучение разбирать результаты ваших автоматизированных тестов

229 views

Published on

Каждый проект с автоматизированными тестированием тратит время на построение своей системы сбора и генераци отчетов. Это особенно становится большой проблемой, когда у вас крупная распределенная среда и различные платформы для запусков. ReportPortal дает вам единое место для хранения результатов автоматизированного тестирования со всеми скриншотами и бинарными данными. Вы можете категоризировать свои причины падений, постить баги и просматривать историю запусков в один клик. Увидеть результаты тестов в режиме реального времени. Основываясь на истории предыдущих прогонов и собранных паттернов падений, ReportPortal использует мощь AI (машинного обучения) для анализа ваших новых падений, делая вашу работу за вас.

Published in: Education
  • Be the first to comment

  • Be the first to like this

QA Fest 2017. Дмитрий Гуменюк. ReportPortal.io - Как заставить Машинное Обучение разбирать результаты ваших автоматизированных тестов

  1. 1. MACHINE LEARNING
  2. 2. SMART ANALYSISAI IN TESTING MACHINE LEARNING kNN and GradientBoosting
  3. 3. CATEGORIZE AND GROUP FAILS
  4. 4. LEARNING SET java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but found [false] org.testng.Assert.fail(Assert.java:94) org.testng.Assert.failNotEquals(Assert.java:513) org.testng.Assert.assertTrue(Assert.java:42) my.project.web.tests.navigation.checkLinksFromServiceNavigationBarAreClickable(MainNavigatio nServiceNavigationTest.java:61) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:100) org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:189) org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNG SpringContextTests.java:175) org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:201) org.testng.internal.Invoker.invokeMethod(Invoker.java:642) org.testng.internal.Invoker.invokeTestMethod(Invoker.java:811) org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1137) org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) org.testng.TestRunner.privateRun(TestRunner.java:753) org.testng.TestRunner.run(TestRunner.java:607) org.testng.SuiteRunner.runTest(SuiteRunner.java:368) org.testng.SuiteRunner.runSequentially(SuiteRunner.java:363) org.testng.SuiteRunner.privateRun(SuiteRunner.java:321) org.testng.SuiteRunner.run(SuiteRunner.java:270) org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) org.testng.TestNG.runSuitesSequentially(TestNG.java:1284) org.testng.TestNG.runSuitesSequentially(TestNG.java:1280) org.testng.TestNG.runSuitesLocally(TestNG.java:1209) org.testng.TestNG.runSuites(TestNG.java:1124) org.testng.TestNG.run(TestNG.java:1096) my.project.web.ta.run.TestNgRunner.main(TestNgRunner.java:32) java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but found [false] org.testng.Assert.fail(Assert.java:94) org.testng.Assert.failNotEquals(Assert.java:513) org.testng.Assert.assertTrue(Assert.java:42) my.project.web.tests.navigation.checkLinksFromServiceNavigationBarAreClickable(MainNavigation ServiceNavigationTest.java:61) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:100) org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:189) org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNGS pringContextTests.java:175) org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:201) org.testng.internal.Invoker.invokeMethod(Invoker.java:642) org.testng.internal.Invoker.invokeTestMethod(Invoker.java:811) org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1137) org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) org.testng.TestRunner.privateRun(TestRunner.java:753) org.testng.TestRunner.run(TestRunner.java:607) org.testng.SuiteRunner.runTest(SuiteRunner.java:368) org.testng.SuiteRunner.runSequentially(SuiteRunner.java:363) org.testng.SuiteRunner.privateRun(SuiteRunner.java:321) org.testng.SuiteRunner.run(SuiteRunner.java:270) org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) org.testng.TestNG.runSuitesSequentially(TestNG.java:1284) org.testng.TestNG.runSuitesSequentially(TestNG.java:1280) org.testng.TestNG.runSuitesLocally(TestNG.java:1209) org.testng.TestNG.runSuites(TestNG.java:1124) org.testng.TestNG.run(TestNG.java:1096) my.project.web.ta.run.TestNgRunner.main(TestNgRunner.java:32) 95%
  5. 5. BUT THERE IS A LITTLE THING java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but found [false] org.testng.Assert.fail(Assert.java:94) org.testng.Assert.failNotEquals(Assert.java:513) org.testng.Assert.assertTrue(Assert.java:42) my.project.web.tests.navigation.checkLinksFromServiceNavigationBarAreClickable(MainNavigationServiceNavigationTest.java:61) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:100) org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:189) org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNGSpringContextTests.java:175) org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:201) org.testng.internal.Invoker.invokeMethod(Invoker.java:642) org.testng.internal.Invoker.invokeTestMethod(Invoker.java:811) org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1137) org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) org.testng.TestRunner.privateRun(TestRunner.java:753) org.testng.TestRunner.run(TestRunner.java:607) org.testng.SuiteRunner.runTest(SuiteRunner.java:368) org.testng.SuiteRunner.runSequentially(SuiteRunner.java:363) org.testng.SuiteRunner.privateRun(SuiteRunner.java:321) org.testng.SuiteRunner.run(SuiteRunner.java:270) org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) org.testng.TestNG.runSuitesSequentially(TestNG.java:1284) org.testng.TestNG.runSuitesSequentially(TestNG.java:1280) org.testng.TestNG.runSuitesLocally(TestNG.java:1209) org.testng.TestNG.runSuites(TestNG.java:1124) org.testng.TestNG.run(TestNG.java:1096) my.project.web.ta.run.TestNgRunner.main(TestNgRunner.java:32)
  6. 6. java lang assertionError invalid upc service navigation link redirection expected true found false org testng assert fail assert java org testng assert failnotequals assert java org testng assert asserttrue assert java my project tests checkLinksareclickable maintest java 2017-09-02 07:55:37TIME Build #1 Build #2 Build #3 Build #4 Build #5 java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but found [false] org.testng.Assert.fail(Assert.java:94) org.testng.Assert.failNotEquals(Assert.java:513) org.testng.Assert.assertTrue(Assert.java:42) my.project.tests.checkLinksAreClickable(MainTest.java:61) java lang AssertionError Invalid Upc Service Navigation link redirection expected true but found false org testng Assert fail Assert java 94 org testng Assert failNotEquals Assert java 513 org testng Assert assertTrue Assert java 42 my project tests checkLinksAreClickable MainTest java 61 java lang assertionError invalid upc service navigation link redirection expected true but found false org testng assert fail assert java 94 org testng assert failnotequals assert java 513 org testng assert asserttrue assert java 42 my project tests checkLinksareclickable maintest java 61 java lang assertionError invalid upc service navigation link redirection expected true found false org testng assert fail assert java org testng assert failnotequals assert java org testng assert asserttrue assert java my project tests checkLinksareclickable maintest java
  7. 7. REAL-TIME SEARCH • OPEN SOURCED • FULL TEXT SEARCH • ~REAL-TIME • SCALABLE
  8. 8. AssertionError Invalid Service expected found AssertionError expected Invalid AssertionError Invalid found expected found Invalid expected AssertionError Invalid Service expected expected found expected found 4 5 2 7 5
  9. 9. TF - IDF Жил-был в лесу заяц. Все зайцы были как зайцы: летом серые, зимой белые. А этот и зимой и летом был одним цветом. И цвет этот был ни белый, ни серый, а просто грязный, потому что заяц никогда не умывался. Шел как-то он по тропинке, а навстречу ему лиса.- Ты кто? – спрашивает лиса.- Заяц, - ответил заяц.- Не может быть, - замотала головой лиса. – Я никогда таких зайцев не видела, таких страшных не бывает! Может быть ты еж?- Почему? – удивился заяц.- Потому что на тебе солома старая, и шерсть вся свалялась, на иголки стала похожа. Заяц обиделся, но решил, что умываться не будет. TF = 6 / 100 = 0.06IDF = Log (10 000 000/ 1000) = 4 TF-IDF = 0.06 * 4 = 0.24
  10. 10. AssertionError Invalid Serviceexpected found
  11. 11. N* exception M* expected
  12. 12. HOW IT DONE ELASTIC Service- analyzer REPORT PORTAL API
  13. 13. INDEX STRUCTURE "mappings": map[string]interface{}{ "log": map[string]interface{}{ "properties": map[string]interface{}{ "message": map[string]interface{}{ "type": "text", "analyzer": "standard", }, "launch_name": map[string]interface{}{ "type": "keyword” }, "test_item": map[string]interface{}{ "type": "keyword” }, "issue_type": map[string]interface{}{ "type": "keyword” }, "log_level": map[string]interface{}{ "type": "integer” }, }, }, }
  14. 14. ANALYZER ANATOMY"analyzer": map[string]interface{}{ "character_filter": "tokenizer": "token_filters": }
  15. 15. BOOL QUERY return map[string]interface{}{ "size": 10, "query": map[string]interface{}{ "bool": map[string]interface{}{ "must_not": "must": "should": }, },
  16. 16. MUST NOT"must_not": map[string]interface{}{ "wildcard": map[string]interface{}{ "issue_type": "TI*", }, },
  17. 17. MUST "must": []interface{}{ map[string]interface{}{ "term": map[string]interface{}{ "log_level": 40000, }, }, map[string]interface{}{ "exists": map[string]interface{}{ "field": "issue_type", }, }, map[string]interface{}{ "more_like_this": map[string]interface{}{ "fields": []string{"message"}, "like": logMessage, "minimum_should_match": "90%", },
  18. 18. SHOULD "should": map[string]interface{}{ "term": map[string]interface{}{ "launch_name": map[string]interface{}{ "value": launchName, "boost": 2.0, }, }, },
  19. 19. CHALLENGES • IF NOBODY CHECK RESULT – IT DEGRADATE • LEARNING ALWAYS NEEDS TRAINER • IF MACHINE ERROR APPEARS - ERROR INFLUENCE GROWTHS • NEED TO FLUSH TRAINING
  20. 20. OPEN SOURCED GPL V3.0 GITHUB.COM/ REPORTPORTAL
  21. 21. REPORT PORTAL
  22. 22. REAL-TIME INTEGRATION FEW HOURS NO REFACTORING REQUIRED! TestNG / JUnit Xunit, NUnit2/3 VSTest SpecFlow jBehave ScalaTest, Spock Cucumber Cucumber-JVM Cucumber Ruby Python PyTest RobotFramework JS Mocha, JS Jasmine Protractor Node.JS SoapUI / ReadyAPI …
  23. 23. WHO IS THIS GUY? Report Portal – Product Owner. 10 years with EPAM. Development background. Leading initiatives development at Test Competency Center. DZMITRY HUMIANIUK EPAM Systems, Delivery Manager ДМИТРИЙ ГУМЕНЮК
  24. 24. OPEN COMMUNITY
  25. 25. ReportPortal.io GITHUB.COM/ REPORTPORTAL

×