ReportPortal.io - how to make machine learning categorize your test fails

SMART ANALYSISAI IN TESTING
MACHINE LEARNING
kNN and GradientBoosting

LEARNING SET
java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but
found [false]
org.testng.Assert.fail(Assert.java:94)
org.testng.Assert.failNotEquals(Assert.java:513)
org.testng.Assert.assertTrue(Assert.java:42)
my.project.web.tests.navigation.checkLinksFromServiceNavigationBarAreClickable(MainNavigatio
nServiceNavigationTest.java:61)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:100)
org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:189)
org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNG
SpringContextTests.java:175)
org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:201)
org.testng.internal.Invoker.invokeMethod(Invoker.java:642)
org.testng.internal.Invoker.invokeTestMethod(Invoker.java:811)
org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1137)
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
org.testng.TestRunner.privateRun(TestRunner.java:753)
org.testng.TestRunner.run(TestRunner.java:607)
org.testng.SuiteRunner.runTest(SuiteRunner.java:368)
org.testng.SuiteRunner.runSequentially(SuiteRunner.java:363)
org.testng.SuiteRunner.privateRun(SuiteRunner.java:321)
org.testng.SuiteRunner.run(SuiteRunner.java:270)
org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
org.testng.TestNG.runSuitesSequentially(TestNG.java:1284)
org.testng.TestNG.runSuitesLocally(TestNG.java:1209)
org.testng.TestNG.runSuites(TestNG.java:1124)
org.testng.TestNG.run(TestNG.java:1096)
my.project.web.ta.run.TestNgRunner.main(TestNgRunner.java:32)
java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but
found [false]
my.project.web.tests.navigation.checkLinksFromServiceNavigationBarAreClickable(MainNavigation
ServiceNavigationTest.java:61)
org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNGS
pringContextTests.java:175)
95%

BUT THERE IS A LITTLE
THING
java.lang.AssertionError: Invalid Upc Service Navigation link redirection. expected [true] but found [false]
my.project.web.tests.navigation.checkLinksFromServiceNavigationBarAreClickable(MainNavigationServiceNavigationTest.java:61)
org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNGSpringContextTests.java:175)

java lang assertionError invalid upc service navigation
link redirection expected true found false
org testng assert fail assert java
org testng assert failnotequals assert java
org testng assert asserttrue assert java
my project tests checkLinksareclickable maintest java
2017-09-02 07:55:37TIME
Build #1 Build #2 Build #3 Build #4 Build #5
java.lang.AssertionError: Invalid Upc Service Navigation
link redirection. expected [true] but found [false]
my.project.tests.checkLinksAreClickable(MainTest.java:61)
java lang AssertionError Invalid Upc Service Navigation
link redirection expected true but found false
org testng Assert fail Assert java 94
org testng Assert failNotEquals Assert java 513
org testng Assert assertTrue Assert java 42
my project tests checkLinksAreClickable MainTest java 61
link redirection expected true but found false
org testng assert fail assert java 94
org testng assert failnotequals assert java 513
org testng assert asserttrue assert java 42
my project tests checkLinksareclickable maintest java 61
link redirection expected true found false
org testng assert fail assert java
org testng assert failnotequals assert java
org testng assert asserttrue assert java
my project tests checkLinksareclickable maintest java

REAL-TIME SEARCH
• OPEN SOURCED
• FULL TEXT SEARCH
• ~REAL-TIME
• SCALABLE

AssertionError Invalid Service expected found
AssertionError expected
Invalid
AssertionError Invalid found
expected found
Invalid expected
AssertionError Invalid
Service expected
expected found
expected found
4 5 2 7 5

TF - IDF
Жил-был в лесу заяц. Все зайцы были как зайцы: летом серые,
зимой белые. А этот и зимой и летом был одним цветом. И цвет этот
был ни белый, ни серый, а просто грязный, потому что заяц никогда
не умывался. Шел как-то он по тропинке, а навстречу ему лиса.- Ты
кто? – спрашивает лиса.- Заяц, - ответил заяц.- Не может быть, -
замотала головой лиса. – Я никогда таких зайцев не видела, таких
страшных не бывает! Может быть ты еж?- Почему? – удивился заяц.-
Потому что на тебе солома старая, и шерсть вся свалялась, на
иголки стала похожа. Заяц обиделся, но решил, что умываться не
будет.
TF = 6 / 100 =
0.06IDF = Log (10 000 000/ 1000) = 4
TF-IDF = 0.06 * 4 = 0.24

AssertionError
Invalid
Serviceexpected
found

HOW IT DONE
ELASTIC
Service-
analyzer
REPORT PORTAL
API

INDEX STRUCTURE
"mappings": map[string]interface{}{
"log": map[string]interface{}{
"properties": map[string]interface{}{
"message": map[string]interface{}{
"type": "text",
"analyzer": "standard",
},
"launch_name": map[string]interface{}{ "type": "keyword” },
"test_item": map[string]interface{}{ "type": "keyword” },
"issue_type": map[string]interface{}{ "type": "keyword” },
"log_level": map[string]interface{}{ "type": "integer” },
},
},
}

ANALYZER ANATOMY"analyzer": map[string]interface{}{
"character_filter":
"tokenizer":
"token_filters":
}

BOOL QUERY
return map[string]interface{}{
"size": 10,
"query": map[string]interface{}{
"bool": map[string]interface{}{
"must_not":
"must":
"should":
},
},

MUST NOT"must_not": map[string]interface{}{
"wildcard": map[string]interface{}{
"issue_type": "TI*",
},
},

MUST
"must": []interface{}{
map[string]interface{}{
"term": map[string]interface{}{
"log_level": 40000,
},
},
"exists": map[string]interface{}{
"field": "issue_type",
},
},
"more_like_this": map[string]interface{}{
"fields": []string{"message"},
"like": logMessage,
"minimum_should_match": "90%",
},

SHOULD
"should": map[string]interface{}{
"term": map[string]interface{}{
"launch_name": map[string]interface{}{
"value": launchName,
"boost": 2.0,
},
},
},

CHALLENGES
• IF NOBODY CHECK RESULT – IT DEGRADATE
• LEARNING ALWAYS NEEDS TRAINER
• IF MACHINE ERROR APPEARS - ERROR INFLUENCE GROWTHS
• NEED TO FLUSH TRAINING

OPEN SOURCED
GPL V3.0
GITHUB.COM/ REPORTPORTAL

REAL-TIME INTEGRATION
FEW HOURS
NO REFACTORING REQUIRED!
TestNG / JUnit
Xunit, NUnit2/3
VSTest
SpecFlow
jBehave
ScalaTest, Spock
Cucumber
Cucumber-JVM
Cucumber Ruby
Python PyTest
RobotFramework
JS Mocha, JS Jasmine
Protractor Node.JS
SoapUI / ReadyAPI
…

WHO IS THIS GUY?
Report Portal – Product Owner.
10 years with EPAM.
Development background.
Leading initiatives development at
Test Competency Center.
DZMITRY HUMIANIUK
EPAM Systems, Delivery Manager
ДМИТРИЙ ГУМЕНЮК

ReportPortal.io
GITHUB.COM/ REPORTPORTAL

ReportPortal.io - how to make machine learning categorize your test fails

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

More from Dmitriy Gumeniuk

More from Dmitriy Gumeniuk (11)

ReportPortal.io - how to make machine learning categorize your test fails