This document discusses test trend analysis and making tests more robust, reliable, and timely. It proposes collecting test results data and storing it in Elasticsearch. Visualizations would then be created using Kibana to analyze test failures, slow tests, error messages, and step times. This would provide insights and help identify issues to make tests less flaky.

1. @hughleo01 Liberty Information Technology
Liberty IT, Belfast and Dublin
Test trend analysis:Towards robust,
reliable and timely tests
Hugh McCamphill

2. @hughleo01 Liberty Information Technology
2Liberty IT @hughleo01
Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC
Redmond DC
Global DRDC
& Utilities
Kansas City DC
Latin America RDC
GS LIU DRDC
Portsmouth DC
GS LIU DC
Turkey DCs India DCs Asia RDC China DCs
Wausau
Fairfield
Boston
Seattle Blanchardstown
11 major IT offices
7 country data centers (DCs)
DRDC = Disaster Recovery DC
3 US domestic data centers (DCs)
4 international regional data centers
LIU = Liberty International Underwriters
More than 5,000+ IT
employees around the globe
Australia
About Us

3. 3Liberty IT
Hugh McCamphill
Principal Software Engineer in Test
Writing Automation since 2004
@hughleo01

4. @hughleo01 Liberty Information Technology
4Liberty IT @hughleo01
Which tests are slow
Which steps are slow
By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by
Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553

5. @hughleo01 Liberty Information Technology
5Liberty IT @hughleo01
Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, Northumberland
Image Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike
2.0 Generic Licence
Which tests are intermittently passing
What are common failures across tests

6. @hughleo01 Liberty Information Technology
6Liberty IT @hughleo01
“Create commons cute robot mural, Toronto"
by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0
Where are the issues?

7. @hughleo01 Liberty Information Technology
7Liberty IT @hughleo01

8. @hughleo01 Liberty Information Technology
8Liberty IT @hughleo01

9. @hughleo01 Liberty Information Technology
9Liberty IT @hughleo01
~ Henry David Thoreau

10. @hughleo01 Liberty Information Technology
10Liberty IT @hughleo01

11. @hughleo01 Liberty Information Technology
11Liberty IT @hughleo01
Example dashboard

12. @hughleo01 Liberty Information Technology
12Liberty IT @hughleo01
RunTests Results
Post
Results
A Simple Process Flow

13. @hughleo01 Liberty Information Technology
13Liberty IT @hughleo01
A time series database is a
software system that is optimized
for handling time series data,
arrays of numbers indexed
by time
Elasticsearch is a tool for storing, searching, and
analyzing structured and unstructured data

14. @hughleo01 Liberty Information Technology
14Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}

15. @hughleo01 Liberty Information Technology
15Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}

16. @hughleo01 Liberty Information Technology
16Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}

17. @hughleo01 Liberty Information Technology
17Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
curl -XPUT “http://localhost:9200/test-result” –
d @mapping_schema_framework.json

18. @hughleo01 Liberty Information Technology
18Liberty IT @hughleo01
{"index":{"_index":"test-result","_type":"functional" }}
{"project":“TestProject",
"environment":"QA",
"classname":“TestProject.Foo.Bar",
"name":"ShouldBeAbleToAddItem",
"duration":"47.12",
"created":"2016-09-14 21:35:53",
"status":"Passed"}
curl -XPOST “http://localhost:9200/test-
result/_bulk?pretty” --data-binary @testResults.json
Posting of results

19. @hughleo01 Liberty Information Technology
19Liberty IT @hughleo01
A Simple Process Flow
RunTests Results
Post
Results Visualize

20. @hughleo01 Liberty Information Technology
20Liberty IT @hughleo01
Kibana is an open source data visualization plugin for ElasticSearch

21. @hughleo01 Liberty Information Technology
21Liberty IT @hughleo01

22. @hughleo01 Liberty Information Technology
22Liberty IT @hughleo01
New visualization results in a single bucket

23. @hughleo01 Liberty Information Technology
23Liberty IT @hughleo01
Splitting error messages into buckets of top occurring error messages

24. @hughleo01 Liberty Information Technology
24Liberty IT @hughleo01
Showing top ten most occurring error messages

25. @hughleo01 Liberty Information Technology
25Liberty IT @hughleo01
Splitting error messages into buckets of tests affected by that error

26. @hughleo01 Liberty Information Technology
26Liberty IT @hughleo01
Error Messages
Individual Tests
Splitting error messages into buckets of tests affected by that error

27. @hughleo01 Liberty Information Technology
27Liberty IT @hughleo01
Top Failure Reasons By Test

28. @hughleo01 Liberty Information Technology
28Liberty IT @hughleo01
Top Slowest Tests

29. @hughleo01 Liberty Information Technology
29Liberty IT @hughleo01
Top Failing Tests

30. @hughleo01 Liberty Information Technology
30Liberty IT @hughleo01
Step Time Versus Number Of Times Executed

31. @hughleo01 Liberty Information Technology
31Liberty IT @hughleo01
Step Time Versus Number Of Times Executed

32. @hughleo01 Liberty Information Technology
32Liberty IT @hughleo01
Step Time Versus Number Of Times Executed

33. @hughleo01 Liberty Information Technology
33Liberty IT @hughleo01
Example Dashboard With Time Filter

34. @hughleo01 Liberty Information Technology
34Liberty IT @hughleo01
References
http://martinfowler.com/articles/nonDeterminism.htm
https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/

35. @hughleo01 Liberty Information Technology
35Liberty IT @hughleo01
Thank you
Hugh McCamphill
@hughleo01
Run Tests Results
Post
Results Visualize
In Summary