This document discusses test observability and providing visibility into test results. It proposes collecting and indexing various types of test data, including unit, integration, API, UI, and manual tests, to provide unified reporting and allow tracing between code and tests. This would provide a heatmap of coverage, easy understanding of test coverage evolution over time, and ability to easily trace common errors to understand weaknesses. The goal is to provide a comprehensive view of testing that tells the story of a system's tests.

1. Test Observability
Ignacio Esmite07.11.2017

2. Context

11. Development Flow

12. FLOW

13. Create a Branch

14. Add Commits

15. Open a Pull Request

16. Discuss and Review your Code

17. Deploy

18. Merge

19. A Test Pyramid
E2E
INTEGRATION
UNIT
HUMAN
TESTING
COMPONENT
CONTRACT

20. Test Observability

23. Tests Results

25. Tests Results
STATUS
EXEC TIME
DESCRIPTION
LOGS
ENVIRONMENT
- Classic Pie chart
- Evolution over time
- Flakiness Analysis
- Skipped Tests
- Exec Time Evolution
- Parallelization Analysis
- Quality Of Service Analysis
ERROR INFO
- Indexing by words on description
- Search correlations on different types of tests
- Index on errors (Search)
- Common errors
- Classification
- Metrics per environment
- Comparison cross environment
- Evolution of tests per environment
STRUCTURE
- Analysis
- Independent of CI
- Graph
- Browsing Results
- Classic Metrics

26. API Testing / Endpoints
COVERAGE
ENDPOINTS
TRENDS
DEVIATION ON
SDLC
MICRO
BENCHMARKS
- On Endpoints & Methods
- Payload type?
- Others
- Popular Endpoints
- # Tests Over Time per Endpoint
- Others
- Response Times
- Endpoint Coverage
- New Endpoints Coverage

27. UI TESTS
CONSOLE LOG
STEPS TO
REPRODUCE
BROWSER
- Javascript Errors
- Indexing on errors
- Error classification
- Errors by browser
- Skipped by browser
- Other classifications
- Screenshots storage
- Videos
- Story Telling

28. Keywords
RISK
ASSESSMENT
BACKLOG
BUILDING
DEVIATION ON
SDLC
- Big Picture of Coverage
- Easy to understand
- Evaluate weak areas
- Compare Test Focus with Tested Areas
- What Happen on new
Releases
- Evolution of tests over time

29. Manual Testing
COVERAGE
UNIFIED
SOURCE
NOTE TAKING
- Scenarios
- Keywords
- Reporting
- Intersect Manual and Automated
- Unified Reports
- Easy to communicate
- NOT Lost & Found
- Exploratory Sessions/Threads
- Indexed for Search

30. Summary & Possible Implementation
API Testing(Endpoints) Collector
UI Tests Collector
Keyword Collector
Tree Test Collector
Manual Testing Collector
HeatMap of
- Keywords
- Endpoints
Unified Report
Manual/Automation
Understanding
of CoverageTracing Between
Software Artifacts
and How we tested
Risk Based
Testing
Understanding of
Evolution or
Involution of Tests
Easy to trace errors
and understand
common weaknesses

31. Story Telling

32. UNIT INTEGRATION E2E HUMAN
Story Telling

35. Gracias!!