2. www.eurostarconferences.com
Introduction
• > 15 years working in both operations and in test (Telco
industry)
• made responsible for operations of the systems I had
been testing for years.
• had to eat my own dogfood.....
• now back in testing
www.eurostarconferences.com
3. www.eurostarconferences.com
Excuses, excuses…
• Serious issues after go live
• “How could the testers have missed this?”
• We know all the reasons, right?
– can't test everything
– time
– wasn't in the design...
But in operations we knew within hours...
5. www.eurostarconferences.com
The situation
• Business process travel through many systems
• Systems coupled by enterprise service bus (ESB)
• Documentation is changed based, not system based
• No feedback loop test and operations
• Operations also handled business process issues
7. www.eurostarconferences.com
Findings
• Testers and operations use different tools
• Testers focus on their own test, ignore many other errors
• Test was finished when cases passed: No final check if we could perform the
functions with no errors left
• Forking nature service bus makes defining expected result hard
• Operations is experience based
• Operations cannot ignore errors → a customer is impacted
It differs! Both ops and testers had exceptions
8. www.eurostarconferences.com
How does operations check?
• Operations
– has no expected result, they have errors
– is experience based, they learn where to look
– Acts on errors from monitoring and reported issues
• Analysis on errors: trace the business process
• One ‘advantage’ over test: every error counts..
9. www.eurostarconferences.com
Down the rabbit hole
• Issues detected, unlike with test:
– unkown actions by user (no test case)
– not always expected result
• Analysis:
– Start at system with error(s)
– Trace back through logs in the systems
10. www.eurostarconferences.com
How can we use this?
• Lessons learned:
– Don’t just focus on the test case, get a view on the errors in
the systems
– Track back logs of your tests in all the systems, also the ones
you were unaware of
– When all test cases are done, (re)test the core business
processes and expect no errors
Gtac 2007: logs are a huge untapped resource
11. www.eurostarconferences.com
DIY or get operations involved
• Applying the skills and tools of operations improves
testing
• Operations is experience based and evolves -> get
involvement not just lessons
12. www.eurostarconferences.com
Classical operational analytics
• Digging through log files and databases is a lot of work
– Log in to many systems
– Manually search for the info of your process in many lines of
data
– Easy to miss lines or even logs
– The big picture is hard to see
13. www.eurostarconferences.com
Operational analytics software
• indexes logging information in real time
• indexes from multiple systems
• each line is turned into a time based event
• shows all the errors warnings etc.
• can show anything relating to your business process
• visualizes in real time
• provides history (is this error new)
• tackles the hard to reproduce issues
14. www.eurostarconferences.com
The haystack
• Operational analytics helps you to:
– See the big picture
– Find the needle
• The big picture: graphs, history, numbers
• The needle: find the error related to your business
process
Shouldn’t we be doing that in test as well?
19. www.eurostarconferences.com
Additional advantages for test
• After the fact data masking
• New testers up and running fast (central location of
info)
• Make ‘protected logs’ available for testing
• Monitor third party activity
• Quickly debunk emotional claims…..
– Or prove the seriousness!
20. www.eurostarconferences.com
Use cases for test
• Search (Google) for errors
• Full business process overview
• Fast root cause analysis
• Reproduce the hard cases
• Reliability dashboards (nr of errors/warnings vs. nr of (unique)
business processes)
• Outsourcing monitoring
• Production shadow (southbound systems)
21. www.eurostarconferences.com
Search for errors
• Actual error messages
• Statistics on errors (10% fail)
• Find the strange error that we couldn’t reproduce
• Advantages:
– Find unknown errors
– Find the cause for multiple functional errors
• -
22. www.eurostarconferences.com
Production shadow
• All life cycle
events in both
systems
• OA tools show
all errors
• Bills can be
compared
• Works ‘south
bound’
order
manager
Old billing
system
ESB
Customer
services system
Mediation
Usage (CDR’s)
New billing
system
Customer
services system
Existing system
Duplication point
Copy system (validation)
Greenfield interface
23. www.eurostarconferences.com
So no more testing?
• OA can show errors but not functionality failure
(2+2=5) ??
• With no one testing, there is nothing logged
It augments testing
24. www.eurostarconferences.com
Lessons learned
• Operations people are hard to get into a testing mode
• Resistance to OA, ‘we already have tool for that’
• Just teaching ops skills to testing misses the learning
part
• OA is useless without the right and skilled people
25. www.eurostarconferences.com
Summary
• Get people from ops involved
• Systems are too complex to expect testers to fully
understand expected result
• Logs are a huge resource
• Operational analytics helps the able
