The document discusses various topics related to software testing and development including testing environments vs reality, risk management approaches, monitoring software usage through event processing and generating alerts. It notes that testing environments often differ significantly from production with unstable conditions, humans and latency. Two approaches to risk management are described: narrowly scoping problems and extensive testing vs rapid iteration, testing changes in real world, and only keeping what works. Event processing and monitoring tools can provide real-time insights into software usage and changes.

1. I have a test network. You may know it as
production.
– Andreas Thienemann
Testing

2. Asking the right questions
● Information is not a scarce resource, attention
is
– Herb Simon

3. Asking the right questions
● Information is not a scarce resource, attention
is
– Herb Simon
● What do you know?

4. Asking the right questions
● Information is not a scarce resource, attention
is
– Herb Simon
● What do you know?
● What do you not know?

5. Asking the right questions
● Information is not a scarce resource, attention
is
– Herb Simon
● What do you know?
● What do you not know?
● What do you not know that you do not know?

6. Feedback loops

7. Fast Feedback Loops
● Test driven business

8. Fast Feedback Loops
● Test driven business
– It's like Test Driven
Design

9. Fast Feedback Loops
● Test driven business
– It's like Test Driven
Design
● Enable IT to speak
the language of
business

10. Fast Feedback Loops
● Test driven business
– It's like Test Driven
Design
● Enable IT to speak
the language of
business
– Show me the data!

11. Fast Feedback Loops
● Test driven business
– It's like Test Driven
Design
● Enable IT to speak
the language of
business
– Show me the data!
– HIPPO

12. What makes software “good”?

13. What makes software “good”?
● Developer metrics

14. What makes software “good”?
● Developer metrics
– Object Oriented/Functional/
– Is testable
– Follows DRY
– ...

15. What makes software “good”?
● Developer metrics
– Object Oriented/Functional/
– Is testable
– Follows DRY
– …
● Operational metrics
– Bug “free”
– Scalable
– Secure

16. What makes software “good”?
● Business metrics

17. What makes software “good”?
● Business metrics
– Does it help me make money?

18. What makes software “good”?
● Business metrics
– Does it help me make money?
– Does it save me money?

19. What makes software “good”?
● Business metrics
– Does it help me make money?
– Does it save me money?
– Does it generate value?

20. The Classical SDLC

21. Classical SDLC feedback loop

22. Plans vs reality

24. Testing vs Reality
● Stable environment ● Highly unstable

25. Testing vs Reality
● Stable environment
● No humans
● Highly unstable
● Humans

26. Testing vs Reality
● Stable environment
● No humans
● Low latency
● Highly unstable
● Humans
● Potentially high
latency

27. Testing vs Reality
● Stable environment
● No humans
● Low latency
● Not always the same
size of dataset
● Highly unstable
● Humans
● Potentially high
latency
● Large, ever changing
datasets

28. Users
● Humans do strange things

29. Users
● Humans do strange things
● Or sometimes make mistakes

30. Users
● Humans do strange things
● Or sometimes make mistakes
● They come up with different requirements

31. Users
● Humans do strange things
● Or sometimes make mistakes
● They come up with different requirements
● They change the world your software works in

32. Risk management
● Approach 1:
– Scope your problems well
– Test a lot
– Release stable code
– Avoid changing a working system

33. Risk management
● Approach 2:
– Accept that you have an ill-defined problem
– Iterate rapidly
– Make a large number of small changes
– Build software to be able to isolate these changes
– Test them in the real world
– Keep only what works

34. What if When it breaks?
● Fix fast (maybe)
– “Do it right the first time” does not apply
● Business process for handling failure
– Hardware will eventually fail, software will
eventually work

35. The lifetime of code
● How long does your code live?

36. The lifetime of code
● How long does your code live?
– Hours or days?
● This should be most code out there

37. The lifetime of code
● How long does your code live?
– Hours or days?
● This should be most code out there
– Months?
● A little code, often “libraries” with a single application

38. The lifetime of code
● How long does your code live?
– Hours or days?
● This should be most code out there
– Months?
● A little code, often “libraries” with a single application
– Years?
● Very little. Just core libraries

39. The lifetime of code
● How long does your code live?
– Hours or days?
● This should be most code out there
– Months?
● A little code, often “libraries” with a single application
– Years?
● Very little. Just core libraries
● It is safe to delete code, if you are using version
control

40. Event processing
● Generate information about software use and
changes in realtime
● For more information and tooling:
– https://www.quora.com/Are-there-any-open-source-
CEP-tools?share=1
– https://en.wikipedia.org/wiki/Complex_event_proces
sing

41. Event processing
● Generate information about software use and
changes in realtime
● For more information and tooling:
– https://www.quora.com/Are-there-any-open-source-
CEP-tools?share=1
– https://en.wikipedia.org/wiki/Complex_event_proces
sing

42. Monitoring/Alerting
● Process events to generate graphs

43. Monitoring/Alerting
● Process events to generate graphs
● Riemann is an excellent tool for generating
alerts from event streams

44. Monitoring/Alerting
● Process events to generate graphs
● Riemann is an excellent tool for generating
alerts from event streams
● Generate graphs as close to realtime as
possible
– Developers doing rollouts know that something else
is changing

45. Monitoring/Alerting
● Process events to generate graphs
● Riemann is an excellent tool for generating
alerts from event streams
● Generate graphs as close to realtime as
possible
– Developers doing rollouts know that something else
is changing
– Any major problems will be caught really quickly

46. Monitoring/Alerting
● Process events to generate graphs
● Riemann is an excellent tool for generating alerts
from event streams
● Generate graphs as close to realtime as possible
– Developers doing rollouts know that something else is
changing
– Any major problems will be caught really quickly
● Isolation of changes means that you can track
longer term effects of each change