The document discusses testing strategies that involve cross-functional teamwork between development, operations, and testing roles. It advocates for developers to write checks during development, for operations teams to provide testing environments and insights, and for testing to be a collaborative effort across teams using techniques like Twitter's Diffy tool. The goal is to move beyond silos and enable more learning through production-like testing.
21. “Testing is the process of evaluating a product
by learning about it through exploration and
experimentation, which includes to some degree:
questioning, study, modeling, observation,
inference, etc.”
-James Bach
23. Ops teams can help provide quick access to
testing environments with:
• Arbitrary versions of services
• Access to the service’s infrastructure, logs,
deployment artifacts
• Potentially chaotic monkeys
Enabling learning
Software Engineer with Booz Allen
Talk about my journey into testing, the challenges in the industry I’m seeing today,
Ultimately a better vision for the future
In 2010
My first project was a pretty big – 20 developers, > 1 million registered users across multiple agencies
I like to think that had an intuition that going fast (getting software out the door) was good – we were agile, we did scrums, story point estimations, all that good stuff
But we always had something holding us back from going fast –
Which was testing, specifically end to end tests
Testing is generally slow for most teams, but for us it was even more complicated
We had one code base that we were using for multiple client demands, that means we were doing multiple runs of regression tests for every change
So how do we move testing faster?
Shove more and more people into a room and make them work day and night
Removing windows was attempted
Watching this room grow and grow with people, was really sad
Became very interested in how we could test faster
Found some tools that worked for me -- spent some late nights at the computer
And I thought, I had developed a system to that would accomplish what that room full of people were doing in an eighth of the time, with more consistency, and on-demand – great done.
And it turns out we weren’t alone in this problem, so I started helping others accomplish this
2 years, 6 different clients, over 20 projects later … I’m convinced we’re not done.
Go back to the notion of going fast is better
Can we go fast in an organization like this ?
Silos are common place in development organizations
“Doesn’t work in production? Must be your problem… you figure it out”
Silos create reward systems that don’t necessarily contribute to the optimization of the greater good – in this case delivering software
A better approach is to find ways to help one another – working as a team
Sports do this really well – there excellent examples of organizations that diligently break down reward systems that don’t contribute to the greater goal – scoring points
In football there are lots of individuals willing to make sacrifices against there personal stats to help the team win
This philosophy is commonly known today as DevOps
Creating a team that has a empathy for one another
“Doesn’t work in production? Whoa that’s weird, maybe its environmental, what does production look like, how can I replicate that on my local machine, so I can catch this earlier”
Teams that buy in to this philosophy tend to end up with development workflows like this
Boring operation tasks are automated by the dev team, ops teams are feeding developers with information
And now things get fast
Really fast…
And it turns out being fast isn’t just fun
The market has proven being fast if useful in real life
Amazon, github, get huge advantages by being able to move fast
But wait a minute, lets go back to that workflow
Not again… testing is holding us back
approval is a dirty word when it comes to fast
Lets see how we can do better
Well having ops and developers working together seemed to get us pretty far…
Lets try getting testers out of their silo
So how can they help?
This is the famous test automation pyramid (explain quickly)
Automating tests helps us go fast, and fast is good
But developers are already overbooked and don’t have time to build all of these tiers
Time for testers to the rescue, they can help us write the UI tests –
Lets see what happens
This doesn’t look right
But in my experience this is how most teams are positioned
And in fact its even worse than that, let me explain…
DevOps are having a great time, enjoying themselves
Here comes Testers trying to join the club
Oh no…
Things get slower…
A good tester can think of an infinite amount of scenarios to automate
but this is really expensive to automate – lots of development and maintenance efforts
They have a hard time adding to other tiers
Integration tests aren’t existed because dev teams want to depend on the UI tests
Unit tests are blamed heavily for bugs which leads to code coverage metrics
So how to fix this problem?
You could hire people like me… and we can help make this tier faster and less expensive to maintain
I think we can do better though
We need to redefine what testing means here
In order to keep our intended pyramid we need to do only a few tests that verify critical elements of application
And these tests need to be written and maintained by the people that know the source code best
Not good. Soo how to fix this? Well you can fire your testers and hire me to fix those UI based tests. Awesome, you all have my contact information are there any questions? Hmm, not satisfied, I know I’m not. So what can we do better ? Lets try another strategy.
So this idea wasn’t so great
I think this is a much better model
Checks are simple, quick, tests that provide a yes or no answer to an applications state
Developers need to be clever about putting in only a few checks that cover critical elements
In this model we can achieve fast tests but at a cost…
We are only doing a few checks instead of a large number of tests
We are going to leak bugs all over the place if we don’t add back in tests
So how can we work as a team to help testers provide us with more bugs faster?
Testing is really a process of learning
We’ve helped them a bit already with our series of checks, so they don’t have to waste time with the boring stuff
But we can do better
So if testing is all about learning, who knows more about the reality of the application in its native environment then your ops team ? The people in ops are already constantly striving to unearth as much data as they can about the actual workings of the application. Testers should be doing belly flops into this lake of data in order to unearth the deep questions that may be affecting the quality of your app.
Who better to help the process of learning than the team that holds all the data about the running of the application ?
This is where testers should be integrating with teams
How can ops enable learning ?
They can
provide access to information that is usually not accessible to the UI
With these kinds of information deeper insights can be made, and more bugs can be found faster.
New Relic, APM, Google analytics and others “operational tools” can be leveraged by testing activities as well, to provide more behind the scenes operations. They can be leveraged by the tester while they are exploring in a session or by giving historical data to drive session focus and interesting experiments.
It doesn’t have to be a one way street of Ops providing testers with information. Testers can help drive what data operations is recording. Here you can see Etsy monitoring their forumns for spikes in technical issues after each deploy. This information would be gold for testers to feedback into their exploratory testing session scheduling. Testers should strive to communicate the data sets that they could translate into more efficient testing sessions.
Lets walk through an example of how the team can work together to provide the speed of DevOps with high quality tested code.
Best explained by an example
Open source project aimed at identifying changes between versions of software – some changes may be good, some changes may be indicators of problems
Enables testing and checking much more efficiently than classic Selenium approaches
Testers can start to investigate surprising changes, and checks can be added to the end for rules that are less ambiguous
Contrast this to Selenium -- try asking a tester to utilize selenium tests, its not easy, the data provided isn’t there.
In the need for speed testers have been kicked around a lot
On the playing field they have always been the slow guys holding us back
But, in order to really go faster though we need to work as a team to help pick them up
We can do a lot better at working with testers, so that they can help us move as fast as we can.
