My first experience with incorporating live user test/crowd testing in a real project.

1. Initial Experiences with Live User Test for Games
I have always been interested in Live User Test since I first heard about it, but I
have never had the opportunity to try it out until now. Last year I wrote an
article about my theoretical thoughts on Live User Test [1], and I thought I would
follow up on that now that I have some practical experience.
We had problem reported by a fraction of our customers that we had not found
in our internal testing, and could not reproduce once we knew about it. Either it
was intermittent or environment or configuration dependent. Developers were
analyzing the problem, but we also had the option of using the services of a
crowd testing (live user test) company, so we jumped at the opportunity.
I had my preconceived views of that it would require a lot of time from me to set
it up, but I was happily surprised when it took me only 5 minutes or so to set
everything up on my end. This was because the problem we had was quite easy
to find under the right circumstances, and the tests I had to request were very
easy to write down.
An hour after I had sent in my request, I received a link to an excel sheet, which
immediately started to get populated by tester names. I saw my test cases
written in different columns, and the testers, who had already downloaded the
build I had sent, immediately started testing.
All in all 150 testers from different countries with different environments,
configurations, devices and operating systems executed the 4 tests I had sent
them, and out of those 150 around 5% actually had the problems our customers
had.
Suddenly we had a lot more information. We knew something about under which
circumstances the problem occurred.
When our developers produced a fix, it was easy to send it to the 5% of the
crowd testers, who had experienced this problem, to see if the fix worked for
them.
After this had been verified we could send the fix out to all our customers and
had finally solved the problem.
The experience I had with crowd testing confirmed my prior beliefs in how to
effectively use this form of testing.
Small, focused, easy-to-understand test scopes with a clear objective is key to
avoid unnecessary overhead, such as going through large amounts of bugs that
have already been reported or are working as designed. As I wrote in my
previous article I think it is most suited for finding intermittent bugs, bugs that
are dependent on environment or configuration, bugs which require a lot of

2. simultaneous users or bugs which require certain combinations that can be hard
to cover in a normal test scope.
I was positively surprised by the whole experienced, and it worked much
smoother and more effectively than I had thought. I will definitely try to use live
user testing more in the future, when the circumstances are right.
References
[1] Initial Thoughts on Live User Testing for Games
http://www.gamasutra.com/blogs/JohanHoberg/20141023/228441/Initial_tho
ughts_on_live_user_testing_for_games.php