This document discusses how Code Hunt crowdsources code and processes through an educational gaming platform. It began as dynamic symbolic execution tool Pex that was adapted into a game called Coding Duel. This evolved into Code Hunt, an online platform where users write code in the browser to solve puzzles. Code Hunt provides test cases to give feedback until the user's code matches the secret implementation. It has been used by hundreds of thousands of students to learn programming in a game-like environment. The document argues that Code Hunt can scale crowdsourcing of code and processes while identifying top coders through competitive coding contests hosted on the platform. Code Hunt data from these contests has been publicly released to further research.

1. Crowdsourcing Code and
Process via Code Hunt
Tao Xie, Judith Bishop, Nigel Horspool, Nikolai
Tillmann, Jonathan de Halleux
University of Illinois at Urbana-Champaign, USA;
Microsoft Research, USA; University of Victoria, Canada
https://www.codehunt.com

2. Testing Tool  Educational Gaming
Dynamic Symbolic Execution (Pex)
Pex for Fun/Code Hunt:
Interactive Gaming for
Teaching and Learning
Support
http://research.microsoft.com/pex/ http://pex4fun.com
https://www.codehunt.com

3. http://research.microsoft.com/pex/

4. Pex is Part of Visual Studio 2015!
• As new feature of “Smart Unit Tests”
http://www.visualstudio.com/en-us/news/vs2015-preview-vs#Testing
Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation
Tool to Practice: From Pex to Fakes and Code Digger. In ASE 2014, Experience Papers.
http://web.engr.illinois.edu/~taoxie/publications/ase14-pexexperiences.pdf

5. Coding Duels
http://pex4fun.com/
Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning
Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE.
http://web.engr.illinois.edu/~taoxie/publications/icse13see-pex4fun.pdf

6. Coding Duel Competition
@ICSE 2011
http://pexforfun.com/icse2011

7. Code Hunt: Resigned As Game
https://www.codehunt.com/

8. Gameplay
1. User writes code in browser
2. Cloud analyzes code – test cases show differences
As long as there are differences: User must adapt code, repeat
When they are no more differences: User wins level!
code
test cases

9. Behind the Scene of Code Hunt
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
9

10. It’s a game!
• iterative gameplay
• adaptive
• personalized
• no cheating
• clear winning criterion
code
test cases

11. Towards a Course Experience

12. Code Hunt as Crowdsourcing Platform
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
12
Take 1: Code

13. Code Hunt as Crowdsourcing Platform
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
13
Take 2: Process class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}

14. Code Hunt as Crowdsourcing Platform
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
14
Take 3: Code (w/o secret)
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Passing Player Contributed
Implementations
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Judge
Metrics

15. Why Code Hunt as Crowdsourcing
Platform?

16. Why Code Hunt as Crowdsourcing
Platform?
Crowdsourcing Scale
& Open Data

17. Coding Duels
1,667,521 clicked 'Ask Pex!'
http://pex4fun.com/

18. 2014 Beauty of Programming Contest
Code Hunt can identify
top coders

19. Code Hunt - the APCS (default) Zone
• Opened in March 2014
• 129 problems covering the Advanced Placement
Computer Science course
• By August 2014, over 45,000 users started
• By now, over 725,000 users
0
10000
20000
30000
40000
50000
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.101.111.121.131.141.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8
Players
Sector and Level
APCS Zone, First three sectors, 45K to 1K

20. Code Hunt Usage
Code Hunt has had several hundred thousands of
users since launch in March 2014
Stats from Visual Studio Analytics over the period
9/19/14 and 10/20/14 indicate 1,150 users a day
Stickiness (loyalty) is very high

21. APCS 725,199
ICSE 627
BoPQuali 13,778
BopA 1,017
BopB 1,103
BopSemi 1,164
CSTA 61
TEALS 14
BootCamp 30
Laser Sept 12
Polska Oct 344
ImCupJan 542
ImCupDec 222
ImCupNov 379
ImCupOct 831
ImCupSept 257
Total Try
Count
Average Try
Count
Max Try
Count
Total
Solved
Users
13374 363 1306 1581
Size of Code Hunt Data (so far)
For ImCupSept
257 users x 24 puzzles x approx. 10 tries =
about 13,000 programs
#users

22. Code Hunt Data Released to the Public!
http://research.microsoft.com/codehuntcommunity/
https://github.com/Microsoft/Code-Hunt
The currently released data set contains the programs written by students
(only) worldwide during a contest over 48 hours. There are approximately 250
users, 24 puzzles and about 13,000 programs.
Judith Bishop, Nigel Horspool, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Code Hunt:
Experience with Coding Contests at Scale. In ICSE 2015 JSEET.
http://web.engr.illinois.edu/~taoxie/publications/icse15jseet-codehunt.pdf
http://research.microsoft.com/chese2015/

23. Summary: Testing Tool Educational Gaming
DSE/Pex
Pex for Fun/Code Hunt:
Interactive Gaming for
Teaching and Learning
Support
http://pex4fun.com
https://www.codehunt.com
http://research.microsoft.com/codehuntcommunity/
https://github.com/Microsoft/Code-Hunt

24. Testing Tool  Educational Gaming
Support
http://research.microsoft.com/pex/ http://pex4fun.com
Q & A
Thank you!
http://research.microsoft.com/codehuntcommunity/
https://github.com/Microsoft/Code-Hunt