Your SlideShare is downloading. ×
0
Thursday, March 18, 2010
Lua in Testing


                                            Paul Du Bois
                                 Double Fine Pro...
About us

         » Paul: Senior Programmer
         » Bert: Software Test Engineer
         » RoBert:
           Robot b...
120-second pitch

                 » Unit testing is well understood
                 » “But how do we test game logic…”
 ...
120-second pitch
         The result


         » Framework for writing very high-level
           code to exercise game
 ...
120-second pitch
         The result


         » Everyone at Double Fine loves RoBert
              (even though it gives...
120-second pitch
         The result




                           Demo time!




Thursday, March 18, 2010
Overview of talk

         » Motivation
         » Implementation
         » Uses and examples




Thursday, March 18, 2010
Problem summary




Thursday, March 18, 2010
Problem summary

         » Brütal Legend is big
         » …big technical challenge
         » …big design
         » …bi...
Problem summary

         » Double Fine is small
         » Test team is very small




Thursday, March 18, 2010
¨
                    Implementation

Thursday, March 18, 2010
Preëxisting Tech

         » Console, networked
         » Input abstraction
         » Reflection
         » Lua 5.1




T...
Reflection

           Entity A02_Headbanger2F3

            CoPhysics        CoController   CoDamageable
           Pos:
 ...
Reflection
      Entity* e = ...;
      Co* co = e->GetComponent(“CoDamageable”);

      Class* cl = co->GetClass();
      ...
Lua implementation

         » Class system with inheritance
         » Wrapper for Entity instances
         » Smidgen of...
Lua and Tests
         » Class system with simple inheritance
         » See Programming in Lua § 13.4.1

          class ...
Lua and Entities

     player.CoDamageable.Health


     Entity* e = ...;
     Co* co = e->GetComponent(“CoDamageable”);

...
Lua and Entities

     player.CoDamageable.Health


                     Lua table               Entity A02...

          ...
Lua and Entities

     player.CoDamageable.Health

     player -> { __cpp_ref = 0x1ea7f00,
                 <mt>      = en...
Lua and Entities

     player.CoDamageable.Health

     player -> { __cpp_ref = 0x1ea7f00,
                 <mt>      = en...
Lua and Entities

     player.CoDamageable.Health

     player -> { __cpp_ref = 0x1ea7f00,
                 <mt>      = en...
Lua and Entities
         » Entity keeps table alive



                     Lua table               Entity A02...

      ...
Lua and Entities
         » Entity may be deleted at will
         » A destructor clears out out _cpp_ref

               ...
Lua and Tests


         function Class:waitForActiveLine(self, ent)
           while true do
             self:sleep(0)
 ...
Lua and Tests

         » Test can function as input source
         » Mutate a state block
         » Use blocking calls ...
Example: providing input
          -- push some button for time t1

          self.input.buttons[btn] = true
          sel...
Example: simple mission
     function Class:Run()
       function fightSpiders(entity)
         self:attackSmallSpiders()
...
Example: reproduce a bug
          function Class:Run()
            function waitForActiveLine()
              while true ...
Role of the human

         » Bot farm means more time writing bugs
         » Half time writing new tests, updating old
 ...
̊
           Uses and Examples

Thursday, March 18, 2010
Uses and Examples

         » Does the game start/quit/leak memory?
         » Do these entities spawn properly?
         ...
Uses and Examples

         » Can player interact with this unit?
         » Can bot fly across the world without the
     ...
Uses and Examples

         » Can I complete each contact's mission?
         » Can I successfully fail the mission?
     ...
Diagnostic “tests”

         » What is our memory usage as a function
           of time?
         » How does it change fr...
Thursday, March 18, 2010
Diagnostic “tests”

         » What does our performance look like as a
           function of time?
         » How does i...
Thursday, March 18, 2010
Non-test tests

         » Reproduce tricky bugs
         » Typically involve feedback between test
           and program...
Use by programmers

         » Pre-checkin verification
         » Soak testing for risky changes
         » Can use Debug ...
Thursday, March 18, 2010
Use by designers

         » Write a series of balance “tests”
         » Throw permutations of unit groups at each
      ...
Use by artists

         » They don’t run it themselves…
         » …but they do see it running
         » See parts of th...
Takeaway
          Why was this project successful?
         » Rapid development: doing a prototype
           was cheap a...
Fill out forms!

Thursday, March 18, 2010
‘’
                           Questions?

                           dubois@doublefine.com
Thursday, March 18, 2010
Number of bugs found
               Date through               bot           total


                   2006-05-01
       ...
Number of bugs found

         » Raw bug count undersells RoBert
         » Query didn’t catch all RoBert bugs
         » ...
Types of bugs found

         » Almost all crashes and asserts
         » Middleware bugs
         » Logic bugs manifest a...
What we test

         » Most tests merely exercise behavior
         » Unsuccessful at verifying behavior
         » Corr...
What we don’t test

         » No testing of visuals
         » Limited testing of performance
         » Specific behavior...
Problems and future work

         » Big tests can take hours to complete
         » Still a lot of human-required work
  ...
Our takeaway

         » Doesn’t replace a test team
         » Does take tedious work off their plate
         » Hillclim...
Upcoming SlideShare
Loading in...5
×

Robotic Testing to the Rescue - Paul Dubois (DoubleFine)

1,922

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,922
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Robotic Testing to the Rescue - Paul Dubois (DoubleFine)"

  1. 1. Thursday, March 18, 2010
  2. 2. Lua in Testing Paul Du Bois Double Fine Productions Thursday, March 18, 2010
  3. 3. About us » Paul: Senior Programmer » Bert: Software Test Engineer » RoBert: Robot brainchild Automated tester Thursday, March 18, 2010
  4. 4. 120-second pitch » Unit testing is well understood » “But how do we test game logic…” » We implemented a prototype Thursday, March 18, 2010
  5. 5. 120-second pitch The result » Framework for writing very high-level code to exercise game » Runs on any idle devkit » Used directly by ❖ Test ❖ Gameplay, System programmers ❖ Designers Thursday, March 18, 2010
  6. 6. 120-second pitch The result » Everyone at Double Fine loves RoBert (even though it gives them bugs) » Game would be significantly smaller without it » Permanent part of our process Thursday, March 18, 2010
  7. 7. 120-second pitch The result Demo time! Thursday, March 18, 2010
  8. 8. Overview of talk » Motivation » Implementation » Uses and examples Thursday, March 18, 2010
  9. 9. Problem summary Thursday, March 18, 2010
  10. 10. Problem summary » Brütal Legend is big » …big technical challenge » …big design » …big landmass Thursday, March 18, 2010
  11. 11. Problem summary » Double Fine is small » Test team is very small Thursday, March 18, 2010
  12. 12. ¨ Implementation Thursday, March 18, 2010
  13. 13. Preëxisting Tech » Console, networked » Input abstraction » Reflection » Lua 5.1 Thursday, March 18, 2010
  14. 14. Reflection Entity A02_Headbanger2F3 CoPhysics CoController CoDamageable Pos: (3,4,5) State: Idle Health: 30 Mass: 10 Ragdoll: true Thursday, March 18, 2010
  15. 15. Reflection Entity* e = ...; Co* co = e->GetComponent(“CoDamageable”); Class* cl = co->GetClass(); Attribute* a = cl->GetAttribute(“Health”); Any any = a->GetValueAny(co); float health = any.GetAs<float>(); Thursday, March 18, 2010
  16. 16. Lua implementation » Class system with inheritance » Wrapper for Entity instances » Smidgen of coroutines » No debugger ☹ Thursday, March 18, 2010
  17. 17. Lua and Tests » Class system with simple inheritance » See Programming in Lua § 13.4.1 class = { Run = <function>, Teardown = <function>, bFastSpeed = true } instance = { <mt> = { __index = class } } Thursday, March 18, 2010
  18. 18. Lua and Entities player.CoDamageable.Health Entity* e = ...; Co* co = e->GetComponent(“CoDamageable”); Class* cl = co->GetClass(); Attribute* a = cl->GetAttribute(“Health”); Any any = a->GetValueAny(co); float health = any.GetAs<float>(); Thursday, March 18, 2010
  19. 19. Lua and Entities player.CoDamageable.Health Lua table Entity A02... CoScript _cpp_ref: 0x1ea7f000 Table ref: <opaque> Thursday, March 18, 2010
  20. 20. Lua and Entities player.CoDamageable.Health player -> { __cpp_ref = 0x1ea7f00, <mt> = ent_mt } Thursday, March 18, 2010
  21. 21. Lua and Entities player.CoDamageable.Health player -> { __cpp_ref = 0x1ea7f00, <mt> = ent_mt } player.CoDamageable -> { __ent = player, __comp = ‘CoDamageable’, <mt> = comp_mt } Thursday, March 18, 2010
  22. 22. Lua and Entities player.CoDamageable.Health player -> { __cpp_ref = 0x1ea7f00, <mt> = ent_mt } player.CoDamageable -> { __ent = player, __comp = ‘CoDamageable’, <mt> = comp_mt } Player.CoDamageable.Health -> 60 Thursday, March 18, 2010
  23. 23. Lua and Entities » Entity keeps table alive Lua table Entity A02... CoScript _cpp_ref: 0x1ea7f000 Table ref: <opaque> Thursday, March 18, 2010
  24. 24. Lua and Entities » Entity may be deleted at will » A destructor clears out out _cpp_ref Lua table Entity A02... CoScript _cpp_ref: false Table ref: <opaque> Thursday, March 18, 2010
  25. 25. Lua and Tests function Class:waitForActiveLine(self, ent) while true do self:sleep(0) if not ent.CoVoice.HasActiveVoiceLine then return end end end Thursday, March 18, 2010
  26. 26. Lua and Tests » Test can function as input source » Mutate a state block » Use blocking calls to make API convenient » Manipulate joystick in “world coordinates” Thursday, March 18, 2010
  27. 27. Example: providing input -- push some button for time t1 self.input.buttons[btn] = true self:sleep(t1) self.input.buttons[btn] = false -- move towards world-space pos x,y,z self.input.joy1 = test.GetInputDir(x,y,z) Thursday, March 18, 2010
  28. 28. Example: simple mission function Class:Run() function fightSpiders(entity) self:attackSmallSpiders() self:killHealerSpiders() self:basicFightFunc(entity) self:waypointAttack( "P1_050_1", "Monster", 40, fightSpiders) self:attackEntitiesOfTypeInRadius( "Monster", 50, fightSpiders) self:attackBarrier("A_WebBarrierA", 100) self:waypointTo{"P1_050_ChromeWidowLair"} Thursday, March 18, 2010
  29. 29. Example: reproduce a bug function Class:Run() function waitForActiveLine() while true do self:sleep(0) if not player.CoVoice.HasActiveVoiceLine then return streams = sound.GetNumStreams() while true do game.SayLine( 'MIIN001ROAD' ) game.SayLine( 'MIIN001ROAD' ) waitForActiveLine() if sound.GetNumStreams() > streams then self:sleep(1) self:ASSERT(sound.GetNumStreams() <= streams) Thursday, March 18, 2010
  30. 30. Role of the human » Bot farm means more time writing bugs » Half time writing new tests, updating old tests, writing/regressing bugs » Half time on infrastructure work Thursday, March 18, 2010
  31. 31. ̊ Uses and Examples Thursday, March 18, 2010
  32. 32. Uses and Examples » Does the game start/quit/leak memory? » Do these entities spawn properly? » Can this unit pathfind properly? Thursday, March 18, 2010
  33. 33. Uses and Examples » Can player interact with this unit? » Can bot fly across the world without the game crashing? » Can I go to each mission contact and talk to them? Thursday, March 18, 2010
  34. 34. Uses and Examples » Can I complete each contact's mission? » Can I successfully fail the mission? » Multiplayer! Thursday, March 18, 2010
  35. 35. Diagnostic “tests” » What is our memory usage as a function of time? » How does it change from build to build? » Where are the danger spots? Thursday, March 18, 2010
  36. 36. Thursday, March 18, 2010
  37. 37. Diagnostic “tests” » What does our performance look like as a function of time? » How does it change from build to build? » What is it like in certain troublesome scenes? Thursday, March 18, 2010
  38. 38. Thursday, March 18, 2010
  39. 39. Non-test tests » Reproduce tricky bugs » Typically involve feedback between test and programming » Guess at the fail case, try to exercise it Thursday, March 18, 2010
  40. 40. Use by programmers » Pre-checkin verification » Soak testing for risky changes » Can use Debug builds! Thursday, March 18, 2010
  41. 41. Thursday, March 18, 2010
  42. 42. Use by designers » Write a series of balance “tests” » Throw permutations of unit groups at each other » Print out results in a structured fashion » Examined by a human for unexpected results Thursday, March 18, 2010
  43. 43. Use by artists » They don’t run it themselves… » …but they do see it running » See parts of the game they normally wouldn’t » Notice things that don’t look right Thursday, March 18, 2010
  44. 44. Takeaway Why was this project successful? » Rapid development: doing a prototype was cheap and low-risk » Lua doesn’t crash the game » Coroutines and higher-order functions let you write powerful, concise code » Performance not an issue Thursday, March 18, 2010
  45. 45. Fill out forms! Thursday, March 18, 2010
  46. 46. ‘’ Questions? dubois@doublefine.com Thursday, March 18, 2010
  47. 47. Number of bugs found Date through bot total 2006-05-01 2006-09-01 2007-01-01 2007-05-01 2007-09-01 2008-01-01 2008-05-01 2008-09-01 2009-01-01 (to date) 2009-05-01 (projected) 2009-05-01 0 750 1,500 2,250 3,000 Thursday, March 18, 2010
  48. 48. Number of bugs found » Raw bug count undersells RoBert » Query didn’t catch all RoBert bugs » Not all problems found get entered Thursday, March 18, 2010
  49. 49. Types of bugs found » Almost all crashes and asserts » Middleware bugs » Logic bugs manifest as “Bot stuck in mission” failures » Complementary to bugs found by human testers Thursday, March 18, 2010
  50. 50. What we test » Most tests merely exercise behavior » Unsuccessful at verifying behavior » Correctness of test is an issue Thursday, March 18, 2010
  51. 51. What we don’t test » No testing of visuals » Limited testing of performance » Specific behaviors, game logic Thursday, March 18, 2010
  52. 52. Problems and future work » Big tests can take hours to complete » Still a lot of human-required work » Bot cheats a lot Thursday, March 18, 2010
  53. 53. Our takeaway » Doesn’t replace a test team » Does take tedious work off their plate » Hillclimbing development strategy worked well » Very curious what others are doing! Thursday, March 18, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×