AAA Automated Testing

AAA Automated Testing… For AAA Games Francesco Carucci Technical Director

How much does a game cost?

Development costs spiraling up

EA Exec: “Cost of games is crazy!”

Some examples:

Killzone 2: > 40 M$

Halo 3: > 30 M$

GTA 4: > 100 M$ (rumored)

Clearly, this is not sustainable in the long term

Development cost partitioning

To keep it simple:

Producing assets

Writing code (game code, technology, tools)

Debugging and maintaining code

Code dictates cost of producing assets

Better tools means less time to create assets

More efficient code means less time spent optimizing assets

Solutions?

Better production frameworks

SCRUM or some variation of it

More experienced teams

Higher senior/junior ratio actually costs less

Better tools (3 rd party engines, off the shelf packages)

Out sourcing (yay for assets, nay for code)

Higher quality code

Code is not the King

But it’s a first class citizen and we want to keep its cost down

Higher code quality reduces Cost Of Ownership

Less bugs

Less over engineering

Higher code quality means code easier to change and iterate upon

Higher code quality means more time to be spent on polishing the game

Higher code quality means less money spent to make a game!

And a bigger chance to ship on time and hit that market sweet spot

Change Prevention Process

Many teams follow a change management process whose main goal is to prevent “feature creep”

It’s effectively a Change Prevention Process

But you can’t design “fun” up front

Games must be developed iteratively until “fun” is found…

So?

Embrace Change 

Cost of Change

Game Design requirements change

“ The aim system is good, but I would like it more peachy, you know a bit more of this, but slightly less of that, more rocking, less oompfh. Can you change it?”

Traditionally Cost Of Change rises sharply over time

“ No, I can’t change it, It would break X and Y and probably Z too, I don’t really know”

We want to keep Cost Of Change as flat as possible

“ Yes we can!”

Enter Automated Testing…

Automated Testing has been a broadly accepted tool for several decades in the Software Industry aimed at reducing cost of developing working code

In fact… (warning: Boring Slides Ahead)

Assessing test-driven development at IBM

“ In a software development group of IBM Retail Store Solutions, we built a non-trivial software system based on a stable standard specification using a disciplined, rigorous unit testing and build approach based on the test- driven development (TDD) practice. Using this practice, we reduced our defect rate by about 50 percent compared to a similar system that was built using an ad-hoc unit testing approach . The project completed on time with minimal development productivity impact. Additionally, the suite of automated unit test cases created via TDD is a reusable and extendable asset that will continue to improve quality over the lifetime of the software system. The test suite will be the basis for quality checks and will serve as a quality contract between all members of the team.” http://collaboration.csc.ncsu.edu/laurie/Papers/MAXIMILIEN_WILLIAMS.PDF http://portal.acm.org/citation.cfm?id=776892

An initial investigation of test driven development in industry

Simposium on Applied Computing

“ Test Driven Development (TDD) is a software development practice in which unit test cases are incrementally written prior to code implementation. In our research, we ran a set of structured experiments with 24 professional pair programmers. One group developed code using TDD while the other a waterfall-like approach. Both groups developed a small Java program. We found that the TDD developers produced higher quality code, which passed 18% more functional black box test cases. However, TDD developer pairs took 16% more time for development. A moderate correlation between time spent and the resulting quality was established upon analysis. It is conjectured that the resulting high quality of code written using the TDD practice may be due to the granularity of TDD, which may encourage more frequent and tighter verification and validation. Lastly, the programmers which followed a waterfall-like process often did not write the required automated test cases after completing their code, which might be indicative of the tendency among practitioners toward inadequate testing. This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code development ” http://portal.acm.org/citation.cfm?id=952753&dl=GUIDE&coll=GUIDE&CFID=46811546&CFTOKEN=92849791

Google

“ Standard Google practices include unit tests and functional tests, continuous builds, last known good build, release branches with authorized bug fix check-ins, focused teams for release engineering, production engineering and QA, bug tracking and logging of production runs.” http://agile2007.agilealliance.org/index.php%3Fpage=sub%252F&id=713.html

Professionalism and Test-Driven Development

“ Test-driven development is a discipline that helps professional software developers ship clean, flexible code that works, on time. In this article, the author discusses how test-driven development can help software developers achieve a higher degree of professionalism” http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F52%2F4163008%2F04163026.pdf&authDecision=-203

Unit Testing in Games

“ In our experience, the introduction of automated tests and continuous integration makes development teams more efficient and results in more reliable, and often quite simply, better software. Additionally, it reduces the pressure and workload on development teams by reducing the effort for manual testing, and allows bugs to be found earlier in the development process. Certainly, automated tests alone won't make your game a hit. But almost as certainly, they will make life easier for developers, artists, project leaders, producers and even players.” http://www.gamasutra.com/features/20050329/roken_pfv.htm

Unit Testing in Games (contd…)

“ Test-driven development can be a very effective development technique. We have successfully applied it to game development in a variety of situations, and we’re convinced of the many benefits it has provided us. Right now, the idea of writing code without writing tests first feels quite alien to most of us, and we treat TDD like the scaffolding in building construction: a necessary tool that will not be shipped to the customer but that helps tremendously during development” http://gamesfromwithin.com/?p=50

Unit Testing in Games (contd…)

“ Wir haben in den letzten fünf Jahren ausschließlich positive Erfahrungen mit automatisierten Tests und Continuous Integration gesammelt. Die automatisierten Tests sichern Stabilität auf hohem Niveau, während Continuous Integration zusätzlich eine deutliche Arbeitserleichterung darstellt. Der Aufwand für die Einrichtung und Wartung der Verfahren ist überschaubar und hat sich schnell amortisiert. Wenn man die anfängliche Skepsis überwindet, möchte man unserer Erfahrung nach diese Art der automatisierten Qualitätssicherung sehr schnell nie mehr missen.” http://www.igda.dimajix.net/fileadmin/ressources/report-03/07_roeken_summary.pdf

What’s an Automated Unit Test

A snippet of code that exercise a Unit Under Test (method of a class), assert its correctness and is executed automatically every time the system is built

Running unit test is effectively a compilation step

Compiler checks if the code is syntactically correct

Unit tests check if the code is semantically correct

Test First vs Test Last

Writing automated tests is accepted to produce higher quality code at lower cost

More tests == less time spent debugging

Should tests be written after production code? (Test-Last)

Should tests be written before production code? (Test-First)

“ This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code development”

Test Driven Development == Test-First

Test-Driven Development

Write a failing test before the code is written

Write the simplest code that could possibly make the test pass

Refactor the code to eliminate all possible duplications and code smells

Red Green Refactor

That’s the coding rhythm

A Failing Test

First write a failing test:

CRY_TEST (FireButtonPressAndRelease) { StartFire(); ASSERT_IS_ TRUE (IsFiring()); }

The test is dictating the interface and the contract of the unit being written:

When StartFire method is invoked, the weapon is firing

… then implement the minimal code to make it pass

bool IsFiring() const { return true; }

Another Failing Test

Write another failing test to dictate more behavior:

CRY_TEST (FireButtonPressAndRelease) { StartFire(); St op Fire(); ASSERT_IS_ FALSE (IsFiring()); }

When StartFire method is invoked, and then StopFire method is invoked, the weapon is not firing.

The actual word explanation is redundant, the whole information needed is in the code itself in readable form!

Another Failing Test (contd.)

Minimal implementation:

bool StartFiring() {

m_isFiring = true; } bool StopFiring() { m_isFiring = false; } bool IsFiring() const

{ return m_isFiring; }

How does it work

Tests are small, simple, straightforward and test only one condition

Long and complicated tests must be avoided

Tests are self documenting

Tests describe the design of the code to be written

Contract

Invariants

Post-conditions

Calling conventions

A TDD cycle should take few minutes (often less than a minute)

30 minutes to write a test means a bad test

Break bigger tests in smaller steps (help test coverage)

Begin with baby steps, increase step size when becoming more comfortable

Why TDD?

Small and clearly defined steps lead to the final implementation

No over engineering: write only the code that makes failing test pass

Minimal maintenance

Less code for the same needed functionality (including tests!)

Less time needed to change the code when requirements change

Less coupling between units

Less cost to maintain units

Tests are a by product of design and code construction

They keep contracts tested automatically through the life of the project

Code is easier to modify, broken contracts easier to discover

Defects and design inconsistencies are found early on

It costs less to fix fewer bugs!

When TDD is not appropriate

Exploratory coding, throw away prototype code to quickly explore different solutions for a problem

But the proper solution must be implemented test-driven to reduce cost

Code wrapping around existing libraries

GUI layer

Very low level code dealing with the hardware

Excuses for not Testing

Excuses for not Testing (contd.)

… writing tests takes time and this time is stolen from writing code

People forget about the time spent debugging and maintaining code which is much larger than the time spent writing it

Tests written test-first are part of the design process, that has to be done anyway. TDD is just a formalized way of designing code. Overall it takes less time to produce working code test-driven

When the code base is covered, modifying code is safer and takes less time cause of regression provided by the automated tests

By writing tests first, fewer bugs are experienced by working with generally tested code (no huge amount of time spent debugging)

… tests can’t cover everything

Some test coverage is still better than no test coverage at all

Legacy code shouldn’t be tested unless it’s being refactored

Rome wasn’t built in a day, but it was built eventually (and conquered the known world  )

… but this code is temporary and I will change it later, I don’t need to test it now

All code will be modified soon or later: making code simpler to change is the main reason to write it test-driven: reduce over engineering to the minimum, keep it simple

Requirements change: modify the affected tests and production code. If unit tested, the code base is more decoupled and easier to modify with less side effects

… game development is different: automated testing might not work

Game development as a whole is different…

… but writing code is the same if not simpler

Game code is mostly algorithmic, which is easier to test than GUI code, for example

An A* algorithm is mostly the same, be it in a game or in a web crawler

Game code changes often : more changes means more pressure for automated tests to protect against side effects

… games are about fun: you can’t capture fun with automated tests

But you can capture bugs, and less time spent debugging means more time looking for the “fun” part

Writing a test for a bug fix ensures that bug will not resurface later on

30% of bug fixes create a new bug caused by side effects

… this is AAA Game, it’s too risky to do automated testing on it

AAA game means highly polished, highly technologic, great game that delivers great fun

In order to achieve AAA, the most efficient production tools are needed, Automated Testing is an efficient tool

It’s too risky to NOT do automated testing on a AAA game

Be among the first to gain the competitive advantage and improve the Industry

A Slice of a bigger Pie

Automated tests (even test-driven) don’t solve all problems

Chatting up a girl will still be a problem for a programmer 

Automated tests must be part of a wider agile framework including:

Continuous integration

Team code ownership

Daily deliverables

Simple code that works

Functional and Integration testing

These practices work together and strengthen each other with the goal of producing more functionalities in less time .

What’s next?

More automated tests beyond Unit testing

Functional testing

Test more units functioning together at higher level

Requires more time to build a Functional Testing Framework

Can capture bugs generated by the interaction of different units

Smoke testing

Load different levels after every check-in looking for crashes

Run scripted sequences looking for crashes

Generate random inputs looking for crashes

What’s next? (contd.)

Asset testing

Load different assets from the engine to check for crashes and correctness

Animations, Meshes, Textures

Example: “ a Mesh must be smaller than 5000 polygons if it’s a character, 20000 polygons if it’s a building and must have a physics mesh smaller than 100 polygons associated”