4Developers 2015: Gamedev-grade debugging - Leszek Godlewski

PROIDEA
PROIDEAPROIDEA
Gamedev-grade debugging
Leszek Godlewski, The Astronauts
Source: http://igetyourfail.blogspot.com/2009/01/reaching-out-tale-of-failed-skinning.html
● Engine Programmer, The Astronauts (Nov 2014 –
present)
– PS4 port of The Vanishing of Ethan Carter
● Programmer, Nordic Games (early 2014 – Nov 2014)
● Freelance Programmer (Sep 2013 – early 2014)
● Generalist Programmer, The Farm 51 (Mar 2010 –
Aug 2013)
Who is this guy?
Agenda
● How is gamedev different?
● Bug species
● Case studies
● Conclusions
StartStart Exit?Exit?
EndEnd
Yes
No
UpdateUpdate DrawDraw
How is gamedev different?
33 milliseconds
● How much time you have to get shit done™
– 30 Hz → 33⅓ ms per frame
– 60 Hz → 16⅔ ms per frame
EditorEditor
Level toolsLevel tools
Asset toolsAsset tools
EngineEngine
PhysicsPhysics
RenderingRendering AudioAudio
NetworkNetwork
PlatformPlatform
InputInput
Network
back-end
Network
back-end
GameGame
UIUI LogicLogic AIAI
Interdisciplinary working
environment
● Designers
– Game, Level, Quest, Audio…
● Artists
– Environment, Character, 2D, UI, Concept…
● Programmers
– Gameplay, Engine, Tools, UI, Audio…
● Writers
● Composers
● Actors
● Producers
● PR & Marketing Specialists
● … }
Tightly
woven
teams
Severe, fixed hardware
constraints
● Main reason for extensive use of native code
Different trade-offs
Robustness
Cost
Performance
Fun
/Coolness
Enterprise/B2B/webdev Gamedev
Indeterminism &
complexity
● Leads to poor testability
– Parts make no sense in isolation
– What exactly is correct?
– Performance regressions?
Source: https://github.com/memononen/recastnavigation
Aversion to general
software engineering
● Modelling
● Object-Oriented Programming
● Design patterns
● C++ STL
● Templates in general
● …
Agenda
● How is gamedev different?
● Bug species
● Case studies
● Conclusions
Source: http://benigoat.tumblr.com/post/100306422911/press-b-to-crouch
Bug species
General programming bugs
● Memory access violations
● Memory stomping/buffer overflows
● Infinite loops
● Uninitialized variables
● Reference cycles
● Floating point precision errors
● Out-Of-Memory/memory fragmentation
● Memory leaks
● Threading errors
Bad maths
● Incorrect transform order
– Matrix multiplication not commutative
– AB ≠ BA
● Incorrect transform space
Source: http://leadwerks.com/wiki/index.php?title=TFormQuat
Temporal bugs
● Incorrect update order
– for (int i = 0; i < entities.size(); ++i)
entities[i].update();
● Incorrect interpolation/blending
– Bad alpha term
– Bad blending mode (additive/modulate)
● Deferred effects
– After n frames
– After n times an action happens
– n may be random, indeterministic
Graphical glitches
● Incorrect render state
● Shader code bugs
● Precision
Source: http://igetyourfail.blogspot.com/2009/01/visit-lake-fail-this-weekend.html
Content bugs
● Incorrect scripts
● Buggy assets
Source: http://www.polycount.com/forum/showpost.php?p=1263124&postcount=10466
Worst part?
● Most cases are two or more of the
aforementioned, intertwined
Agenda
● How is gamedev different?
● Bug species
● Case studies
● Conclusions
Most material captured by
Case studies
Video settings not updating
Incorrect weapon after
demon mode
foreshadowing
Post-death sprint camera
anim
Corpses teleported on
death
Corpses teleported on
death
● In normal gameplay, pawns have simplified
movement
– Sweep the actor's collision primitive through the
world
– Slide along slopes,
stop against walls
Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
Corpses teleported on
death
● Upon death, pawns switch to physics-based
movement (ragdoll)
Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
Corpses teleported on
death (cont.)
● Physics bodies have separate state from the
game actor
– Actor does not drive physics bodies, unless
requested
– If actor is driven by
physics simulation,
their location is
synchronized to
the hips bone
body's
Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
Corpses teleported on
death (cont.)
● Idea: breakpoint in FarMove()?
– One function because world octree is updated
– Function gets called a gazillion times per frame �
– Terrible noise
● Breakpoint condition?
– Teleport from arbitrary point A to arbitrary point B
– Distance?
● Breakpoint sequence?
– Break on death instead
– When breakpoint hit, break in FarMove()
Corpses teleported on
death (cont.)
● Cause: physics body driving the actor with out-
of-date state
● Fix: request physics body state synchronization
to animation before switching to ragdoll
Weapons floating away
from the player
Weapons floating away
from the player
Weapons floating away
from the player
● Extremely rare, only encountered on consoles
– Reproduction rate somewhere at 1 in 50 attempts
– And never on developer machines �
● Player pawn in a special state for the
rollercoaster ride
– Many things could go wrong
● For the lack of repro, sprinkled the code with
debug logs
Weapons floating away
from the player (cont.)
● Cause: incorrect update order
– for (int i = 0;
i < entities.size();
++i)
entities[i].update();
– Player pawn forced to update after rollercoaster car
– Possible for weapons to be updated before player
pawns
● Fix: enforce weapon update after player pawns
Characters with “rapiers”
Characters with “rapiers”
● UE3 has ”content cooking” as part of game
build pipeline
– Redistributable builds are ”cooked” builds
● Artifact appears only in cooked builds
Characters with “rapiers”
(cont.)
● Logs contained assertions for ”out-of-bounds vertices”
● Mesh vertex compression scheme
– 32-bit float → 16-bit short int (~50% savings)
– Find bounding sphere for all vertices
– Normalize all vertices to said sphere radius
– Map [-1; 1] floats to [-32768; 32767] 16-bit integers
● Assert condition
– for (int i = 0; i < 3; ++i)
assert(v[i] >= -1.f && v[i] <= 1.f,
”Out-of-bound vertex!”);
Characters with “rapiers”
(cont.)
● v[i] was NaN
– Interesting property of NaN: all comparisons fail
– Even with itself
● float f = nanf();
bool b = (f == f);
// b is false
● How did it get there?!
● Tracked the NaN all the way down to the raw
engine asset!
Characters with “rapiers”
(cont.)
● Cause: ???
● Fix: re-export the mesh from 3D software
– Magic!
Meta-case: undeniable
assertion
Undeniable assertion
● Happened while debugging ”rapiers”
● Texture compression library without sources
● Flood of non-critical assertions
– For almost every texture
– Could not ignore in bulk �
– Terrible noise
● Solution suggestion taken from [SINILO12]
Undeniable assertion
(cont.)
● Enter disassembly
Undeniable assertion
(cont.)
● Locate assert message function call instruction
Undeniable assertion
(cont.)
● Enter memory view and look up the adress
– 0xE8 is the CALL opcode
– 4-byte address argument
Undeniable assertion
(cont.)
● NOP it out!
– 0x90 is the NOP opcode
Undeniable assertion
(cont.)
Incorrect player movement
Incorrect player movement
Incorrect player movement
● Recreating player movement from one engine
in another (Pain Engine → Unreal Engine 3)
● Different physics engines (Havok vs PhysX)
● Many nuances
– Air control
– Jump and fall heights
– Slope & stair climbing & sliding down
Incorrect player movement
(cont.)
● Main nuance: capsule vs cylinder
Incorrect player movement
(cont.)
● Switching our pawn collision to capsule-based
was not an option
● Emulate by sampling the ground under the
cylinder instead
● No clever way to debug, just make it ”bug out”
and break in debugger
Incorrect player movement
(cont.)
● Situation when getting stuck
● Cause: vanilla UE3 code sent a player locked
between non-walkable surfaces into the
”falling” state
● Fix: keep the player in the “walking” state
Incorrect player movement
(cont.)
● Situation when moving without player intent
● Added visualization of sampling, turned on
collision display
● Cause: undersampling
● Fix: increase radial sampling resolution
1) 2)
Blinking full-screen
damage effects
Blinking full-screen
damage effects
● Post-process effects are organized in one-way
chains
Blinking full-screen
damage effects (cont.)
● No debugger available to observe the PP chain
● Rolled my own overlay that walked and
dumped the chain contents
MaterialEffect 'Vignette'
Param 'Strength' 0.83 [IIIIIIII ]
MaterialEffect 'FilmGrain'
Param 'Strength' 0.00 [ ]
UberPostProcessEffect 'None'
SceneHighLights (X=0.80,Y=0.80,Z=0.80)
SceneMidTones (X=0.80,Y=0.80,Z=0.80)
…
MaterialEffect 'Blood'
Param 'Strength' 1.00 [IIIIIIIIII]
Blinking full-screen
damage effects (cont.)
● Cause: entire PP chain override
– Breakpoint in chain setting revealed the level script
as the source
– Overeager level designer ticking one checkbox too
many when setting up thunderstorm effects
● Fix: disable chain overriding altogether
– No use case for it in our game anyway
Incorrect animation states
Incorrect animation states
Incorrect animation states
Incorrect animation states
● Animation in UE3 is done by evaluating a tree
– Branches are weight-blended (either replacement or
additive blend)
– Sequences (raw animations) for whole-skeleton
poses
– Skeletal
controls for
fine-tuning of
individual
bones
Source: http://udn.epicgames.com/Three/AnimTreeEditorUserGuide.html
Incorrect animation states
(cont.)
● Prominent case for domain-specific debuggers
● No tools for that in UE3, rolled my own visualizer
– Walks the animation tree and dumps active branches
– Allows inspection of states, but not transitions
– Conventional
debugging
still required,
but greatly
narrowed
down
Incorrect animation states
(cont.)
● Animation bug “checklist”
● Inspect the animation state in slow motion
– Is the correct blending mode used?
● Inspect the AI and cutscene state
– Capable of full animation overrides
● Inspect the assets (animation sequences)
– Is the root bone correctly oriented?
– Is the root bone motion correct?
– Are inverse kinematics targets present and correctly placed?
– Is the mesh skeleton complete and correct?
Incorrect animation states
(cont.)
● Incorrect blend of reload animation
– Cause: bad root bone orientation in animation
sequence
● Left hand off the weapon
– Cause: left hand inverse kinematics was off
– Fix: revise IK state control code
● Left hand incorrectly oriented
– Cause: bad IK target marker orientation on weapon
mesh
Viewport stretched when
portals are in view
Viewport stretched when
portals are in view
● Graphics debugging is:
– Tracing & recording graphics API calls
– Replaying the trace
– Reviewing the renderer state and resources
● Trace may be
somewhat unreadable
at first…
Viewport stretched when
portals are… (cont.)
● Traces may be annotated for clarity
– Direct3D: ID3DUserDefinedAnnotation
– OpenGL:
GL_KHR_debug
(more info:
[GODLEWSKI01])
Viewport stretched when
portals are… (cont.)
● Quick renderer state inspection revealed that
viewport dimensions were off
– 1024x1024, 1:1 aspect ratio instead of 1280x720, 16:9
– Looks like shadow map resolution?
● Found the latest glViewport() call
– Shadow map code indeed
● Why wasn't the viewport updated for main scene
rendering?
Viewport stretched when
portals are… (cont.)
● Renderer state changes are expensive
– New state needs to be validated
– Modern graphics APIs are asynchronous
– State reading may requrie synchronization →
stalls
● Cache the current renderer state to avoid
redundant calls
– Cache ↔ state divergence → bugs!
Viewport stretched when
portals are… (cont.)
● Cause: cache ↔ state divergence
– Difference between Direct3D and OpenGL:
viewport dimensions as part of render target state,
or global state
● Fix: tie viewport dimensions to render target in
the cache
Black artifacts
Black artifacts
Black artifacts
Black artifacts
Black artifacts
Black artifacts
● First thing to do is to inspect the state
● Nothing suspicious found, turned to shaders
● On OpenGL 4.2+, shaders could be debugged in NSight…
● OpenGL 2.1, so had to resort to early returns from shader with
debug colours
– Shader equivalent of debug logs, a.k.a. ”Your Mum's Debugger”
● ”Shotgun debugging” with is*() functions
– isnan(), isinf()
● isnan() returned true!
Black artifacts (cont.)
● Cause: undefined behaviour in NVIDIA's pow() implementation
– Results are undefined if x < 0.
Results are undefined if x = 0 and y <= 0. [GLSL120]
– Undefined means the implementation is free to do whatever
● NVIDIA returns QNaN the Barbarian (displayed as black, poisoning all involved
calculations)
● Other vendors usually return 0
● Fix: for all pow() calls, clamp either:
– Arguments to their proper ranges
– Output to [0; ∞)
Mysterious crash
Mysterious crash
● Game in content lock (feature freeze) for a while
● Playstation 3 port nearly done
● Crash ~3-5 frames after entering a specific room
● First report included a perfectly normal callstack but no
obvious reason
● QA reassigned to another task, could not pursue more
● Concluded it must've been an OOM crash
Mysterious crash (cont.)
● Bug comes back, albeit with wildly different callstack
● Asked QA to reproduce mutliple times, including other platforms
– No crashes on X360 & Windows!
● Totally different callstack each time
● Confusion!
– OOM? Even in 512 MB developer mode (256 MB in retail units)?
– Bad content?
– Console OS bug?
– Audio thread?
– ???
Mysterious crash (cont.)
● Reviewed a larger sample of callstacks
● Most ended in dlmalloc's integrity checks
– Assertions triggered upon allocations and frees
● Memory stomping…? Could it be…?
Mysterious crash (cont.)
● Started researching memory debugging
– No tools provided by Sony
● Tried using debug allocators (dmalloc et al.)
– Most use the concept of memory fences
– Difficult to hook up to UE3
malloc
Regular allocation Fenced allocation
malloc
Mysterious crash (cont.)
● Found and integrated a community-developed tool,
Heap Inspector [VANDERBEEK14]
– Memory analyzer
– Focused on consumption and usage patterns monitoring
– Records callstacks for allocations and frees
● Several reproduction attempts revealed a correlation
– Crash adress
– Construction of a specific class
● Gotcha!
Mysterious crash (cont.)
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent'
(Crash);
Mysterious crash (cont.)
● Cause: buffer overflow vulnerability in
UnrealScript VM
– No manifestation on X360 & Windows due to larger
allocation alignment (8 vs 16 bytes)
● Fix: make copy-construction fail when template
is a subclassed object
● I wish I had Valgrind! [GODLEWSKI02]
Agenda
● How is gamedev different?
● Bug species
● Case studies
● Conclusions
Takeaway
● Time is of the essence!
● Always on a tight schedule
● Constantly in motion
– Temporal visualization is key
– Custom, domain-specific tools
● Complex and indeterministic
– Difficult to automate testing
– Wide knowledge required
● Prone to bugs outside the code
– Custom, domain-specific tools, again
Takeaway (cont.)
● Rendering is a whole separate beast
– Absolutely custom tools in isolation from the rest of the
game
– Still far from ideal usability
● Good to know your machine down to the metal
● Good memory debugging tools make a world's
difference
● You are never safe, not even in managed
languages!
Questions?
@ leszek.godlewski@theastronauts.com
t @TheIneQuation
K www.inequation.org
Thank you!
References
● SINILO12 – Sinilo, M. ”Coding in a debugger” [link]
● GODLEWSKI01 – Godlewski, L. ”OpenGL (ES)
debugging” [link]
● GLSL120 – Kessenich, J. ”The OpenGL® Shading
Language”, Language Version: 1.20, Document
Revision: 8, p. 57 [link]
● VANDERBEEK14 – van der Beek, J. ”Heap
Inspector” [link]
● GODLEWSKI02 – Godlewski, L. ”Advanced Linux
Game Programming” [link]
1 of 90

Recommended

Gamedev-grade debugging by
Gamedev-grade debuggingGamedev-grade debugging
Gamedev-grade debuggingLeszek Godlewski
1.3K views92 slides
OpenGL (ES) debugging by
OpenGL (ES) debuggingOpenGL (ES) debugging
OpenGL (ES) debuggingLeszek Godlewski
1.7K views36 slides
Linux as a gaming platform - Errata by
Linux as a gaming platform - ErrataLinux as a gaming platform - Errata
Linux as a gaming platform - ErrataLeszek Godlewski
16.5K views15 slides
Html5game presentation by
Html5game presentationHtml5game presentation
Html5game presentationrandyhoyt
1.8K views129 slides
Practical Guide for Optimizing Unity on Mobiles by
Practical Guide for Optimizing Unity on MobilesPractical Guide for Optimizing Unity on Mobiles
Practical Guide for Optimizing Unity on MobilesValentin Simonov
10.2K views39 slides
Choosing your Game Engine (2009) by
Choosing your Game Engine (2009)Choosing your Game Engine (2009)
Choosing your Game Engine (2009)Mark DeLoura
5K views76 slides

More Related Content

Viewers also liked

Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI by
 Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI  Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI
Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI PROIDEA
191 views18 slides
MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer? by
MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer?MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer?
MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer?PROIDEA
237 views13 slides
JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft... by
JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft...JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft...
JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft...PROIDEA
133 views68 slides
Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t... by
Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t...Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t...
Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t...PROIDEA
213 views8 slides
4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac... by
4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac...4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac...
4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac...PROIDEA
124 views33 slides
4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila by
4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila
4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach AgilaPROIDEA
107 views22 slides

Viewers also liked(18)

Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI by PROIDEA
 Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI  Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI
Atmosphere 2016 - Pawel Leszczynski - Hadoop without CLI
PROIDEA191 views
MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer? by PROIDEA
MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer?MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer?
MCE^3 - Karolina Cikowska, Van Anh Dam - Why Your Kid Won’t Be a Programmer?
PROIDEA237 views
JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft... by PROIDEA
JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft...JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft...
JDD2015: Jak dogadywać się z obcymi formami inteligencji - poradnik dla craft...
PROIDEA133 views
Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t... by PROIDEA
Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t...Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t...
Atmosphere 2016 - Eugenij Safanov - Web Application Security: from reactive t...
PROIDEA213 views
4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac... by PROIDEA
4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac...4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac...
4Developers2016: Michał Mycka- Racjonalnie o emocjach użytkowników w badaniac...
PROIDEA124 views
4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila by PROIDEA
4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila
4Developers: Miroslaw Dąbrowski - Poznaj skalę i rozmach Agila
PROIDEA107 views
[4developers2016] - ScalaJS – on the way to light server (Jaroslaw Ratajski) by PROIDEA
[4developers2016] - ScalaJS – on the way to light server (Jaroslaw Ratajski)[4developers2016] - ScalaJS – on the way to light server (Jaroslaw Ratajski)
[4developers2016] - ScalaJS – on the way to light server (Jaroslaw Ratajski)
PROIDEA65 views
4Developers: Dns vs webapp by PROIDEA
4Developers: Dns vs webapp4Developers: Dns vs webapp
4Developers: Dns vs webapp
PROIDEA178 views
PLNOG14: Jak budowaliśmy kolejną serwerownię - Sylwester Biernacki by PROIDEA
PLNOG14: Jak budowaliśmy kolejną serwerownię - Sylwester BiernackiPLNOG14: Jak budowaliśmy kolejną serwerownię - Sylwester Biernacki
PLNOG14: Jak budowaliśmy kolejną serwerownię - Sylwester Biernacki
PROIDEA591 views
PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study by PROIDEA
PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case StudyPLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study
PLNOG 13: Piotr Szolkowski: 100G Ethernet – Case Study
PROIDEA561 views
CONFidence 2015: Defensive Time-Out or unclear digressions about past present... by PROIDEA
CONFidence 2015: Defensive Time-Out or unclear digressions about past present...CONFidence 2015: Defensive Time-Out or unclear digressions about past present...
CONFidence 2015: Defensive Time-Out or unclear digressions about past present...
PROIDEA251 views
MCE^3 - Ricardo Brito - New Design Mindset by PROIDEA
MCE^3 - Ricardo Brito - New Design MindsetMCE^3 - Ricardo Brito - New Design Mindset
MCE^3 - Ricardo Brito - New Design Mindset
PROIDEA202 views
[CONFidence 2016] Marco Ortisi - Recover a RSA private key from a TLS session... by PROIDEA
[CONFidence 2016] Marco Ortisi - Recover a RSA private key from a TLS session...[CONFidence 2016] Marco Ortisi - Recover a RSA private key from a TLS session...
[CONFidence 2016] Marco Ortisi - Recover a RSA private key from a TLS session...
PROIDEA76 views
JDD 2016 - Michał Balinski, Oleksandr Goldobin - Practical Non Blocking Micro... by PROIDEA
JDD 2016 - Michał Balinski, Oleksandr Goldobin - Practical Non Blocking Micro...JDD 2016 - Michał Balinski, Oleksandr Goldobin - Practical Non Blocking Micro...
JDD 2016 - Michał Balinski, Oleksandr Goldobin - Practical Non Blocking Micro...
PROIDEA81 views
JDD2014: The responsible developer - Thomas Sundberg by PROIDEA
JDD2014: The responsible developer - Thomas SundbergJDD2014: The responsible developer - Thomas Sundberg
JDD2014: The responsible developer - Thomas Sundberg
PROIDEA469 views
PLNOG15: Lego Bricks - How to use Amazon Services to create a lovely product ... by PROIDEA
PLNOG15: Lego Bricks - How to use Amazon Services to create a lovely product ...PLNOG15: Lego Bricks - How to use Amazon Services to create a lovely product ...
PLNOG15: Lego Bricks - How to use Amazon Services to create a lovely product ...
PROIDEA506 views
4Developers2016: Szymon Heliosz- To content zepsuje ci dzień by PROIDEA
4Developers2016: Szymon Heliosz- To content zepsuje ci dzień4Developers2016: Szymon Heliosz- To content zepsuje ci dzień
4Developers2016: Szymon Heliosz- To content zepsuje ci dzień
PROIDEA110 views
4Developers: Time series databases by PROIDEA
4Developers: Time series databases4Developers: Time series databases
4Developers: Time series databases
PROIDEA100 views

Similar to 4Developers 2015: Gamedev-grade debugging - Leszek Godlewski

BSidesDelhi 2018: Headshot - Game Hacking on macOS by
BSidesDelhi 2018: Headshot - Game Hacking on macOSBSidesDelhi 2018: Headshot - Game Hacking on macOS
BSidesDelhi 2018: Headshot - Game Hacking on macOSBSides Delhi
442 views56 slides
Gdc gameplay replication in acu with videos by
Gdc   gameplay replication in acu with videosGdc   gameplay replication in acu with videos
Gdc gameplay replication in acu with videosCharles Lefebvre
1.2K views54 slides
Developing Next-Generation Games with Stage3D (Molehill) by
Developing Next-Generation Games with Stage3D (Molehill) Developing Next-Generation Games with Stage3D (Molehill)
Developing Next-Generation Games with Stage3D (Molehill) Jean-Philippe Doiron
1.8K views78 slides
Mastering Multiplayer Stage3d and AIR game development for mobile devices by
Mastering Multiplayer Stage3d and AIR game development for mobile devicesMastering Multiplayer Stage3d and AIR game development for mobile devices
Mastering Multiplayer Stage3d and AIR game development for mobile devicesJean-Philippe Doiron
2.2K views63 slides
Finding Monsters Adventure VR Experience by
Finding Monsters Adventure VR ExperienceFinding Monsters Adventure VR Experience
Finding Monsters Adventure VR ExperienceRafael Ferrari
611 views89 slides
Sephy engine development document by
Sephy engine development documentSephy engine development document
Sephy engine development documentJaejun Kim
813 views55 slides

Similar to 4Developers 2015: Gamedev-grade debugging - Leszek Godlewski(20)

BSidesDelhi 2018: Headshot - Game Hacking on macOS by BSides Delhi
BSidesDelhi 2018: Headshot - Game Hacking on macOSBSidesDelhi 2018: Headshot - Game Hacking on macOS
BSidesDelhi 2018: Headshot - Game Hacking on macOS
BSides Delhi442 views
Gdc gameplay replication in acu with videos by Charles Lefebvre
Gdc   gameplay replication in acu with videosGdc   gameplay replication in acu with videos
Gdc gameplay replication in acu with videos
Charles Lefebvre1.2K views
Developing Next-Generation Games with Stage3D (Molehill) by Jean-Philippe Doiron
Developing Next-Generation Games with Stage3D (Molehill) Developing Next-Generation Games with Stage3D (Molehill)
Developing Next-Generation Games with Stage3D (Molehill)
Mastering Multiplayer Stage3d and AIR game development for mobile devices by Jean-Philippe Doiron
Mastering Multiplayer Stage3d and AIR game development for mobile devicesMastering Multiplayer Stage3d and AIR game development for mobile devices
Mastering Multiplayer Stage3d and AIR game development for mobile devices
Finding Monsters Adventure VR Experience by Rafael Ferrari
Finding Monsters Adventure VR ExperienceFinding Monsters Adventure VR Experience
Finding Monsters Adventure VR Experience
Rafael Ferrari611 views
Sephy engine development document by Jaejun Kim
Sephy engine development documentSephy engine development document
Sephy engine development document
Jaejun Kim813 views
Dynamic Wounds on Animated Characters in UE4 by Michał Kłoś
Dynamic Wounds on Animated Characters in UE4Dynamic Wounds on Animated Characters in UE4
Dynamic Wounds on Animated Characters in UE4
Michał Kłoś4.3K views
Porting games from ps3 or web to shield and ouya [final] by Jean-Philippe Doiron
Porting games from ps3 or web to shield and ouya [final]Porting games from ps3 or web to shield and ouya [final]
Porting games from ps3 or web to shield and ouya [final]
Chrome game programming_with_for_play by firenze-gtug
Chrome game programming_with_for_playChrome game programming_with_for_play
Chrome game programming_with_for_play
firenze-gtug406 views
Chrome game programming_with_for_play by firenze-gtug
Chrome game programming_with_for_playChrome game programming_with_for_play
Chrome game programming_with_for_play
firenze-gtug556 views
Tools for developing Android Games by Platty Soft
 Tools for developing Android Games Tools for developing Android Games
Tools for developing Android Games
Platty Soft4.7K views
Make believe - Droidcon UK 2015 by Shanee Nishry
Make believe - Droidcon UK 2015Make believe - Droidcon UK 2015
Make believe - Droidcon UK 2015
Shanee Nishry377 views
Ghajini - The Game Development by Imran K
Ghajini - The Game DevelopmentGhajini - The Game Development
Ghajini - The Game Development
Imran K1.8K views
Going Multi-Node by SmartLogic
Going Multi-NodeGoing Multi-Node
Going Multi-Node
SmartLogic683 views
Create a Scalable and Destructible World in HITMAN 2* by Intel® Software
Create a Scalable and Destructible World in HITMAN 2*Create a Scalable and Destructible World in HITMAN 2*
Create a Scalable and Destructible World in HITMAN 2*
Intel® Software5.9K views
Computer Games Inner Workings - I. Loukeris AIT by AIT_Communications
Computer Games Inner Workings - I. Loukeris AITComputer Games Inner Workings - I. Loukeris AIT
Computer Games Inner Workings - I. Loukeris AIT
AIT_Communications471 views
Killzone Shadow Fall: Threading the Entity Update on PS4 by jrouwe
Killzone Shadow Fall: Threading the Entity Update on PS4Killzone Shadow Fall: Threading the Entity Update on PS4
Killzone Shadow Fall: Threading the Entity Update on PS4
jrouwe11.8K views

Recently uploaded

The Era of Large Language Models.pptx by
The Era of Large Language Models.pptxThe Era of Large Language Models.pptx
The Era of Large Language Models.pptxAbdulVahedShaik
7 views9 slides
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptxanimuscrm
15 views19 slides
Programming Field by
Programming FieldProgramming Field
Programming Fieldthehardtechnology
6 views9 slides
nintendo_64.pptx by
nintendo_64.pptxnintendo_64.pptx
nintendo_64.pptxpaiga02016
5 views7 slides
Keep by
KeepKeep
KeepGeniusee
78 views10 slides
Introduction to Maven by
Introduction to MavenIntroduction to Maven
Introduction to MavenJohn Valentino
6 views10 slides

Recently uploaded(20)

2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm15 views
Understanding HTML terminology by artembondar5
Understanding HTML terminologyUnderstanding HTML terminology
Understanding HTML terminology
artembondar56 views
Ports-and-Adapters Architecture for Embedded HMI by Burkhard Stubert
Ports-and-Adapters Architecture for Embedded HMIPorts-and-Adapters Architecture for Embedded HMI
Ports-and-Adapters Architecture for Embedded HMI
Burkhard Stubert26 views
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI... by Marc Müller
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Marc Müller42 views
FOSSLight Community Day 2023-11-30 by Shane Coughlan
FOSSLight Community Day 2023-11-30FOSSLight Community Day 2023-11-30
FOSSLight Community Day 2023-11-30
Shane Coughlan6 views
360 graden fabriek by info33492
360 graden fabriek360 graden fabriek
360 graden fabriek
info33492143 views
Software evolution understanding: Automatic extraction of software identifier... by Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports by Ra'Fat Al-Msie'deen
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug ReportsBushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... by Marc Müller
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
Marc Müller41 views
Sprint 226 by ManageIQ
Sprint 226Sprint 226
Sprint 226
ManageIQ10 views
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by NimaTorabi2
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
NimaTorabi215 views

4Developers 2015: Gamedev-grade debugging - Leszek Godlewski

  • 1. Gamedev-grade debugging Leszek Godlewski, The Astronauts Source: http://igetyourfail.blogspot.com/2009/01/reaching-out-tale-of-failed-skinning.html
  • 2. ● Engine Programmer, The Astronauts (Nov 2014 – present) – PS4 port of The Vanishing of Ethan Carter ● Programmer, Nordic Games (early 2014 – Nov 2014) ● Freelance Programmer (Sep 2013 – early 2014) ● Generalist Programmer, The Farm 51 (Mar 2010 – Aug 2013) Who is this guy?
  • 3. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
  • 5. 33 milliseconds ● How much time you have to get shit done™ – 30 Hz → 33⅓ ms per frame – 60 Hz → 16⅔ ms per frame EditorEditor Level toolsLevel tools Asset toolsAsset tools EngineEngine PhysicsPhysics RenderingRendering AudioAudio NetworkNetwork PlatformPlatform InputInput Network back-end Network back-end GameGame UIUI LogicLogic AIAI
  • 6. Interdisciplinary working environment ● Designers – Game, Level, Quest, Audio… ● Artists – Environment, Character, 2D, UI, Concept… ● Programmers – Gameplay, Engine, Tools, UI, Audio… ● Writers ● Composers ● Actors ● Producers ● PR & Marketing Specialists ● … } Tightly woven teams
  • 7. Severe, fixed hardware constraints ● Main reason for extensive use of native code
  • 9. Indeterminism & complexity ● Leads to poor testability – Parts make no sense in isolation – What exactly is correct? – Performance regressions? Source: https://github.com/memononen/recastnavigation
  • 10. Aversion to general software engineering ● Modelling ● Object-Oriented Programming ● Design patterns ● C++ STL ● Templates in general ● …
  • 11. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
  • 13. General programming bugs ● Memory access violations ● Memory stomping/buffer overflows ● Infinite loops ● Uninitialized variables ● Reference cycles ● Floating point precision errors ● Out-Of-Memory/memory fragmentation ● Memory leaks ● Threading errors
  • 14. Bad maths ● Incorrect transform order – Matrix multiplication not commutative – AB ≠ BA ● Incorrect transform space Source: http://leadwerks.com/wiki/index.php?title=TFormQuat
  • 15. Temporal bugs ● Incorrect update order – for (int i = 0; i < entities.size(); ++i) entities[i].update(); ● Incorrect interpolation/blending – Bad alpha term – Bad blending mode (additive/modulate) ● Deferred effects – After n frames – After n times an action happens – n may be random, indeterministic
  • 16. Graphical glitches ● Incorrect render state ● Shader code bugs ● Precision Source: http://igetyourfail.blogspot.com/2009/01/visit-lake-fail-this-weekend.html
  • 17. Content bugs ● Incorrect scripts ● Buggy assets Source: http://www.polycount.com/forum/showpost.php?p=1263124&postcount=10466
  • 18. Worst part? ● Most cases are two or more of the aforementioned, intertwined
  • 19. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
  • 20. Most material captured by Case studies
  • 21. Video settings not updating
  • 22. Incorrect weapon after demon mode foreshadowing
  • 25. Corpses teleported on death ● In normal gameplay, pawns have simplified movement – Sweep the actor's collision primitive through the world – Slide along slopes, stop against walls Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
  • 26. Corpses teleported on death ● Upon death, pawns switch to physics-based movement (ragdoll) Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
  • 27. Corpses teleported on death (cont.) ● Physics bodies have separate state from the game actor – Actor does not drive physics bodies, unless requested – If actor is driven by physics simulation, their location is synchronized to the hips bone body's Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
  • 28. Corpses teleported on death (cont.) ● Idea: breakpoint in FarMove()? – One function because world octree is updated – Function gets called a gazillion times per frame � – Terrible noise ● Breakpoint condition? – Teleport from arbitrary point A to arbitrary point B – Distance? ● Breakpoint sequence? – Break on death instead – When breakpoint hit, break in FarMove()
  • 29. Corpses teleported on death (cont.) ● Cause: physics body driving the actor with out- of-date state ● Fix: request physics body state synchronization to animation before switching to ragdoll
  • 32. Weapons floating away from the player ● Extremely rare, only encountered on consoles – Reproduction rate somewhere at 1 in 50 attempts – And never on developer machines � ● Player pawn in a special state for the rollercoaster ride – Many things could go wrong ● For the lack of repro, sprinkled the code with debug logs
  • 33. Weapons floating away from the player (cont.) ● Cause: incorrect update order – for (int i = 0; i < entities.size(); ++i) entities[i].update(); – Player pawn forced to update after rollercoaster car – Possible for weapons to be updated before player pawns ● Fix: enforce weapon update after player pawns
  • 35. Characters with “rapiers” ● UE3 has ”content cooking” as part of game build pipeline – Redistributable builds are ”cooked” builds ● Artifact appears only in cooked builds
  • 36. Characters with “rapiers” (cont.) ● Logs contained assertions for ”out-of-bounds vertices” ● Mesh vertex compression scheme – 32-bit float → 16-bit short int (~50% savings) – Find bounding sphere for all vertices – Normalize all vertices to said sphere radius – Map [-1; 1] floats to [-32768; 32767] 16-bit integers ● Assert condition – for (int i = 0; i < 3; ++i) assert(v[i] >= -1.f && v[i] <= 1.f, ”Out-of-bound vertex!”);
  • 37. Characters with “rapiers” (cont.) ● v[i] was NaN – Interesting property of NaN: all comparisons fail – Even with itself ● float f = nanf(); bool b = (f == f); // b is false ● How did it get there?! ● Tracked the NaN all the way down to the raw engine asset!
  • 38. Characters with “rapiers” (cont.) ● Cause: ??? ● Fix: re-export the mesh from 3D software – Magic!
  • 40. Undeniable assertion ● Happened while debugging ”rapiers” ● Texture compression library without sources ● Flood of non-critical assertions – For almost every texture – Could not ignore in bulk � – Terrible noise ● Solution suggestion taken from [SINILO12]
  • 42. Undeniable assertion (cont.) ● Locate assert message function call instruction
  • 43. Undeniable assertion (cont.) ● Enter memory view and look up the adress – 0xE8 is the CALL opcode – 4-byte address argument
  • 44. Undeniable assertion (cont.) ● NOP it out! – 0x90 is the NOP opcode
  • 48. Incorrect player movement ● Recreating player movement from one engine in another (Pain Engine → Unreal Engine 3) ● Different physics engines (Havok vs PhysX) ● Many nuances – Air control – Jump and fall heights – Slope & stair climbing & sliding down
  • 49. Incorrect player movement (cont.) ● Main nuance: capsule vs cylinder
  • 50. Incorrect player movement (cont.) ● Switching our pawn collision to capsule-based was not an option ● Emulate by sampling the ground under the cylinder instead ● No clever way to debug, just make it ”bug out” and break in debugger
  • 51. Incorrect player movement (cont.) ● Situation when getting stuck ● Cause: vanilla UE3 code sent a player locked between non-walkable surfaces into the ”falling” state ● Fix: keep the player in the “walking” state
  • 52. Incorrect player movement (cont.) ● Situation when moving without player intent ● Added visualization of sampling, turned on collision display ● Cause: undersampling ● Fix: increase radial sampling resolution 1) 2)
  • 54. Blinking full-screen damage effects ● Post-process effects are organized in one-way chains
  • 55. Blinking full-screen damage effects (cont.) ● No debugger available to observe the PP chain ● Rolled my own overlay that walked and dumped the chain contents MaterialEffect 'Vignette' Param 'Strength' 0.83 [IIIIIIII ] MaterialEffect 'FilmGrain' Param 'Strength' 0.00 [ ] UberPostProcessEffect 'None' SceneHighLights (X=0.80,Y=0.80,Z=0.80) SceneMidTones (X=0.80,Y=0.80,Z=0.80) … MaterialEffect 'Blood' Param 'Strength' 1.00 [IIIIIIIIII]
  • 56. Blinking full-screen damage effects (cont.) ● Cause: entire PP chain override – Breakpoint in chain setting revealed the level script as the source – Overeager level designer ticking one checkbox too many when setting up thunderstorm effects ● Fix: disable chain overriding altogether – No use case for it in our game anyway
  • 60. Incorrect animation states ● Animation in UE3 is done by evaluating a tree – Branches are weight-blended (either replacement or additive blend) – Sequences (raw animations) for whole-skeleton poses – Skeletal controls for fine-tuning of individual bones Source: http://udn.epicgames.com/Three/AnimTreeEditorUserGuide.html
  • 61. Incorrect animation states (cont.) ● Prominent case for domain-specific debuggers ● No tools for that in UE3, rolled my own visualizer – Walks the animation tree and dumps active branches – Allows inspection of states, but not transitions – Conventional debugging still required, but greatly narrowed down
  • 62. Incorrect animation states (cont.) ● Animation bug “checklist” ● Inspect the animation state in slow motion – Is the correct blending mode used? ● Inspect the AI and cutscene state – Capable of full animation overrides ● Inspect the assets (animation sequences) – Is the root bone correctly oriented? – Is the root bone motion correct? – Are inverse kinematics targets present and correctly placed? – Is the mesh skeleton complete and correct?
  • 63. Incorrect animation states (cont.) ● Incorrect blend of reload animation – Cause: bad root bone orientation in animation sequence ● Left hand off the weapon – Cause: left hand inverse kinematics was off – Fix: revise IK state control code ● Left hand incorrectly oriented – Cause: bad IK target marker orientation on weapon mesh
  • 65. Viewport stretched when portals are in view ● Graphics debugging is: – Tracing & recording graphics API calls – Replaying the trace – Reviewing the renderer state and resources ● Trace may be somewhat unreadable at first…
  • 66. Viewport stretched when portals are… (cont.) ● Traces may be annotated for clarity – Direct3D: ID3DUserDefinedAnnotation – OpenGL: GL_KHR_debug (more info: [GODLEWSKI01])
  • 67. Viewport stretched when portals are… (cont.) ● Quick renderer state inspection revealed that viewport dimensions were off – 1024x1024, 1:1 aspect ratio instead of 1280x720, 16:9 – Looks like shadow map resolution? ● Found the latest glViewport() call – Shadow map code indeed ● Why wasn't the viewport updated for main scene rendering?
  • 68. Viewport stretched when portals are… (cont.) ● Renderer state changes are expensive – New state needs to be validated – Modern graphics APIs are asynchronous – State reading may requrie synchronization → stalls ● Cache the current renderer state to avoid redundant calls – Cache ↔ state divergence → bugs!
  • 69. Viewport stretched when portals are… (cont.) ● Cause: cache ↔ state divergence – Difference between Direct3D and OpenGL: viewport dimensions as part of render target state, or global state ● Fix: tie viewport dimensions to render target in the cache
  • 75. Black artifacts ● First thing to do is to inspect the state ● Nothing suspicious found, turned to shaders ● On OpenGL 4.2+, shaders could be debugged in NSight… ● OpenGL 2.1, so had to resort to early returns from shader with debug colours – Shader equivalent of debug logs, a.k.a. ”Your Mum's Debugger” ● ”Shotgun debugging” with is*() functions – isnan(), isinf() ● isnan() returned true!
  • 76. Black artifacts (cont.) ● Cause: undefined behaviour in NVIDIA's pow() implementation – Results are undefined if x < 0. Results are undefined if x = 0 and y <= 0. [GLSL120] – Undefined means the implementation is free to do whatever ● NVIDIA returns QNaN the Barbarian (displayed as black, poisoning all involved calculations) ● Other vendors usually return 0 ● Fix: for all pow() calls, clamp either: – Arguments to their proper ranges – Output to [0; ∞)
  • 78. Mysterious crash ● Game in content lock (feature freeze) for a while ● Playstation 3 port nearly done ● Crash ~3-5 frames after entering a specific room ● First report included a perfectly normal callstack but no obvious reason ● QA reassigned to another task, could not pursue more ● Concluded it must've been an OOM crash
  • 79. Mysterious crash (cont.) ● Bug comes back, albeit with wildly different callstack ● Asked QA to reproduce mutliple times, including other platforms – No crashes on X360 & Windows! ● Totally different callstack each time ● Confusion! – OOM? Even in 512 MB developer mode (256 MB in retail units)? – Bad content? – Console OS bug? – Audio thread? – ???
  • 80. Mysterious crash (cont.) ● Reviewed a larger sample of callstacks ● Most ended in dlmalloc's integrity checks – Assertions triggered upon allocations and frees ● Memory stomping…? Could it be…?
  • 81. Mysterious crash (cont.) ● Started researching memory debugging – No tools provided by Sony ● Tried using debug allocators (dmalloc et al.) – Most use the concept of memory fences – Difficult to hook up to UE3 malloc Regular allocation Fenced allocation malloc
  • 82. Mysterious crash (cont.) ● Found and integrated a community-developed tool, Heap Inspector [VANDERBEEK14] – Memory analyzer – Focused on consumption and usage patterns monitoring – Records callstacks for allocations and frees ● Several reproduction attempts revealed a correlation – Crash adress – Construction of a specific class ● Gotcha!
  • 83. Mysterious crash (cont.) // class declaration class Crasher extends ActorComponent; var int DummyArray[1024]; // in ammo consumption code Crash = new class'Crasher'; Comp = new class'ActorComponent' (Crash);
  • 84. Mysterious crash (cont.) ● Cause: buffer overflow vulnerability in UnrealScript VM – No manifestation on X360 & Windows due to larger allocation alignment (8 vs 16 bytes) ● Fix: make copy-construction fail when template is a subclassed object ● I wish I had Valgrind! [GODLEWSKI02]
  • 85. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
  • 86. Takeaway ● Time is of the essence! ● Always on a tight schedule ● Constantly in motion – Temporal visualization is key – Custom, domain-specific tools ● Complex and indeterministic – Difficult to automate testing – Wide knowledge required ● Prone to bugs outside the code – Custom, domain-specific tools, again
  • 87. Takeaway (cont.) ● Rendering is a whole separate beast – Absolutely custom tools in isolation from the rest of the game – Still far from ideal usability ● Good to know your machine down to the metal ● Good memory debugging tools make a world's difference ● You are never safe, not even in managed languages!
  • 90. References ● SINILO12 – Sinilo, M. ”Coding in a debugger” [link] ● GODLEWSKI01 – Godlewski, L. ”OpenGL (ES) debugging” [link] ● GLSL120 – Kessenich, J. ”The OpenGL® Shading Language”, Language Version: 1.20, Document Revision: 8, p. 57 [link] ● VANDERBEEK14 – van der Beek, J. ”Heap Inspector” [link] ● GODLEWSKI02 – Godlewski, L. ”Advanced Linux Game Programming” [link]

Editor's Notes

  1. So here&amp;apos;s the agenda for our meeting today. It appears a bit lacklustre, but the devil is in the details. I&amp;apos;d like to lay down just a little bit of context before diving into the case studies.
  2. This is a schematic for a typical game game loop. The algorithm in real application may be slightly different, but that&amp;apos;s the gist of it: until the exit condition is met, there is an endless game state update and display loop. The “drawing” here is an umbrella term for generating all of the game output – video, audio, tactile/haptic feedback, everything. &amp;lt;number&amp;gt;
  3. Apologies for the S-bomb, but that&amp;apos;s just calling it out for what it is. ;) This is a simplified structural diagram of a modern game. As you can see, there are many components that need to do their own jobs in that short time.
  4. Programmers almost never work without interaction with non-technical team-mates. Usually you&amp;apos;ll be working in tightly woven teams of people of diverse skills.
  5. Especially on consoles. PCs are obviously more lax. Not unlike embedded development. Most prominent reason for extensive use of native code.
  6. Primary balance between different factors.
  7. Emergent gameplay – unscripted interaction of game systems with interesting results Crysis series Watch_Dogs Nemesis in Shadow of Mordor How do you define correctness? People have different tastes, difficult to measure fun Even in technical parts, e.g. rendering – per-pixel comparison makes little sense (hardware – rasterization/derivation patterns, shader compilers, approximations etc.) What really caused the regression? CPU/GPU/IO load Configuration New content
  8. There is a prevailing attitude that special problems need special, not generic solutions. Of course there is a lot of ego, elitist bullshit and the NIH (Not Invented Here) syndrome in this, so take it with a pinch of salt. ;)
  9. Not sure what this game is, just found this GIF on the net, gave me a chuckle. ;) The categorization here is nothing formal or canonical, it&amp;apos;s just something that helps me make sense of my work. &amp;lt;number&amp;gt;
  10. Memory access violations Dangling or uninitialized pointers Use-after-free Threading errors Race conditions Deadlocks
  11. Especially in linear algebra. Also calculus, if you&amp;apos;re doing sufficiently advanced stuff like physics or graphics programming.
  12. This is a huge thing in game development. Most things in games are in some kind of motion and so their state changes over time. Again, this entity update code is just a schematic; specific implementations may differ in details, but that&amp;apos;s the gist of it. If you access the entity #i + n (where n &amp;gt; 0) from entity #i&amp;apos;s update, that&amp;apos;s reading the old state from the previous frame. That&amp;apos;s why update order matters.
  13. Incorrect render state Resource (texture, shader, buffer) binding Blending mode Tests (depth, stencil, alpha, backface cull) Shader code bugs Regular programming Precision Low-precision texture formats Transcendental function approximations
  14. Scripts may be code or graphs, like the illustration shows With assets I mean not just in substantively incorrect things like a human body with bad proportions, a dog with 5 legs etc. They may be technically invalid, illegal: Mesh vertices Texture formats Texture colour space Texture compression formats Animations Root bone orientation Intended blend mode (replace/add/modulate) Compression Custom, domain-specific debugging comes into play here
  15. It&amp;apos;s almost never just one thing; you&amp;apos;ll get content bugs that manifest (or not) because of rendering glitches, temporal bugs coming out of bad maths, threading issues resulting from bad content updated in a specific order or all the other ways around, all the while sprinkled with generic programming errors.
  16. Most of the cases I&amp;apos;m going to discuss are illustrated by material captured by Quantic Lab, who were the QA contractors for those projects. Not all bugs have been documented so well, though, so some don&amp;apos;t have fancy screens or videos. I have vowed to better document the bugs of my next games, however, so that blooper rolls can be put together. I had a hard time trying to come up with an order to show the cases in. Areas are not a good choice because of overlap; chronology doesn&amp;apos;t make much sense either; so I just tried to guesstimate how interesting they would appear to the audience and sorted them on that key in ascending order. &amp;lt;number&amp;gt;
  17. &amp;lt;number&amp;gt;
  18. &amp;lt;number&amp;gt;
  19. &amp;lt;number&amp;gt;
  20. Just a screenshot of the game mode, not an illustration of the case. The mode was programmed mostly by an intern and had some custom death logic so that living players can help and revive those who died. When a player really died (i.e. could not be revived anymore), they would be teleported into a seemingly random point in the map. Some investigation proved it was always very close to the spawn point where their pawn was originally inserted into the game. &amp;lt;number&amp;gt;
  21. Again, just to explain the context in a whim. This will actually be a box in UE3, it&amp;apos;s a contrived illustration.
  22. This one is the real thing, though.
  23. Actor actually does drive some part of the physics state, but that&amp;apos;s outside the scope of this talk.
  24. &amp;lt;number&amp;gt;
  25. &amp;lt;number&amp;gt;
  26. Strangely enough, the debug logs worked!
  27. I noticed that the printouts in bugged runs of the level had an inverted update order. Mechanisms to enforce update order are beyond the scope of this talk.
  28. &amp;lt;number&amp;gt;
  29. Upon inspection, cooking logs included warnings about “out-of-bounds vertices”. This led me to the mesh vertex compression code. The savings are around 50% because while most vertex attributes get cut by half, some may not be floats, and also you need to store another single float for the sphere radius. This isn&amp;apos;t literally the code I worked on, but a close paraphrase.
  30. What was the cause? Hell if I know. Re-exporting fixed it. Magic!
  31. Now this one isn&amp;apos;t something that we get all the time, just once in a while, but it proves it&amp;apos;s good to know the low-level stuff. &amp;lt;number&amp;gt;
  32. Maciek has some more examples of that technique on his blog, so if this is interesting, go check it out.
  33. Of course, this only lasts until the end of the debugging session. See [SINILO12] for more cool examples.
  34. Painkiller HD was a remake. I was tasked with closely replicating the player movement physics in the new engine. The first versions of the code didn&amp;apos;t perform well in tight spaces, getting the player stuck… &amp;lt;number&amp;gt;
  35. …or moving without player&amp;apos;s intent. &amp;lt;number&amp;gt;
  36. Before I discuss the debugging of this, let me lay out some context. Different game engines meant, for instance, different measurements of distance and velocity. It may sound absurd if you think of physics as a strict science, but different physics simulation engines will behave absolutely differently with the same data. That is beyond the scope of this talk, though.
  37. Since a capsule has half-spheres on both of its ends, you can probably imagine how it performs more smoothly than a cylinder on uneven terrain.
  38. Every sample that would not hit the ground would contribute to a force vector. This vector would then be applied when the player does not express intent to move. This would emulate the effects of the capsule closely enough to achieve a similar player movement experience to the original. I&amp;apos;ve found no clever way to debug movement code, just reproduce the bug, pause the game and break in debugger. It&amp;apos;s easier to do when time is slowed down.
  39. Going back to the first of the observed bugs: when debugged – got stuck, then just put a breakpoint in the movement function – the situation turned out to be as in the illustration. Blue lines are surface normals; green is the player&amp;apos;s collision primitive. Both surface normals are “unwalkable” (i.e. too steep) by the configurable minimum normal Z (45 degrees by default). Vanilla UE3 would switch to “falling” state when no walkable floor was found, and since our “falling” logic was already modified to disallow air manouvering and the surfaces were otherwise level, the player would be stuck.
  40. Simple visualization – coloured line segments reflecting the traces made when sampling – aptly revealed that undersampling was to blame. Red arrows indicate “ground found”, green ones – “no ground”. Black arrow is “slip force”. You can probably see how this situation ends in oscillation. Simply adding more sample points (rings) fixed it.
  41. There was one level in the entire game where the blood effects would periodically disappear for a second or two. &amp;lt;number&amp;gt;
  42. They are simply applied from left to right. Left-most node is the raw scene texture, right-most node&amp;apos;s output gets drawn onto the screen.
  43. This is not the real deal but a contrived illustration, as I can&amp;apos;t really get ahold of the sources right now, but it&amp;apos;s very close to what I got. It would just sit there in the corner, using the reflection mechanism to inspect the effects&amp;apos; parameters and drawing progress bars where it makes sense. The text is in red because my testing showed that surprisingly, the colour red was used very sparingly in the game&amp;apos;s pallette.
  44. The tool made it evident that when blood disappeared, it was something overriding the entire PP chain. The way we&amp;apos;ve been managing PP FX, we had no use case for the option anyway.
  45. Pay attention to that character from the waist up. &amp;lt;number&amp;gt;
  46. The screenshot has been brightened to make sure it&amp;apos;s legible on a projector, the colours are not as blown out in the actual game. &amp;lt;number&amp;gt;
  47. Ditto. &amp;lt;number&amp;gt;
  48. Blend weights and other parameters that made sense that way were drawn with progress bars. The most useful part of this is that you could observe how the state changes over time. Running the game in slow motion helped a lot, too.
  49. Animation state usually gave a good hint on where to start debugging, like locomotion state. Also, NPCs in cover were another programmer&amp;apos;s responsibility as a whole, so this step served as a global “if” when assigning bug tickets. Additive blending of regular sequences and vice versa look horribly wrong (long neck syndrome anyone?). Root bone is a virtual bone that is the root to the entire skeleton hierarchy. Bad root bone transforms can miserably break otherwise perfectly correct animations.
  50. This one is slightly newer, it only happened this year. :) There&amp;apos;s a section in Darksiders where portals can be used for fast travelling and puzzle solving. &amp;lt;number&amp;gt;
  51. The exact debugging workflow varies depending on the tool. Especially the granularity: some capture entire API call streams, some capture single frames. The principle remains the same, though.
  52. And this is basically what you do. Shameless plug about OpenGL debugging!
  53. Some context again.
  54. In OpenGL, viewport dimensions are a parameter of the global state. In Direct3D, it&amp;apos;s part of render target state. So if you switch render targets in OpenGL, viewport dimensions don&amp;apos;t change, but in Direct3D they do. This port used to assume the Direct3D way, which was not correct.
  55. But that wasn&amp;apos;t so bad, graphics bugs can get much worse. This was on the PS3. &amp;lt;number&amp;gt;
  56. This as well. &amp;lt;number&amp;gt;
  57. This was NVIDIA on Linux. &amp;lt;number&amp;gt;
  58. NVIDIA on Linux. &amp;lt;number&amp;gt;
  59. So that you don&amp;apos;t think only PKHD was this broken – here&amp;apos;s Darksiders on NVIDIA on Linux. &amp;lt;number&amp;gt;
  60. NVIDIA also provided the PS3 GPU. Poisoning in this context means that all calculations in which the NaN is an operand result in NaN. Undefined is “do whatever”, including formatting setting your machine on fire. Other vendors – AFAIK. Former is mathematically correct, latter is simpler and cheaper on most GPUs. Thankfully, max(0.0, NaN) returns 0.0. It&amp;apos;s defined as if (x &amp;lt; y) {return y;} else {return x;} and any comparisons involving NaN always fail.
  61. This place in PKHD gave me the creeps for weeks. We had a very mysterious crash upon entering that cellar door. &amp;lt;number&amp;gt;
  62. And I thought no more of it.
  63. I asked colleagues for opinion, but we could only take shots in the dark.
  64. After the initial confusion blew over, I figured it would be best to review the hard data we had, so I took to a larger sample of callstacks. Could it be memory stomping? A real bug for real men? ;)
  65. I&amp;apos;ve never done memory debugging before, so I needed research. Memory fences work by reserving additional area between allocation that should somehow raise errors when that area is overwritten. I tried hooking several debug allocators up (dmalloc, Electric Fence etc.) but they either couldn&amp;apos;t be easily integrated because of the PS3 OS API, were too memory-hungry (e.g. reserved an entire page for each and every allocation) and so on.
  66. Heap Inspector isn&amp;apos;t really the best tool for the job, but it was easy to integrate and captured callstacks reasonably fast.
  67. This is not the literal code, but a paraphrase. It&amp;apos;s UnrealScript, so a managed language in a VM. Can you spot the culprit? There&amp;apos;s a complete class declaration at the top. At the bottom, we first instantiate the Crasher class, and then copy-construct an object of its base class with the new Crasher object as template.
  68. Our culprit class only had 2 or 3 new fields; the Windows/X360 allocation alignment was large enough to hide the problem. Shameless plug again – GDCE slides discuss Valgrind.
  69. Personally, I miss three things in graphics debugging: Speed – waiting for state resolution drives me nuts “Touched pixels/vertices” counter – to warn about the dreaded “my draw call has no effect” bug Full source-level shader code debugging Make a game large enough and you will eventually get down to disassembly.
  70. &amp;lt;number&amp;gt;