Part of the WGK 2014 conference proceedings: http://wgk.gd/eng/ The lecture contained some live demos which are missing from this slide deck for obvious reasons. An overview of practical techniques for debugging OpenGL rendering applications, focusing on the vendor- and version-independent debug extensions rather than IHV-supplied tools.

1. Leszek Godlewski
Programmer, Nordic Games
OpenGL (ES) debugging

2. Nordic Games GmbH
● Started in 2011 as a sister company to Nordic Games
Publishing (We Sing)
● Base IP acquired from JoWooD and DreamCatcher (SpellForce,
The Guild, Aquanox, Painkiller)
● Initially focusing on smaller, niche games
● Acquired THQ IPs in 2013 (Darksiders, Titan Quest, Red
Faction, MX vs. ATV)
● Now shifting towards being a production company with
internal devs
● Since fall 2013: internal studio in Munich, Germany (Grimlore
Games)

3. Who is this guy?
Leszek Godlewski
Programmer, Nordic Games (early 2014 – now)
– Linux port of Darksiders
Freelance Programmer (Sep 2013 – early 2014)
– Linux port of Painkiller Hell & Damnation
– Linux port of Deadfall Adventures
Generalist Programmer, The Farm 51 (Mar 2010 – Aug
2013)
– Painkiller Hell & Damnation, Deadfall Adventures

4. Demo code available
is.gd/GDCE14Linux

5. Agenda
Overview of available IHV tools
Debug callback
– Setup and implementation
– Verbosity control
– Noise filtering
API call tracing and replaying
– Using apitrace
– Annotating the call trace
Resource leak checking

6. NVIDIA Nsight

7. NVIDIA Nsight (cont.)
Supports CUDA, OpenGL and Direct3D
Plugin for Visual Studio and Eclipse
Very good feature set
– Including debugging shaders just like CPU code! See [PLASTIC13]
– Operation on one machine is slow, it's suggested to have two (sic!)
Hardware limitations
– Recent NVIDIA GPUs only
Software limitations
– Windows only (Visual Studio edition)
– Windows or Linux (Eclipse edition)
– OpenGL 4.2 or newer

8. AMD CodeXL

9. AMD CodeXL (cont.)
Supports OpenCL, OpenGL and advanced AMD CPU
features
Stand-alone applicaiton + plugin for Visual Studio
Reasonable feature set
– Includes functionality of gDEBugger
– No shader debugging (although OpenCL kernels can be debugged)
Hardware limitations
– Some functionality limited to AMD GPUs

10. AMD GPU PerfStudio

11. AMD GPU PerfStudio (cont.)
Supports OpenGL and Direct3D
Cross-platform client, Windows stand-alone server/GUI
Reasonable feature set
– Shader debugging only for Direct3D
Hardware limitations
– Some functionality limited to AMD GPUs
Software limitations
– OpenGL 4.2 or newer

12. Intel Graphics Performance Analyzer

13. Intel Graphics Performance Analyzer (cont.)
Supports OpenGL ES and Direct3D
Windows/Android client, Windows/Linux stand-alone
server/GUI
Reasonable feature set
– No shader debugging
Hardware limitations
– Only Intel GPUs
Software limitations
– Windows or Android only
– OpenGL ES only (Android)

14. Agenda
Overview of available IHV tools
Debug callback
– Setup and implementation
– Verbosity control
– Noise filtering
API call tracing and replaying
– Using apitrace
– Annotating the call trace
Resource leak checking

15. Ye Olde Way
Call glGetError() after each OpenGL call
Get 1 of 8 (sic!) error codes
Look the call up in the manual
See what this particular error means in this particular
context
Check which of the parameters was wrong
– Usually by attaching a regular debugger and replaying the scenario
…
This sucks!

16. Ye Olde Way
Call glGetError() after each OpenGL call
Get 1 of 8 (sic!) error codes
Look the call up in the manual
See what this particular error means in this particular
context
Check which of the parameters was wrong
– Usually by attaching a regular debugger and replaying the scenario
…
This sucks! used to suck ☺

17. Debug callback
Never call glGetError() again!
Much more detailed information
– Including Performance tips from the driver
– Good to check what different drivers say
May not work without a debug OpenGL context
– GLX_CONTEXT_DEBUG_BIT_ARB
– WGL_CONTEXT_DEBUG_BIT_ARB

18. Debug callback (cont.)
Provided by either of (ABI-compatible):
GL_KHR_debug or
GL_ARB_debug_output

19. Debug callback (cont.)
void callback(GLenum source,
GLenum type,
GLuint id,
GLenum severity,
GLsizei length,
const GLchar* message,
const void* userParam);
Filter by
source, type,
severity or
individual
messages

20. Debug callback (cont.)
Verbosity can be controlled
– glDebugMessageControl[ARB]()
– [OPENGL01][OPENGL02]
Turn to 11 for valuable perf information:
– Which memory type a buffer is backed by
– Memory wasted by unused mip levels
– More!
– glDebugMessageControl(GL_DONT_CARE,
GL_DONT_CARE, GL_DONT_CARE, 0, 0, GL_TRUE);

21. Agenda
Overview of available IHV tools
Debug callback
– Setup and implementation
– Verbosity control
– Noise filtering
API call tracing and replaying
– Using apitrace
– Annotating the call trace
Resource leak checking

22. API call tracing
Record a trace of the run of the application
Replay and review the trace
– Look up OpenGL state at a particular call
– Inspect state variables, resources and objects:
●
Textures
●
Shaders
●
Buffers
●
…
apitrace or VOGL

23. Well, this is not helpful...

24. Much better!

25. Annotating the call stream

26. Annotating the call stream (cont.)
All aforementioned extensions supported by apitrace
even if not by driver
Recommended: GL_KHR_debug
– Best vendor coverage
●
GL_KHR_debug is slightly less common
●
GL_ARB_debug_output has no debug groups or object labels
– Emulation wrapper for Mac OS X [PIPELINE13]

27. Annotating the call stream (cont.)
Call grouping
– glPushDebugGroup()/glPopDebugGroup() (KHR_debug)
One-off messages
– glDebugMessageInsert()
(KHR_debug/ARB_debug_output)
– glStringMarkerGREMEDY() (GREMEDY_string_marker)

28. Object labelling
Buffer, shader, program, vertex array, query, program
pipeline, transform feedback, sampler, texture, render
buffer, frame buffer, display list
– glObjectLabel(), glGetObjectLabel()
Sync objects
– glObjectPtrLabel(), glGetObjectPtrLabel()

29. Annotation caveats
Multi-threaded/multi-context OpenGL application may
break debug group hierarchy
glDebugMessageInsert() calls the debug callback,
polluting error streams
– Workaround: drop if source ==
GL_DEBUG_SOURCE_APPLICATION

30. Agenda
Overview of available IHV tools
Debug callback
– Setup and implementation
– Verbosity control
– Noise filtering
API call tracing and replaying
– Using apitrace
– Annotating the call trace
Resource leak checking

31. Resource leak checking
When created correctly (glGen*()), object names are
integers, consecutive & recycled
– Not necessarily!
– Desktop GL names may be user-supplied
– GLES may be not recycled
Stupid idea: iterate over names [1; ∞)?

32. Resource leak checking (cont.)
Courtesy of Eric Lengyel & Fabian Giesen
static void check_for_leaks()
{
GLuint max_id = 10000; // better idea would be to keep track of assigned names.
GLuint id;
// if brute force doesn't work, you're not applying it hard enough
for ( id = 1 ; id <= max_id ; id++ )
{
#define CHECK( type ) if ( glIs##type( id ) )
fprintf( stderr, "GLX: leaked " #type " handle 0x%xn", (unsigned int) id )
CHECK( Texture );
CHECK( Buffer );
CHECK( Framebuffer );
CHECK( Renderbuffer );
CHECK( VertexArray );
CHECK( Shader );
CHECK( Program );
CHECK( ProgramPipeline );
#undef CHECK
}
}

33. Takeaway
IHV tools are cool, but complex & have their limits
– Valuable, so pick what works best for your HW+SW combo
Debug callbacks work everywhere
Debug callbacks will show you exactly what the problem
is (most of the time)
API call tracing works everywhere across-the-board
Annotating the trace helps you find your way
Resource leak checks? glIs*()!

34. @ l g o d l e w s k i @ n o r d i c ga m e s . a t
t @ T h e I n e Q u a t i o n
K w ww . i n e qu a t i o n . o r g
Questions?

35. F u rt h e r N o rd i c G a m e s i n f o rm a t i o n :
K w w w . n o r d i c g a m e s . a t
D e v e l o p me n t i n f o r ma t i o n :
K ww w . g ri ml o re ga m e s . c o m
Thank you!

36. References
 PLASTIC13 – Staniszewski, M., Szymczyk, M. ”Nsight” [link]
 OPENGL01 – “ARB_debug_output” [link]
 OPENGL02 – “KHR_debug” [link]
 PIPELINE13 – Menzer, R. ”(Simulating) KHR_debug on MacOS X” [link]