Your SlideShare is downloading. ×
0
1
2
Mark J. Kilgard, NVIDIA
Kurt Akeley, Microsoft Research
13 December 2008
Singapore
Modern OpenGL:
Its Design and Evoluti...
3
Introductions
4
Kurt Akeley
• Led development of OpenGL at Silicon Graphics (SGI)
• Co-founded SGI
• Lead development of SGI’s high-end ...
5
Mark Kilgard
• Principal System Software Engineer, NVIDIA, Austin, Texas
• Developed original OpenGL driver for 1st
GeFo...
6
Marc Levoy
• Moderator for our facilitated discussion
• Professor of Computer Science and Electrical
Engineering
• Stanf...
7
Course Schedule
• Modern OpenGL (Kilgard)
• OpenGL’s evolution: a personal retrospective (Akeley)
• Writing better OpenG...
8
Check Out the Course Notes (1)
• Look to www.opengl.org web site for our final slides
• New Material
• “An Incomplete Hi...
9
Check Out the Course Notes (2)
• Paper Reprints
• OpenGL design rationale from its specification co-
authors (Segal, Ake...
10
Modern OpenGL
Mark Kilgard
Principal System Software Engineer
NVIDIA
11
Modern OpenGL
• History
• How did OpenGL get where it is now?
• Present
• Version 3.0
• Functionality beyond 3.0
12
An Overview History of OpenGL
• Pre-history 1991
• IRIS GL, a proprietary Graphics Library by SGI
• OpenGL, an open sta...
13
OpenGL’s Pre-history
IRIS GL 1
Window system: MEX
IRIS GL 2
Window system: MEX
Operating system: UNIX
IRIS GL 3
Window ...
14
OpenGL’s Design Philosophy
• High-performance
• Assumes hardware
acceleration
• Defined by a specification
• Rather tha...
15
Timeline of OpenGL’s Development
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2...
16
Competitive 3D APIs
• OpenGL has always existed in
competition with other APIs
• Strengthened OpenGL by driving
feature...
17
OpenGL 1.0
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
18
OpenGL State Machine
• From OpenGL 3.0 specification, unchanged since 1.0
19
SGI “Classic” Hardware View of OpenGL
3D Application
or Game
• Entirely fixed-function, no programmability
• High-end S...
20
OpenGL 1.1
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
21
The Look of OpenGL 1.1
SGI skyfly demoSGI skyfly demo
StenciledStenciled
shadow volumesshadow volumes
Ideas in MotionId...
22
OpenGL 1.2
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
23
Akeley’s (Modernized) OpenGL Data Flow
vertex
shading
rasterization
& fragment
shading
texture
raster
operations
frameb...
24
OpenGL 1.2 Imaging Subset
Color Table
Convolution
(separable or general)
Post-convolve
Scale & Bias
Post-convolve
Color...
25
OpenGL 1.2.1
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added ...
26
Multi-texture Poster Child:
Quake 2 Light Maps
×
(modulate)
=
lightmaps onlylightmaps only
decal onlydecal only
combine...
27
GeForce 256 (NV10) View of OpenGL
3D Application
or Game
• Vertex pulling (vertex buffer objects) via DMA
• Dual-textur...
28
Hardware Cube Maps
Rendered sceneRendered scene
DynamicallyDynamically
createdcreated
cube map imagecube map image
Imag...
29
OpenGL 1.3
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
30
GeForce 3 & 4 Ti (NV2x) View of OpenGL
3D Application
or Game
• Programmable vertex processing
• Highly configurable fr...
31
Vertex Programmability
Paletted matrixPaletted matrix
skinningskinning
Twister vertex programTwister vertex program
Per...
32
Configurable Fragment Processing
Bumpy shiny environment mappingBumpy shiny environment mappingChromaticChromatic
aberr...
33
OpenGL 1.4
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
34
Hardware Shadow Mapping
Without shadow mappingWithout shadow mapping WithWith shadow mappingshadow mapping
Depth map fr...
35
Shadow Mapping Explained
Planar distance from lightPlanar distance from light Depth map projected onto sceneDepth map p...
36
OpenGL 1.5
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
37
GeForce FX (NV3x) View of OpenGL
3D Application
or Game
• Programmable fragment processing
• 16 texture units, IEEE 754...
38
Floating-point Fragment Programmability
39
OpenGL Fragment
Program Flowchart
More
Instructions?
Read Interpolants
and/or Registers
Map Input values:
Swizzle, Nega...
40
Key Trend:
Configurability becomes Programmability
Fixed-function Programmable
Simple
Configurability
Complex
Configura...
41
Core OpenGL fragment texturing & coloring
Point
Rasterization
Line
Rasterization
Polygon
Rasterization
Pixel Rectangle
...
42 NV1x OpenGL fragment texturing & coloring
Point
Rasterization
Line
Rasterization
Polygon
Rasterization
Pixel Rectangle
...
43
Texture Shader 3
…
Texture Shader 1
Texture Shader 0
Register
Combiners
NV2x OpenGL fragment texturing & colorin
Point
...
44
Fragment Program
Instruction 0
Texture Shader 3
…
Texture Shader 1
Texture Shader 0
NV3x OpenGL fragment texturing & co...
45
OpenGL 2.0
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
46
GeForce 6 & 7 (NV4x/G7x) View of OpenGL
3D Application
or Game
• Limited vertex texturing
• Fragment branching
• Multip...
47
Primitive
Program
GeForce 8 & 9 (G8x/G9x) View of OpenGL
3D Application
or Game
• Primitive (geometry) programs
• Param...
48
Primitive
Program
OpenGL Pipeline Fixed-function Steps
• Much of functional pipeline remains fixed-function
• Vital to ...
49
Primitive
Program
OpenGL Pipeline Programmable Domains
• New geometry shader domain for per-primitive programmable proc...
50
OpenGL 2.1
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
51
OpenGL 3.0
1992 1994 1996 1998 2000 2002 2004 2006 2008
OpenGL 1.0 approved
OpenGL 1.1
OpenGL 1.2
Multitexture added (1...
52
Areas of 3.0 Functionality Improvement
• Programmability
• Shader Model 4.0 features
• OpenGL Shading Language (GLSL) 1...
53
OpenGL 3.0 Programmability
• Shader Model 4.0 additions
• True signed & unsigned integer values
• True integer operator...
54
OpenGL 3.0 Texturing Functionality
• Texture representation
• Texture arrays: indexed access to a set of 1D or 2D
textu...
55
Texture Arrays
• Conventional texture = One logical pre-filtered image
• Texture array = index-able plurality of pre-fi...
56
Texture Arrays Example
• Multiple skins packed in texture array
• Motivation: binding to one multi-skin texture array a...
57
Compact Floating-point Textures
• Shared exponent & packed float representations are ideal
of High Dynamic Range (HDR) ...
58
Compact Floating-point Texture Formats
• Packed float format
• No sign bit, independent exponents
• Shared exponent for...
59
1- and 2-component
Block Compression Scheme
• Basic 1-component block compression format
• Borrowed from alpha compress...
60
Framebuffer Operations
• Framebuffer objects
• Standardized framebuffer objects (FBOs) for rendering to textures
and re...
61
Framebuffer Object Example
• Depth peeling for correctly ordered transparency
• Great render-to-texture application for...
62
Depth Peeling Behind the Scenes
• Depth buffer has closest fragment at all pixels
• Save depth buffer
• Render again, b...
63
Delicate Color Fidelity with sRGB
• Problem: PC display devices have non-linear (sRGB) display gamut
—delicate color sh...
64
What is sRGB?
• A standard color space
• Intended for monitors, printers, and the Internet
• Created cooperatively by H...
65
So why sRGB? Standard Windows Display
is Not Gamma Corrected
• 25+ years of PC graphics, icons, and images depend on no...
66
Vertex Processing
• Vertex array configuration
• Objects to manage vertex array configuration client
state
• Half-preci...
67
Miscellaneous
• Pixel Processing
• Half-precision floating-point pixel external formats
• Buffer Management
• Non-block...
68
ARB Extensions to OpenGL 3.0
• OpenGL 3.0 standard provides new ARB extensions
• Extensions go beyond OpenGL 3.0
• Stan...
69
Transform Feedback for Terrain Generation
by Recursive Subdivision
• Geometry shaders + transform feedback
1. Render qu...
70
Skin Deformation
• Capture & re-use geometric deformations
Transform
feedback allows
the GPU to
calculate the
interacti...
71
Silhouette Edge Rendering
• Uses geometry shader
silhouette
edge
detection
geometry
shader
Complete mesh
Silhouette edg...
72
More Geometry Shader Examples
Shimmering
point sprites
Generate
fins for
lines
Generate
shells for
fur
rendering
73
Improved Interpolation Techniques
•Using geometry shader functionality
Quadratic normal
interpolation
True quadrilatera...
74
“Fair” Quadrilateral Interpolation
• glBegin(GL_QUADS);
• glColor3fv(red);
glVertex3fv(lowerLeft);
• glColor3fv(green);...
75
OpenGL 2.x ARB Extensions
• Many OpenGL 3.0 extensions have corresponding ARB extensions for
OpenGL 2.1 implementations...
76
Beyond OpenGL 3.0
OpenGL 3.0
• EXT_gpu_shader4
• NV_conditional_render
• ARB_color_buffer_float
• NV_depth_buffer_float...
77
OpenGL Version Evolution
• Now OpenGL is part of Khronos Group
• Previously OpenGL’s evolution was governed by the Open...
78
29%
17%
15%
15%
4%
2%
2%
2%
2%
2%
2%
2%
1% 1%
4%
15%
Multi-vendor
Silicon Graphics
Architectural Review Board
NVIDIA
AT...
79
What’s Driving OpenGL Modernization?
Human desire for Visual
Intuition and Entertainment
Embarrassing
Parallelism of
Gr...
80
Kurt Akeley
Principal Researcher
Microsoft Research Silicon Valley
OpenGL’s Evolution:
A Personal Retrospective
81
AA personalpersonal retrospectiveretrospective
• My background:
• Silicon Graphics, 1982-2001
• OpenGL, 1990-2004
• Tod...
82
Jim Clark and the Geometry EngineJim Clark and the Geometry Engine
• This text is 24 points
– Sub bullets look like thi...
83
Jim’s helpers: the Stanford gangJim’s helpers: the Stanford gang
IRIS GL
Geometry Engine
IRIS GL
Hardware back-end
Hard...
84
Success!Success! (in 1995)(in 1995)
85
Computer Architecture
86
What is computer architecture?What is computer architecture?
• Architecture: “the minimal set of
properties that determ...
87
Example: the analog clockExample: the analog clock
• Architecture
• Circular dial divided into twelfths
• Hour hand (sh...
88
A useful distinctionA useful distinction
• NVIDIA 8800
• SIMD, or
• SPMD ?
L2
FB
SP SP
L1
TF
ThreadProcessor
Vertex Thr...
89
The mainstream viewThe mainstream view
• Table of Contents:
• Fundamentals
• Instruction Sets
• Pipelining
• Advanced P...
90
OpenGL is an architecture
Blaauw/Brooks OpenGL
Different
implementations
IBM 360 30/40/50/65/75
Amdahl
SGI Indy/Indigo/...
91
But OpenGL is an APIBut OpenGL is an API
(Application Programming Interface)(Application Programming Interface)
• Yes, ...
92
We didn’t know …We didn’t know …
• No mention in spec (even 3.0)
• “We view OpenGL as a state …”
• First use in “ARB”
•...
93
Fred is magnanimousFred is magnanimous
94
What is implied by “programmable”?What is implied by “programmable”?
• What does it mean to teach programming?
• Does r...
95
My try at some formalizationMy try at some formalization
• Key ideas:
• Composition  choice of placement, sequence
• N...
96
OpenGL has always been programmableOpenGL has always been programmable
• Follows directly from being an “architecture”
...
97
Example multi-pass OpenGL “program”Example multi-pass OpenGL “program”
glEnable(GL_DEPTH_TEST);
glDisable(GL_LIGHTING);...
98
Example multi-pass OpenGL “program”Example multi-pass OpenGL “program”
glEnable(GL_DEPTH_TEST);
glDisable(GL_LIGHTING);...
99
InvarianceInvariance
Corollary 1 Fragment generation is invariant with respect to
the state values marked with in Rule ...
100
• Intended to capture complete
sequence of operations
• Also inspired design changes
101
Vertex assembly
Primitive assembly
Rasterization
Fragment operations
Display
Vertex operations
Application
Primitive o...
102
Culture and Process
103
Suppose …Suppose …
http://www.opengl.org/registry/
Name
ARB_texture_cube_map
Name Strings
GL_ARB_texture_cube_map
Noti...
104
Complete specificationComplete specification
Name
Name Strings
Notice
Contact
Status
Version
Number
Dependencies
Overv...
105
19 issues19 issues
The spec just linearly interpolates the reflection vectors computed
per-vertex across polygons. Is ...
106
19 issues …19 issues …
What happens if an (s,t,q) is passed to cube map generation that
is close to (0,0,0), ie. a deg...
107
Trust and integrityTrust and integrity
• Lots of collaboration during the initial design
• But final decisions made by...
108
Five Kinkos in Austin TexasFive Kinkos in Austin Texas
The OpenGL Graphics System: A Specification (Version 1.1)
Mark ...
109
Extension factsExtension facts
• 442 Vendor and “EXT” extension specifications
• Vendor: specific to a single vendor
•...
110
““Specification” sizesSpecification” sizes
Lines Words Chars
56 ARB Extensions 48,674 263,908 2,221,347
All 442 Extens...
111
Beyond the specificationBeyond the specification
• The ARB (now replaced with Khronos)
• Rules of order, secretary, IP...
112
SummarySummary
• Many mistakes made (see other presentations for lists)
• Created a sustainable culture that values qu...
113
Writing better OpenGL
Mark Kilgard
Principal System Software Engineer
NVIDIA
114
Motivation
• Complex APIs and systems have pitfalls
• After 17 years of designed evolution, OpenGL
certainly has its s...
115
Communicating Vertex Data
• The way you learn OpenGL:
• Immediate mode
• glBegin, glColor3f, glVertex3f, glEnd
• Strai...
116
Example Scenario
• An OpenGL application has to render a set of rectangles
• Rectangle with its parameters
• x, y, hei...
117
Scene Representation
• Each rectangle specified by following RectInfo structure:
• Array of RectInfo structures descri...
118
Example Scene and Rendering Result
• Scene of 4 rectangles:
RectInfo rect_list[4] = {
{ 10, 20, 180, 140, 0.5,
{ 1, 1,...
119
Immediate Mode Rectangle Rendering
• Given sized RectInfo array, render vertices of quads
1st
vertex
2nd
vertex
3rd
ve...
120
Critique of Immediate Mode
• Advantages
• Straightforward to code and debug
• Easy-to-understand conceptual model
• Bu...
121
Vertex Array Approach
• Step 1: Copy vertex attributes into vertex arrays
• From: RectInfo array (CPU memory)
• To: in...
122
Vertex Array Format
• Interleave vertex attributes in color & position arrays
color
position
float = 4 bytes
vertex 0
...
123
Step 1:
Copy Rectangle Attributes to Vertex Arrays
void *initVarrayRectangles(int count, const RectInfo *list)
{
void ...
124
Step 2:
Configure & Render from Vertex Arrays
void drawVarrayRectangles(int count, const RectInfo *list)
{
char *varra...
125
Critique of
Simplistic Vertex Array Rendering
• Advantages
• Far fewer OpenGL commands issued
• Disadvantages
• Every ...
126
Initialize Once, Render Many Approach
• This routine expects base pointer returned by
initVarrayRectangles
void drawIn...
127
Client Memory Vertex Attribute Transfer
GPU
Processor
command
processor
vertex
puller
hardware
rendering
pipeline
CPU
...
128
Vertex Buffer Object Vertex Attribute Pulling
OpenGL
(vertex)
buffer
object
GPU
command
processor
vertex
puller
hardwa...
129
Initializing Vertex Buffer Objects (VBOs)
• Once using vertex arrays, easy to switch to VBOs
• Make the vertex array a...
130
Rendering from Vertex Buffer Objects
• Once initialized, glBindBuffer to bind to buffer ahead of
vertex array configur...
131
Understanding glBindBuffer
• Buffer object bindings are frequent point of confusion for
programmers
• What does glBind...
132
Bind and Query Buffer Targets
Buffer Bind Tokens
• GL_ARRAY_BUFFER
• GL_ELEMENT_ARRAY_BUFFER
Buffer Query Tokens
• GL_...
133
Latched Vertex Array Buffer Bindings
• Here’s the confusing part:
glBindBuffer(GL_ARRAY_BUFFER, 34);
glColorPointer(3,...
134
Binding Buffer Zero is Special
• By default, vertex arrays don’t access buffer objects
• Instead client memory is acce...
135
Texture Coordinate Set Selector
• A selector in OpenGL is
• A state variable that controls what state a subsequent com...
136
OpenGL’s Modern Buffer-centric
Processing Model
Vertex Array Buffer
Object (VaBO)
Transform Feedback
Buffer (XBO)
Para...
137
Usages of OpenGL Buffers Objects
• Vertex uses (VBOs)
• Input to GL: Vertex attribute buffer objects
• Color, position...
138
Continuum of OpenGL Usage
Tweak-able Performance
Immediate
mode
Client vertex
arrays
Vertex buffer
objects (VBOs)
Disp...
139
Mid-session break
15 minutes
140
Implementing OpenGL
Mark Kilgard
Principal System Software Engineer
NVIDIA
141
Topics in OpenGL Implementation
• Dual-core OpenGL driver operation
• What goes into a texture fetch?
• You give me so...
142
OpenGL Drivers for Multi-core CPUs
• Today dual-core processors in PCs is nearly ubiquitous
• 4, 6, 8, and more cores ...
143
Dual-core OpenGL Driver Architecture
Application thread …
Application thread D
Context 1
Application thread A
Applicat...
144
Dual-core Performance Results
• A well-behaved OpenGL application benefiting from a
dual-core mode of OpenGL driver op...
145
Good Dual-core Driver Practices
• General advice
• Display lists execute on the driver’s worker thread!
• You want to ...
146
Consider an OpenGL texture fetch
• Seems very simple
• Input: texture coordinates (s,t,r,q)
• Output: some color (r,g,...
147
Anatomy of a Texture Fetch
Filtered
texel
vector
Texel
Selection
Texel
Combination
Texel
offsets
Texel
data
Texture im...
148
Texture Fetch Functionality (1)
• Texture coordinate processing
• Projective texturing (OpenGL 1.0)
• Cube map face se...
149
Texture Fetch Functionality (2)
• Filter modes
• Minification / magnification transition (OpenGL 1.0)
• Nearest, linea...
150
Texture Fetch Functionality (3)
• Texture formats
• Uncompressed
• Packing: RGBA8, RGB5A1, etc. (OpenGL 1.1)
• Type: u...
151
Texture Fetch Functionality (4)
• Pre-filtering operations
• Gamma correction (OpenGL 2.1)
• Table: sRGB / arbitrary
•...
152
Texture Fetch Functionality (5)
• Optimizations
• Level-of-detail weighting adjustments
• Mid-maps (extra pre-filtered...
153
Phased Data Flow
• Must hide long memory read latency between Selection
and Combination phases
Texel
Selection
Texel
C...
154
What really happens?
• Let’s consider a simple tri-linear mip-mapped 2D
projective texture fetch
• Logically just one ...
155
Medium-Level Dissection
of a Texture Fetch
Convert
texel
coords
to
texel
offsets
integer /
fixed-point
texel
combinati...
156
Interpolation
• First we need to interpolate (s,t,r,q)
• This is the f[TEX3] part of the TXP instruction
• Projective ...
157
Interpolation Operations
• Ax + By + C per scalar linear interpolation
• 2 MADs
• One reciprocal to invert q/w for pro...
158
Texture Space Mapping
• Have interpolated & projected coordinates
• Now need to determine what texels to fetch
• Multi...
159
Mipmap Level-of-detail Selection
• Tri-linear mip-mapping means compute appropriate
mipmap level
• Hardware rasterizes...
160
Level-of-detail Bias and Clamping
• Convert p length to power-of-two level-of-detail and
apply LOD bias
• λ = log2(p) ...
161
Determine Mipmap Levels and
Level Filtering Weight
• Determine lower and upper mipmap levels
• b = floor(λ’)) is botto...
162
Determine Texture Sample Point
• Get (u,v) for selected top and bottom mipmap levels
• Consider a level l which could ...
163
Determine Texel Locations
• Bilinear sample needs 4 texel locations
• (i0,j0), (i0,j1), (i1,j0), (i1,j1)
• With intege...
164
Determine Texel Addresses
• Assuming a texture level image’s base pointer, compute a texel
address of each texel to fe...
165
Initiate Texture Reads
• Initiate texture memory reads at the 8 texel addresses
• addr00, addr01, addr10, addr11 for t...
166
Phased Data Flow
• Must hide long memory read latency between Selection
and Combination phases
Texel
Selection
Texel
C...
167
Texel Combination
• When texels reads are returned, begin filtering
• Assume results are
• Top texels: t00, t01, t10, ...
168
Total Texture Fetch Operations
• Interpolation
• 6 MADs, 3 MULs, & 1 RCP (floating-point)
• Texel selection
• Texture ...
169
Observations about the Texture Fetch
• Lots of ways to implement the math
• Lots of clever ways to be efficient
• Lots...
170
OpenGL’s Future Evolution
Mark Kilgard
Principal System Software Engineer
NVIDIA
171
What drives OpenGL’s future?
• GPU graphics functionality
• Tessellation & geometry amplification
• Ratio of GPU to si...
172
Better Graphics Functionality
• Expect more graphics performance
• Easy prediction
• Rasterization nowhere near peaked...
173
Geometry Amplification
• Tessellation
• Programmable hardware support coming
• True market demand probably not tessell...
174
Programmable Tessellation
• Stunning real-time geometric detail + animation possible
• Programmable tessellation + ver...
175
Continuous Level-of-detail for Tessellation
Increasing tessellation level-of-detail
• Same patch mesh for all 3 scenes
176
Adaptive Programmable Tessellation
Programmable level-of-detail determination allows
more tessellation along silhouett...
177
Limits of Patch Tessellation
• What games tend to want
• Here’s 8 vertices (bounding
box), go draw a fire truck
• Here...
178
Tessellation Not New to OpenGL
• At least three different bi-variate patch tessellation schemes have
been added to Ope...
179
Ratio of CPU core-to-GPU Performance
• Well known computer architecture trends now
• Single-threaded CPU performance t...
180
Déjà vu
• We’ve been here before
• Early 1980s: Graphics terminals used to be
connected to minicomputers by slow speed...
181
What Happened
• Such “scene processor” hardware not very flexible
• Difficult to animate anything beyond rigid dynamic...
182
OpenGL’s Most Powerful Command
• Available since OpenGL 1.0
• Can render essentially anything OpenGL can render!
• Tak...
183
Enhanced Display Lists
• OpenGL 1.0 display lists are too inflexible
• Pixel & vertex data “compiled into” display lis...
184
Other Display List Enhancements
• Conditional display list execution
• Relaxed vertex index and command order
• Parall...
185
Conditional Display List Execution
• Today’s occlusion query
• Application must “query” to learn occlusion result
• La...
186
Relaxed Vertex Index and Command Order
• OpenGL today always executes commands “in order”
• Sequentially requirement
•...
187
Parallel Display List Construction
• Today’s model
• Single thread makes all OpenGL rendering calls
• Minimizes GPU co...
188
Rethinking Display Lists
• Display lists have been proposed for deprecation
• Right as we really need them!
• Much mor...
189
Direct3Disms
• Developing a shader-rich game title costs $$$
• For top titles, often US$ 5,000,000+
• Investment typic...
190
Supporting Direct3D: Not New
• OpenGL has always supported multiple formats well
• OpenGL’s plethora of pixel and vert...
191
Direct3D vs. OpenGL
Coordinate System Conventions
• Window origin conventions
• Direct3D = upper-left origin
• OpenGL ...
192
Direct3D vs. OpenGL
Provoking Vertex Conventions
• Direct3D uses “first” vertex of a triangle or line to
determine whi...
193
BGRA Vertex Array Order
• Direct3D 9’s most common usage for sending per-vertex
colors is 32-bit D3DCOLOR data type:
•...
194
OpenGLisms
• Things about OpenGL’s operation that make it hard for
non-OpenGL applications to port to OpenGL
• Example...
195
Eliminating Selectors from OpenGL
• OpenGL has lots of selectors
• Selectors set state that indicates what state subse...
196
Reasons to Eliminate Selectors
• Direct3D has an “edit-by-name” model of operation
• Means Direct3D has no selectors
•...
197
GLSL Program Object Linking
• GLSL requires shader objects from different domains
(vertex, geometry, fragment) to be l...
198
Separate Shader Objects Example
• Combining different GLSL shaders at once
Specular brick
bump mapping
Red diffuse
Wob...
199
Deprecation
• Part of OpenGL 3.0 is a marking of features for deprecation
• LOTS of functionality is marked for deprec...
200
Synergy between OpenGL and OpenCL
• Complimentary capabilities
• OpenGL 3.0 = state-of-the-art, cross-platform graphic...
201
Four Kinds of Shared Objects
OpenCL 3D image object
cl_mem
OpenGL renderbuffer object
GLuint renderbuffer
OpenGL buffe...
202
OpenGL / OpenCL Sharing
• Requirements for GL object sharing with CL
• CL context must be created with an OpenGL conte...
203
Acquiring GL Objects for Compute Access
• Still must “enqueue acquire” GL objects for compute kernels to
use them
• Ot...
204
Unconventional OpenGL Deployments
• Workstation PCs—Quadro
• Consumer PCs—GeForce
• High-end Visualization—QuadroPlex ...
205
OpenGL in Context
A facilitated conversation
with Dr. Marc Levoy, Stanford University
206
Questions?
Upcoming SlideShare
Loading in...5
×

SIGGRAPH Asia 2008 Modern OpenGL

326,526

Published on

A long-time implementer of OpenGL (Mark Kilgard, NVIDIA) and the system's original architect (Kurt Akeley, Microsoft) explain OpenGL's design and evolution. OpenGL's state machine is now a complex data-flow with multiple programmable stages. OpenGL practitioners can expect candid design explanations, advice for programming modern GPUs, and insight into OpenGL's future.

These slides were presented at SIGGRAPH Asia 2008 for the "Modern OpenGL: Its Design and Evolution" course.

Course abstract: OpenGL was conceived in 1991 to provide an industry standard for programming the hardware graphics pipeline. The original design has evolved considerably over the last 17 years. Whereas capabilities mandated by OpenGL such as texture mapping and a stencil buffer were present only on the world's most expensive graphics hardware back in 1991, now these features are completely pervasive in PCs and now even available in several hand-held devices. Over that time, OpenGL's original fixed-function state machine has evolved into a complex data-flow including several application-programmable stages. And the performance of OpenGL has increased from 100x to over 1,000x in many important raw graphics operations.

In this course, a long-time implementer of OpenGL and the system's original architect explain OpenGL's design and evolution.

You will learn how the modern (post-2006) graphics hardware pipeline is exposed through OpenGL. You will hear Kurt Akeley's personal retrospective on OpenGL's development. You will learn nine ways to write better OpenGL programs. You will learn how modern OpenGL implementations operate. Finally we discuss OpenGL's future evolution.

Whether you program with OpenGL or program with another API such as Direct3D, this course will give you new insights into graphics hardware architecture, programmable shading, and how to best take advantage of modern GPUs.

Published in: Technology, Art & Photos
2 Comments
19 Likes
Statistics
Notes
No Downloads
Views
Total Views
326,526
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
750
Comments
2
Likes
19
Embeds 0
No embeds

No notes for slide
  • An exciting SIGGRAPH for me
  • Didn’t continue to succeed, though.
    One of my sorrows is that OpenGL didn’t seem to contribute to success for SGI
  • Not a required “implementation”, just a concise way to specify the architecture (like ISA registers)
    Directly inspired changes to the specification (especially to pixel operations, e.g., depth buffer of)
  • Not a required “implementation”, just a concise way to specify the architecture (like ISA registers)
    Directly inspired changes to the specification (especially to pixel operations, e.g., depth buffer of)
  • Transcript of "SIGGRAPH Asia 2008 Modern OpenGL"

    1. 1. 1
    2. 2. 2 Mark J. Kilgard, NVIDIA Kurt Akeley, Microsoft Research 13 December 2008 Singapore Modern OpenGL: Its Design and Evolution
    3. 3. 3 Introductions
    4. 4. 4 Kurt Akeley • Led development of OpenGL at Silicon Graphics (SGI) • Co-founded SGI • Lead development of SGI’s high-end graphics hardware • Co-author of OpenGL specification • Returned to Stanford University to complete Ph.D. • Co-developed Cg “C for graphics” language at NVIDIA • Principal Researcher, Microsoft Research Silicon Valley • Spent time at Microsoft Research Asia in Beijing • Member of US National Academy of Engineering
    5. 5. 5 Mark Kilgard • Principal System Software Engineer, NVIDIA, Austin, Texas • Developed original OpenGL driver for 1st GeForce GPU • Specified many key OpenGL extensions • Works on Cg for portable programmable shading • NVIDIA Distinguished Inventor • Before NVIDIA, worked at Silicon Graphics • Worked on X Window System integration for OpenGL • Developed popular OpenGL Utility Toolkit (GLUT) • Wrote book on OpenGL and X, co-authored Cg Tutorial
    6. 6. 6 Marc Levoy • Moderator for our facilitated discussion • Professor of Computer Science and Electrical Engineering • Stanford University • SIGGRAPH Computer Graphics Achievement Award • ACM Fellow
    7. 7. 7 Course Schedule • Modern OpenGL (Kilgard) • OpenGL’s evolution: a personal retrospective (Akeley) • Writing better OpenGL (Kilgard) • Implementing OpenGL (Kilgard) • OpenGL’s future evolution (Kilgard) • OpenGL in Context (Akeley, Kilgard, Levoy) • Facilitated conversation – Mid-session break –
    8. 8. 8 Check Out the Course Notes (1) • Look to www.opengl.org web site for our final slides • New Material • “An Incomplete History of OpenGL” (Kilgard) • How the OpenGL graphics system developed • “Using Vertex Buffer Objects Well” (Kilgard) • Learn how to use Vertex Buffers objects for high vertex processing rates
    9. 9. 9 Check Out the Course Notes (2) • Paper Reprints • OpenGL design rationale from its specification co- authors (Segal, Akeley) • Realizing OpenGL: two implementations of one architecture (Kilgard) • Graphics hardware: GTX, RealityEngine, InfiniteReality, GeForce 6800 • Key developments in graphics hardware design over last 20 years • GPU Programmability: “User-Programmable Vertex Engine” and “Cg” SIGGAPH papers • “How GPUs Work” (Luebke, Humpherys)
    10. 10. 10 Modern OpenGL Mark Kilgard Principal System Software Engineer NVIDIA
    11. 11. 11 Modern OpenGL • History • How did OpenGL get where it is now? • Present • Version 3.0 • Functionality beyond 3.0
    12. 12. 12 An Overview History of OpenGL • Pre-history 1991 • IRIS GL, a proprietary Graphics Library by SGI • OpenGL, an open standard for 3D • Focus: procedural hardware-accelerated 3D graphics • Governed by Architectural Review Board (ARB) • Extensibility planned into design • Competition • Proprietary APIs (1991-1995) • PHIGS & PEX for X Window System (1992-1997) • Microsoft’s Direct3D (1998-)
    13. 13. 13 OpenGL’s Pre-history IRIS GL 1 Window system: MEX IRIS GL 2 Window system: MEX Operating system: UNIX IRIS GL 3 Window system: NeWS/X11 Operating system: IRIX 3.x IRIS GL 4 Window system: Native X11 Operating system: IRIX 4.3 OpenGL 1.0 Window system: Native X11 with GLX Operating system: IRIX 5.1 1991 1993 1988 1986 1983 First work on GL 5.0 proposal 1989 Dates are for shipping commercial SGI implementation 1983-2008 = 25 years
    14. 14. 14 OpenGL’s Design Philosophy • High-performance • Assumes hardware acceleration • Defined by a specification • Rather than a de-facto implementation • Rendering state machine • Procedural • Not a window system, not a scene graph • No initial sub-setting • Extensible • Data type rich • Cross-platform • Window system- independent core • X Window System, Microsoft Windows, OS/2, OS X, etc. • Multi-language bindings • C, FORTRAN, etc. • Not merely an API, rather a system
    15. 15. 15 Timeline of OpenGL’s Development 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC)
    16. 16. 16 Competitive 3D APIs • OpenGL has always existed in competition with other APIs • Strengthened OpenGL by driving feature parity • OpenGL’s competitive strengths: 1. Cross platform, open process 2. API stability, extensibility 3. Clean initial design & specification 1992 1994 1996 1998 2000 2002 2004 2006 2008 Proprietary Unix workstation 3D APIs XGL Doré Starbase IRIS GL X Consortium 3D standard PEX Microsoft Direct3D DirectX 3 DirectX 5 DirectX 6 DirectX 7 DirectX 8 DirectX 9 DirectX 10
    17. 17. 17 OpenGL 1.0 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) •Immediate mode •Vertex transformation and lighting •Points, lines, polygons •Stippling, wide points and lines •Bitmaps, image rectangles, and pixel reads •Pixel store and transfer •1D and 2D textures, fog, and scissor •Display lists and evaluators •RGBA and color index color models •Color, depth, stencil, and accumulation buffers •Selection and feedback modes •Queries
    18. 18. 18 OpenGL State Machine • From OpenGL 3.0 specification, unchanged since 1.0
    19. 19. 19 SGI “Classic” Hardware View of OpenGL 3D Application or Game • Entirely fixed-function, no programmability • High-end SGI hardware manifested functionality in distinct chips OpenGL API Front End Vertex Assembly Vertex Transform & Lighting Primitive Assembly, Clipping, Setup, and Rasterization Texture & Fog Texture Fetch Raster Operations Framebuffer Access Memory Interface Graphics Hardware Boundary 1992 Graphics data flow Memory operations Fixed-function unit Programmable unit
    20. 20. 20 OpenGL 1.1 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • Vertex arrays • Texture objects • Texture internal formats • Texture sub-image updates • Texture proxies • Copy framebuffer-to-texture • Polygon offset • RGBA logical operations
    21. 21. 21 The Look of OpenGL 1.1 SGI skyfly demoSGI skyfly demo StenciledStenciled shadow volumesshadow volumes Ideas in MotionIdeas in Motion
    22. 22. 22 OpenGL 1.2 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • 3D textures • Texture edge clamp wrap mode • Texture level-of-detail clamping • BGRA component order • Packed pixel formats • Imaging subset (optional) • Normal rescaling • Separate specular • Vertex array draw elements range
    23. 23. 23 Akeley’s (Modernized) OpenGL Data Flow vertex shading rasterization & fragment shading texture raster operations framebuffer pixel unpack pixel pack vertex puller client memory pixel transfer glReadPixels / glCopyPixels / glCopyTex{Sub}Image glDrawPixels glBitmap glCopyPixels glTex{Sub}Image glCopyTex{Sub}Image glDrawElements glDrawArrays selection / feedback / transform feedback glVertex* glColor* glTexCoord* etc. blending depth testing stencil testing accumulation storage operations
    24. 24. 24 OpenGL 1.2 Imaging Subset Color Table Convolution (separable or general) Post-convolve Scale & Bias Post-convolve Color Table Color Matrix Post-color matrix Scale & Bias Post-color matrix Color Table Histogram Min-max Look-up Table (RGBA-to-RGBA) Look-up Table (Index-to-RGBA) Scale & Bias Shift & Add Index pixels RGBA pixels Pixel Rectangle Rasterization core functionality ARB_imaging subset discard discard
    25. 25. 25 OpenGL 1.2.1 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • Multi-texture (optional)
    26. 26. 26 Multi-texture Poster Child: Quake 2 Light Maps × (modulate) = lightmaps onlylightmaps only decal onlydecal only combined scenecombined scene
    27. 27. 27 GeForce 256 (NV10) View of OpenGL 3D Application or Game • Vertex pulling (vertex buffer objects) via DMA • Dual-texture, cube maps, and register combiners OpenGL API GPU Front End Vertex Assembly Vertex Transform & Lighting Primitive Assembly, Clipping, Setup, and Rasterization Texture & Fog Texture Fetch Raster Operations Framebuffer Access Memory Interface CPU – GPU Boundary 1999 Attribute Fetch
    28. 28. 28 Hardware Cube Maps Rendered sceneRendered scene DynamicallyDynamically createdcreated cube map imagecube map image Image credit: “Guts” GeForce 2 GTS demo, Thant Thessman
    29. 29. 29 OpenGL 1.3 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • Multi-texture (required now) • Cube map texturing • Compressed texture formats • Texture border clamp • Texture environment functions • Add, combine, dot product • Multisample anti-aliasing • Transpose matrix
    30. 30. 30 GeForce 3 & 4 Ti (NV2x) View of OpenGL 3D Application or Game • Programmable vertex processing • Highly configurable fragment processing OpenGL API GPU Front End Vertex Assembly Vertex Program Primitive Assembly, Clipping, Setup, and Rasterization Multi-texture shaders & Combiners Texture Fetch Raster Operations Framebuffer Access Memory Interface CPU – GPU Boundary 2001 Attribute Fetch
    31. 31. 31 Vertex Programmability Paletted matrixPaletted matrix skinningskinning Twister vertex programTwister vertex program Per-vertexPer-vertex cartooncartoon shadingshading
    32. 32. 32 Configurable Fragment Processing Bumpy shiny environment mappingBumpy shiny environment mappingChromaticChromatic aberrationaberration Offset 2D bumpOffset 2D bump mappingmapping Depth spritesDepth sprites
    33. 33. 33 OpenGL 1.4 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • Automatic mipmap generation • Shadow-mapping • Depth textures and shadow comparisons • Texture level-of-detail bias • Texture mirrored repeat wrap mode • Multi-texture combination • Fog coordinate • Secondary color • Configurable point size attenuation • Color blending improvements • Stencil wrap operations • Window-space raster position specification
    34. 34. 34 Hardware Shadow Mapping Without shadow mappingWithout shadow mapping WithWith shadow mappingshadow mapping Depth map from lightDepth map from light source’s viewsource’s view Darker is closerDarker is closer lightlight positionposition Projective Texturing (1.0) & Polygon Offset (1.1) key enablers
    35. 35. 35 Shadow Mapping Explained Planar distance from lightPlanar distance from light Depth map projected onto sceneDepth map projected onto scene ≤≤ == lessless thanthan True “un-shadowed”True “un-shadowed” region shown greenregion shown green equalsequals
    36. 36. 36 OpenGL 1.5 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • Vertex buffer objects (VBOs) • Occlusion queries • Generalized shadow mapping functions
    37. 37. 37 GeForce FX (NV3x) View of OpenGL 3D Application or Game • Programmable fragment processing • 16 texture units, IEEE 754 32-bit floating-point • Vertex program branching OpenGL API GPU Front End Vertex Assembly Vertex Program Primitive Assembly, Clipping, Setup, and Rasterization Fragment Program Texture Fetch Raster Operations Framebuffer Access Memory Interface CPU – GPU Boundary 2003 Attribute Fetch
    38. 38. 38 Floating-point Fragment Programmability
    39. 39. 39 OpenGL Fragment Program Flowchart More Instructions? Read Interpolants and/or Registers Map Input values: Swizzle, Negate, etc. Perform Instruction Math / Operation Write Output Register with Masking Begin Fragment Fetch & Decode Next Instruction Temporary Registers initialized to 0,0,0,0 Output Depth & Color Registers initialized to 0,0,0,1 Initialize Parameters Emit Output Registers as Transformed Vertex End Fragment Fragment Program Instruction Loop Fragment Program Instruction Memory Texture Fetch Instruction? yes no no Compute Texture Address & Level- of-detail & Fetch Texels Filter Texels yes Texture Images Primitive Interpolants
    40. 40. 40 Key Trend: Configurability becomes Programmability Fixed-function Programmable Simple Configurability Complex Configurability
    41. 41. 41 Core OpenGL fragment texturing & coloring Point Rasterization Line Rasterization Polygon Rasterization Pixel Rectangle Rasterization Bitmap Rasterization From Primitive Assembly DrawPixels Bitmap Conventional Texture Fetching Texture Environment Application Color Sum Fog To raster operations Coverage Application Texture Unit 0 Texture Unit 1 Texture Unit 0 Texture Unit 1
    42. 42. 42 NV1x OpenGL fragment texturing & coloring Point Rasterization Line Rasterization Polygon Rasterization Pixel Rectangle Rasterization Bitmap Rasterization From Primitive Assembly DrawPixels Bitmap Conventional Texture Fetching Texture Environment Application Color Sum Fog To raster operations Coverage Application Register Combiners Texture Unit 0 General Stage 1 Final Stage Texture Unit 1 General Stage 0 Texture Unit 0 Texture Unit 1 GL_REGISTER_COMBINERS_NV enable
    43. 43. 43 Texture Shader 3 … Texture Shader 1 Texture Shader 0 Register Combiners NV2x OpenGL fragment texturing & colorin Point Rasterization Line Rasterization Polygon Rasterization Pixel Rectangle Rasterization Bitmap Rasterization From Primitive Assembly DrawPixels Bitmap Conventional Texture Fetching Texture Environment Application Color Sum Fog To raster operations Coverage Application Texture Shaders General Stage 1 Final Combiner General Stage 0 General Stage 7 …Texture Unit 3 … Texture Unit 1 Texture Unit 0 Texture Unit 3 … Texture Unit 1 Texture Unit 0 GLTEXTURE_SHADER_NV enable GL_REGISTER_COMBINERS_NV enable
    44. 44. 44 Fragment Program Instruction 0 Texture Shader 3 … Texture Shader 1 Texture Shader 0 NV3x OpenGL fragment texturing & coloring Point Rasterization Line Rasterization Polygon Rasterization Pixel Rectangle Rasterization Bitmap Rasterization From Primitive Assembly DrawPixels Bitmap Conventional Texture Fetching Texture Environment Application Color Sum Fog To raster operations Coverage Application Texture Shaders General Stage 1 Final Combiner General Stage 0 General Stage 7 … Texture Unit 3 … Texture Unit 1 Texture Unit 0 Texture Unit 3 … Texture Unit 1 Texture Unit 0 … Fragment Program Fragment Program Instruction 1023 GL_REGISTER_COMBINERS_NV enable GLTEXTURE_SHADER_NV enable GL_FRAGMENT_PROGRAM_NV enable !!FP1.0 or !!ARBfp1.0 programs
    45. 45. 45 OpenGL 2.0 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • Programmable shading • OpenGL Shading Language (GLSL) • Multiple color buffer rendering targets • Non-power-of-two texture dimensions • Point sprites • Separate blend equation • Two-sided stencil testing
    46. 46. 46 GeForce 6 & 7 (NV4x/G7x) View of OpenGL 3D Application or Game • Limited vertex texturing • Fragment branching • Multiple render targets & floating-point blending OpenGL API GPU Front End Vertex Assembly Vertex Program Primitive Assembly, Clipping, Setup, and Rasterization Fragment Program Texture Fetch Raster Operations Framebuffer Access Memory Interface CPU – GPU Boundary 2004 Attribute Fetch
    47. 47. 47 Primitive Program GeForce 8 & 9 (G8x/G9x) View of OpenGL 3D Application or Game • Primitive (geometry) programs • Parameter reads from buffer objects • Transform feedback (stream out) OpenGL API GPU Front End Vertex Assembly Vertex Program , Clipping, Setup, and Rasterization Fragment Program Texture Fetch Raster Operations Framebuffer Access Memory Interface CPU – GPU Boundary 2006 Attribute Fetch Primitive Assembly Parameter Buffer Read
    48. 48. 48 Primitive Program OpenGL Pipeline Fixed-function Steps • Much of functional pipeline remains fixed-function • Vital to maintaining performance and data flow • Hard to compete with hard-wired rasterization, Zcull, and pixel compression GPU Front End Vertex Assembly Vertex Program , Clipping, Setup, and Rasterization Fragment Program Texture Fetch Raster Operations Framebuffer Access Memory Interface 2006 Attribute Fetch Primitive Assembly Parameter Buffer Read
    49. 49. 49 Primitive Program OpenGL Pipeline Programmable Domains • New geometry shader domain for per-primitive programmable processing • Unified Streaming Processor Array (SPA) architecture means same capabilities for all domains GPU Front End Vertex Assembly Vertex Program , Clipping, Setup, and Rasterization Fragment Program Texture Fetch Raster Operations Framebuffer Access Memory Interface 2006 Attribute Fetch Primitive Assembly Parameter Buffer Read Can be unified hardware!
    50. 50. 50 OpenGL 2.1 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • OpenGL Shading Language (GLSL) improvements • Non-square matrices • Pixel buffer objects (PBOs) • sRGB color space texture formats
    51. 51. 51 OpenGL 3.0 1992 1994 1996 1998 2000 2002 2004 2006 2008 OpenGL 1.0 approved OpenGL 1.1 OpenGL 1.2 Multitexture added (1.2.1) OpenGL 1.3 OpenGL 1.4 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 SGI Infinite- Reality OpenGL Utility Toolkit (GLUT) released Mesa 3D open source Khronos controls OpenGL 1st GPU for PCs with single-chip transform & lighting for OpenGL (GeForce) NT 3.51 bring OpenGL to PCs OpenGL ES for embedded devices 1st commercial OpenGL implementation (DEC) • OpenGL Shading Language (GLSL) improvements • New texture fetches • True integer data types and operators • switch/case/default flow control statements • Conditional rendering based on occlusion query results • Transform feedback • Vertex array objects • Floating-point textures, color buffers, and depth buffers • Half-precision vertex arrays • Texture arrays • Integer textures • Red and red-green texture formats • Compressed red and red-green formats • Framebuffer objects (FBOs) • Packed depth-stencil pixel formats • Per-color buffer clearing, blending, and masking • sRGB color space color buffers • Fine-grain buffer mapping and flushing
    52. 52. 52 Areas of 3.0 Functionality Improvement • Programmability • Shader Model 4.0 features • OpenGL Shading Language (GLSL) 1.30 • Texturing • New texture representations and formats • Framebuffer operations • Framebuffer objects • New formats • New copy (blit), clear, blend, and masking operations • Buffer management • Non-blocking and fine-grain update of buffer object data stores • Vertex processing • Vertex array configuration objects • Conditional rendering for occlusion culling • New half-precision vertex attribute formats • Pixel processing • New half-precision external pixel formats All Brand New Core Features
    53. 53. 53 OpenGL 3.0 Programmability • Shader Model 4.0 additions • True signed & unsigned integer values • True integer operators: ^, &, |, <<. >>, %,~ • Texture additions • Texture arrays • Base texture size queries • Texel offsets to fetches • Explicit LOD and derivative control • Integer samplers • Interpolation modifiers: centroid, noperspective, and flat • Vertex array element number: gl_VertexID • OpenGL Shading Language (GLSL) improvements • ## concatenation in pre-processor for macros • switch/case/default statements
    54. 54. 54 OpenGL 3.0 Texturing Functionality • Texture representation • Texture arrays: indexed access to a set of 1D or 2D texture images • Texture formats • Floating-point texture formats • Single-precision (32-bit, IEEE s23e8) • Half-precision (16-bit, s10e5) • Red & red/green texture formats • Intended as FBO framebuffer formats too • Compressed red & red/green texture formats • Shared exponent texture formats • Packed floating-point texture formats
    55. 55. 55 Texture Arrays • Conventional texture = One logical pre-filtered image • Texture array = index-able plurality of pre-filtered images • Rationale is fewer texture object binds when drawing different objects • No filtering between mipmap sets in a texture array • All mipmap sets in array share same format/border & base dimensions • Both 1D and 2D texture arrays • Require shaders, no fixed-function support • Texture image specification • Use glTexImage3D, glTexSubImage3D, etc. to load 2D texture arrays • No new OpenGL commands for texture arrays • 3rd dimension specifies integer array index • No halving in 3rd dimension for mipmaps • So 64×128x17 reduces to 32×64×17 all the way to 1×1×17
    56. 56. 56 Texture Arrays Example • Multiple skins packed in texture array • Motivation: binding to one multi-skin texture array avoids texture bind per object Texture array index 0 1 2 3 4 0 1 2 3 4 Mipmaplevelindex
    57. 57. 57 Compact Floating-point Textures • Shared exponent & packed float representations are ideal of High Dynamic Range (HDR) applications
    58. 58. 58 Compact Floating-point Texture Formats • Packed float format • No sign bit, independent exponents • Shared exponent format • No sign bit, shared exponent, no implied leading 1 5-bit mantissa 5-bit exponent 6-bit mantissa 5-bit exponent 6-bit mantissa 5-bit exponent bit 31 bit 0 9-bit mantissa 5-bit shared exponent 9-bit mantissa 9-bit mantissa bit 31 bit 0
    59. 59. 59 1- and 2-component Block Compression Scheme • Basic 1-component block compression format • Borrowed from alpha compression scheme of S3TC 5 8-bit B8-bit A 2 min/max values 64 bits total per block + 4x4 Pixel Decoded BlockEncoded Block 16 pixels x 8-bit/componet = 128 bits decoded so effectively 2:1 compression 16 bits
    60. 60. 60 Framebuffer Operations • Framebuffer objects • Standardized framebuffer objects (FBOs) for rendering to textures and renderbuffers • Render-to-texture • Multisample renderbuffers for FBOs • Framebuffer operations • Copies from one FBO to another, including multisample data • Per-color attachment color clears, blending, and write masking • Framebuffer formats • Floating-point color buffers • Floating-point depth buffers • Rendering into framebuffer format with 3 small unsigned floating- point values packed in a 32-bit value • Rendering into sRGB color space framebuffers
    61. 61. 61 Framebuffer Object Example • Depth peeling for correctly ordered transparency • Great render-to-texture application for FBOs
    62. 62. 62 Depth Peeling Behind the Scenes • Depth buffer has closest fragment at all pixels • Save depth buffer • Render again, but use depth buffer as shadow map • Discard fragment in front of shadow map’s depth value • Effectively peels one layer of depth! • Resulting color buffer is 2nd closest fragment • And depth buffer for 2nd closest fragments’ depth • Now repeat peeling more layers • Use ping-pong depth buffer scheme • Use occlusion query to detect when no more fragments to peel • Composite color layers front-to-back (or back- to-front) • Front-to-back peeling can be done during the peeling process
    63. 63. 63 Delicate Color Fidelity with sRGB • Problem: PC display devices have non-linear (sRGB) display gamut —delicate color shading looks wrong Conventional rendering (uncorrected color) Gamma correct (sRGB rendered) Softer and more natural Unnaturally deep facial shadows NVIDIA’s Adriana GeForce 8 Launch Demo
    64. 64. 64 What is sRGB? • A standard color space • Intended for monitors, printers, and the Internet • Created cooperatively by HP and Microsoft • Non-linear, roughly gamma of 2.2 • Intuitively “encodes more dark values” • OpenGL 2.1 already added sRGB texture formats • Texture fetch converts sRGB to linear RGB, then filters • Result takes more than 8-bit fixed-point to represent in shader • 3.0 adds complementary sRGB framebuffer support • “sRGB correct blending” converts framebuffer sRGB to linear, blend with linear color from shader, then convert back to sRGB • Works with FrameBuffer Objects (FBOs) sRGB chromaticity
    65. 65. 65 So why sRGB? Standard Windows Display is Not Gamma Corrected • 25+ years of PC graphics, icons, and images depend on not gamma correcting displays • sRGB textures and color buffers compensates for this “Expected” appearance of Windows desktop & icons but 3D lighting too dark Wash-ed out desktop appearance if color response was linear but 3D lighting is correct Gamma 1.0 Gamma 2.2 linear color response
    66. 66. 66 Vertex Processing • Vertex array configuration • Objects to manage vertex array configuration client state • Half-precision floating-point vertex array formats • Vertex output streaming • Stream transformed vertex results into buffer object data stores • Occlusion culling • Skip rendering based on occlusion query result
    67. 67. 67 Miscellaneous • Pixel Processing • Half-precision floating-point pixel external formats • Buffer Management • Non-blocking and fine-grain update of buffer object data stores
    68. 68. 68 ARB Extensions to OpenGL 3.0 • OpenGL 3.0 standard provides new ARB extensions • Extensions go beyond OpenGL 3.0 • Standardized at same time as OpenGL 3.0 • Support features in hardware today • Specifically • ARB_geometry_shader4—provides per-primitive programmable processing • ARB_draw_instanced—gives shader access to instance ID • ARB_texture_buffer_object—allows buffer object to be sampled as a huge 1D unfiltered texture • Shipping today • NVIDIA driver provides all three
    69. 69. 69 Transform Feedback for Terrain Generation by Recursive Subdivision • Geometry shaders + transform feedback 1. Render quads (use 4-vertex line adjacency primitive) from vertex buffer object 2. Fetch height field 3. Stream subdivided positions and normals to transform feedback “other” buffer object 4. Use buffer object as vertex buffer 5. Repeat, ping-pong buffer objects Computation and data all stays on the GPU!
    70. 70. 70 Skin Deformation • Capture & re-use geometric deformations Transform feedback allows the GPU to calculate the interactive, deforming elastic skin of the frog
    71. 71. 71 Silhouette Edge Rendering • Uses geometry shader silhouette edge detection geometry shader Complete mesh Silhouette edges Useful for non-photorealistic rendering Looks like human sketching
    72. 72. 72 More Geometry Shader Examples Shimmering point sprites Generate fins for lines Generate shells for fur rendering
    73. 73. 73 Improved Interpolation Techniques •Using geometry shader functionality Quadratic normal interpolation True quadrilateral rendering with mean value coordinate interpolation
    74. 74. 74 “Fair” Quadrilateral Interpolation • glBegin(GL_QUADS); • glColor3fv(red); glVertex3fv(lowerLeft); • glColor3fv(green); glVertex3fv(lowerRight); • glColor3fv(red); glVertex3fv(upperRight); • glColor3fv(blue); glVertex3fv(upperLeft); • glEnd(); • Geometry shader actually operates on 4-vertex GL_LINE_ADJACENCY primitives instead of quads Wrong, slash triangle split Wrong, backslash triangle split Better: Mean value coordinates
    75. 75. 75 OpenGL 2.x ARB Extensions • Many OpenGL 3.0 extensions have corresponding ARB extensions for OpenGL 2.1 implementations to advertise • Helps get 3.0 functionality out sooner, rather than later • New ARB extensions for 3.0 functionality • ARB_framebuffer_object—framebuffer objects (FBOs) for render-to- texture • ARB_texture_rg—red and red/green texture formats • ARB_map_buffer_region—non-blocking and fine-grain update of buffer object data stores • ARB_instanced_arrays—instance ID available to shaders • ARB_half_float_vertex—half-precision floating-point vertex array formats • ARB_framebuffer_sRGB—rendering into sRGB color space framebuffers • ARB_texture_compression_rgtc—compressed red and red/green texture formats • ARB_depth_buffer_float—floating-point depth buffers • ARB_vertex_array_object—objects to manage vertex array configuration client state
    76. 76. 76 Beyond OpenGL 3.0 OpenGL 3.0 • EXT_gpu_shader4 • NV_conditional_render • ARB_color_buffer_float • NV_depth_buffer_float • ARB_texture_float • EXT_packed_float • EXT_texture_shared_exponent • NV_half_float • ARB_half_float_pixel • EXT_framebuffer_object • EXT_framebuffer_multisample • EXT_framebuffer_blit • EXT_texture_integer • EXT_texture_array • EXT_packed_depth_stencil • EXT_draw_buffers2 • EXT_texture_compression_rgtc • EXT_transform_feedback • APPLE_vertex_array_object • EXT_framebuffer_sRGB • APPLE_flush_buffer_range (modified) In GeForce 8, 9, & 2xx Series but not yet core • EXT_geometry_shader4 (now ARB) • EXT_bindable_uniform • NV_gpu_program4 • NV_parameter_buffer_object • EXT_texture_compression_latc • EXT_texture_buffer_object (now ARB) • NV_framebuffer_multisample_coverage • NV_transform_feedback2 • NV_explicit_multisample • NV_multisample_coverage • EXT_draw_instanced (now ARB) • EXT_direct_state_access • EXT_vertex_array_bgra • EXT_texture_swizzle Plenty of proven OpenGL extensions for OpenGL Working Group to draw upon for OpenGL 3.1
    77. 77. 77 OpenGL Version Evolution • Now OpenGL is part of Khronos Group • Previously OpenGL’s evolution was governed by the OpenGL Architectural Review Board (ARB) • Now officially a Khronos working group • Khronos also standardizes OpenCL, OpenVG, etc. • How OpenGL version updates happen • OpenGL participants proposing extensions • Successful extensions are polished and incorporated into core • OpenGL 3.0 is great example of this process • Roughly 20 extensions folded into “core” • Just 3 of those previously unimplemented
    78. 78. 78 29% 17% 15% 15% 4% 2% 2% 2% 2% 2% 2% 2% 1% 1% 4% 15% Multi-vendor Silicon Graphics Architectural Review Board NVIDIA ATI Apple Mesa3D Sun Microsystems OpenGL ES OpenML IBM Intense3D Hewlett Packard 3Dfx Other EXT SGI SGIS SGIX ARB NV Others Others OpenGL Extensions by Source • 44% of extensions are “core” or multi-vendor • Lots of vendors have initiated extensions • Extending OpenGL is industry-wide collaboration ATI APPLE MESA Source: http://www.opengl.org/registry (Dec 2008)
    79. 79. 79 What’s Driving OpenGL Modernization? Human desire for Visual Intuition and Entertainment Embarrassing Parallelism of Graphics Increasing Semiconductor Density Particularly the hardware-amenable, latency tolerant nature of rasterization Particularly interactive video games
    80. 80. 80 Kurt Akeley Principal Researcher Microsoft Research Silicon Valley OpenGL’s Evolution: A Personal Retrospective
    81. 81. 81 AA personalpersonal retrospectiveretrospective • My background: • Silicon Graphics, 1982-2001 • OpenGL, 1990-2004 • Today’s topics: • Computer architecture • Culture and process • For a more complete coverage see: • https://graphics.stanford.edu/wikis/cs448-07-spring/ • Mark Kilgard’s excellent course notes
    82. 82. 82 Jim Clark and the Geometry EngineJim Clark and the Geometry Engine • This text is 24 points – Sub bullets look like this The Geometry Engine: A VLSI Geometry System for Graphics Computer Graphics, Volume 16, Number 3 (Proceedings of SIGGRAPH 1982) p127-133, 1982
    83. 83. 83 Jim’s helpers: the Stanford gangJim’s helpers: the Stanford gang IRIS GL Geometry Engine IRIS GL Hardware back-end Hardware front-end
    84. 84. 84 Success!Success! (in 1995)(in 1995)
    85. 85. 85 Computer Architecture
    86. 86. 86 What is computer architecture?What is computer architecture? • Architecture: “the minimal set of properties that determine what programs will run and what results they will produce” • Implementation: “the logical organization of the [computer’s] dataflow and controls” • Realization: “the physical structure embodying the implementation”
    87. 87. 87 Example: the analog clockExample: the analog clock • Architecture • Circular dial divided into twelfths • Hour hand (short) and minute hand (long) Example from Computer Architecture, Concepts and Evolution, Gerrit A. Blaauw and Frederick P. Brooks, Jr., Addison-Wesley, 1997 • Implementation • A weight, driving a pendulum, or • A spring, driving a balance wheel, or • A battery, driving an oscillator, or …. • Realization • Gear ratios, pendulum lengths, battery sizes, ... 12 11 10 6 8 9 7 5 4 2 1 3
    88. 88. 88 A useful distinctionA useful distinction • NVIDIA 8800 • SIMD, or • SPMD ? L2 FB SP SP L1 TF ThreadProcessor Vertex Thread Issue Setup / Rasterization / ZCull Primitive Thread Issue Fragment Thread Issue Data Assembler Application SP SP L1 TF SP SP L1 TF SP SP L1 TF SP SP L1 TF SP SP L1 TF SP SP L1 TF SP SP L1 TF L2 FB L2 FB L2 FB L2 FB L2 FB • Architecture: • SPMD • Implementation: • SIMD • Realization: • ASIC SIMD = Single Instruction, Multiple Data SPMD = Single Program, Multiple Data ASIC = Application Specific Integrated Circuit
    89. 89. 89 The mainstream viewThe mainstream view • Table of Contents: • Fundamentals • Instruction Sets • Pipelining • Advanced Pipelining and ILP • Memory-Hierarchy Design • Storage Systems • Interconnection Networks • Multiprocessors
    90. 90. 90 OpenGL is an architecture Blaauw/Brooks OpenGL Different implementations IBM 360 30/40/50/65/75 Amdahl SGI Indy/Indigo/InfiniteReality NVIDIA GeForce, ATI Radeon, … Compatibility Code runs equivalently on all implementations Top-level goal Conformance tests, … Intentional design It’s an architecture, whether it was planned or not . Carefully planned, though mistakes were made Configuration Can vary amount of resource (e.g., memory) No feature sub-setting Configuration attributes (e.g., framebuffer) Speed Not a formal aspect of architecture No performance queries Validity of inputs No undefined operation All errors specified No side effects Little undefined operation Enforcement When implementation errors are found, they are fixed. Specification rules!
    91. 91. 91 But OpenGL is an APIBut OpenGL is an API (Application Programming Interface)(Application Programming Interface) • Yes, Blaauw and Brooks talk about (computer) architecture as though it is always expressed as ISA (Instruction-Set Architecture) • But … • API is just a higher-level programming interface • “Instruction-Set” Architecture implies other types of computer architectures (such as “API” Architecture) • OpenGL has evolved to include ISA-like interfaces (e.g., the interface below GLSL)
    92. 92. 92 We didn’t know …We didn’t know … • No mention in spec (even 3.0) • “We view OpenGL as a state …” • First use in “ARB” • Architecture Review Board • Coined by Bill Glazier from “Palo Alto Architecture Review Board” • First formal usage (I know of) • Mark J. Kilgard, Realizing OpenGL: two implementations of one architecture, Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on Graphics hardware, p.45-55, August 03-04, 1997, Los Angeles, California, United States.
    93. 93. 93 Fred is magnanimousFred is magnanimous
    94. 94. 94 What is implied by “programmable”?What is implied by “programmable”? • What does it mean to teach programming? • Does running a microwave oven count? • Does defining the geometry of a game “level” count? • Does specifying OpenGL modes count? • This seems to be a somewhat open question • Butler Lampson couldn’t tell me . • Microsoft developers of teaching tools couldn’t tell me. • An online search wasn’t very helpful. • Do we just “know it when we see it”? • Justice Potter Stewart’s definition of pornography
    95. 95. 95 My try at some formalizationMy try at some formalization • Key ideas: • Composition  choice of placement, sequence • Non-obvious  semantics are interesting and novel • Imperative  maybe there are other kinds of programming “Composition, the organization of elemental operations into a non-obvious whole, is the essence of imperative programming.” -- Kurt Akeley (Foreword to GPU Gems 3)
    96. 96. 96 OpenGL has always been programmableOpenGL has always been programmable • Follows directly from being an “architecture” • OpenGL commands are instructions (API as an ISA) • They can be “composed” to create programs • Multi-pass rendering is the prototypical example • But Peercy et al. implemented a RenderMan shader compiler • Invariance was specified from the start (e.g., same fragments) • We set out to enable “usage that we didn’t anticipate” • Obvious for a traditional ISA (e.g., IA32) • Not so obvious for a graphics API • Example: texture applies to all primitives, not just triangles
    97. 97. 97 Example multi-pass OpenGL “program”Example multi-pass OpenGL “program” glEnable(GL_DEPTH_TEST); glDisable(GL_LIGHTING); glColorMask(false, false, false, false); glEnable(GL_POLYGON_OFFSET_FILL); glPolygonOffset(maxwidth/2, 1); draw solid objects glDepthMask(GL_FALSE); glColorMask(true, true, true, true); glColor3f(linecolor); glDisable(GL_POLYGON_OFFSET_FILL); glPolygonMode(GL_FRONT_AND_BACK, GL_LINE); draw solid objects again glDisable(GL_DEPTH_TEST); glPolygonMode(GL_FRONT_AND_BACK, GL_FILL); glDepthMask(GL_TRUE); Hidden-line rendering
    98. 98. 98 Example multi-pass OpenGL “program”Example multi-pass OpenGL “program” glEnable(GL_DEPTH_TEST); glDisable(GL_LIGHTING); glColorMask(false, false, false, false); glEnable(GL_POLYGON_OFFSET_FILL); glPolygonOffset(maxwidth/2, 1); draw solid objects glDepthMask(GL_FALSE); glColorMask(true, true, true, true); glColor3f(1, 1, 1); glDisable(GL_POLYGON_OFFSET_FILL); glPolygonMode(GL_FRONT_AND_BACK, GL_LINE); glEnable(GL_CULL_FACE); glCullFace(GL_FRONT); draw solid objects again draw true edges // for a complete hidden-line drawing glDisable(GL_DEPTH_TEST); glPolygonMode(GL_FRONT_AND_BACK, GL_FILL); glDepthMask(GL_TRUE); glDisable(GL_CULL_FACE); Additions to the hidden-line algorithm (previous slide) highlighted in red Silhouette rendering
    99. 99. 99 InvarianceInvariance Corollary 1 Fragment generation is invariant with respect to the state values marked with in Rule 2.
    100. 100. 100 • Intended to capture complete sequence of operations • Also inspired design changes
    101. 101. 101 Vertex assembly Primitive assembly Rasterization Fragment operations Display Vertex operations Application Primitive operations Framebuffer Texture memory Pixel assembly (unpack) Pixel operations Pixel pack Vertex pipelinePixel pipeline Application All primitives (including pixels) are rasterized All vertexes are treated equally (e.g., lighted) All fragments are treated equally (e.g., texture mapped and depth-buffered) Not a required implementation, but “abstraction distance” matters
    102. 102. 102 Culture and Process
    103. 103. 103 Suppose …Suppose … http://www.opengl.org/registry/ Name ARB_texture_cube_map Name Strings GL_ARB_texture_cube_map Notice Copyright OpenGL Architectural Review Board, 1999. Contact Michael Gold, NVIDIA (gold 'at' nvidia.com) Status Complete. Approved by ARB on 12/8/1999 Version Last Modified Date: December 14, 1999 Number ARB Extension #7 Dependencies None. Written based on the wording of the OpenGL 1.2.1 specification but not dependent on it. Overview This extension provides a new texture generation scheme for cube map textures. Instead of the current texture providing a 1D, 2D, or 3D lookup into a 1D, 2D, or 3D texture image, the texture is a set of six 2D images representing the faces of a cube. The (s,t,r) texture coordinates …
    104. 104. 104 Complete specificationComplete specification Name Name Strings Notice Contact Status Version Number Dependencies Overview Issues New Procedures and Functions New Tokens Additions to Chapter 2 of the OpenGL Specification Additions to Chapter 3 of the OpenGL Specification Additions to Chapter 4 of the OpenGL Specification Additions to Chapter 5 of the OpenGL Specification Additions to Chapter 6 of the OpenGL Specification Additions to the GLX Specification Errors New State (type, query mechanism, initial value, attribute set, specification section) Usage Examples
    105. 105. 105 19 issues19 issues The spec just linearly interpolates the reflection vectors computed per-vertex across polygons. Is there a problem interpolating reflection vectors in this way? Probably. The better approach would be to interpolate the eye vector and normal vector over the polygon and perform the reflection vector computation on a per-fragment basis. Not doing so is likely to lead to artifacts because angular changes in the normal vector result in twice as large a change in the reflection vector as normal vector changes. The effect is likely to be reflections that become glancing reflections too fast over the surface of the polygon. Note that this is an issue for REFLECTION_MAP_ARB, but not NORMAL_MAP_ARB.
    106. 106. 106 19 issues …19 issues … What happens if an (s,t,q) is passed to cube map generation that is close to (0,0,0), ie. a degenerate direction vector? RESOLUTION: Leave undefined what happens in this case (but may not lead to GL interruption or termination). Note that a vector close to (0,0,0) may be generated as a result of the per-fragment interpolation of (s,t,r) between vertices.
    107. 107. 107 Trust and integrityTrust and integrity • Lots of collaboration during the initial design • But final decisions made by a small group • SGI played fair • OpenGL 1.0 didn’t favor SGI equipment (our ports were late ) • SGI obeyed all conformance rules • SGI didn’t adjust the spec to match our equipment • The ARB avoided marketing tasks such as benchmarks • We stuck with technical design issues • We documented rigorously • Specification, man pages, …
    108. 108. 108 Five Kinkos in Austin TexasFive Kinkos in Austin Texas The OpenGL Graphics System: A Specification (Version 1.1) Mark Segal Kurt Akeley Editor: Chris Frazier Copyright © 1992-1997 Silicon Graphics, Inc. This document contains unpublished information of Silicon Graphics, Inc.
    109. 109. 109 Extension factsExtension facts • 442 Vendor and “EXT” extension specifications • Vendor: specific to a single vendor • EXT: shared by two or more vendors • 56 “ARB” extensions • Standardized , likely to be in the next spec revision • Lots of text … Source: OpenGL extension registry, December 2008
    110. 110. 110 ““Specification” sizesSpecification” sizes Lines Words Chars 56 ARB Extensions 48,674 263,908 2,221,347 All 442 Extensions 209,426 1,076,008 9,079,063 King James Bible 114,535 823,647 5,214,085 New Testament 27,319 188,430 1,197,812 Old Testament 86,783 632,515 3,998,303
    111. 111. 111 Beyond the specificationBeyond the specification • The ARB (now replaced with Khronos) • Rules of order, secretary, IP, … • The extension process • Categories, token syntax, spec templates, enums, registry, … • Licensing • Conformance • …
    112. 112. 112 SummarySummary • Many mistakes made (see other presentations for lists) • Created a sustainable culture that values quality and rigorous documentation • Defined and evolved the architecture for interactive 3-D computer graphics
    113. 113. 113 Writing better OpenGL Mark Kilgard Principal System Software Engineer NVIDIA
    114. 114. 114 Motivation • Complex APIs and systems have pitfalls • After 17 years of designed evolution, OpenGL certainly has its share • Normal documentation focus: • What can you do? • Rather than: What should you do?
    115. 115. 115 Communicating Vertex Data • The way you learn OpenGL: • Immediate mode • glBegin, glColor3f, glVertex3f, glEnd • Straightforward—no ambiguity about vertex data is • All vertex components are function parameters • The problem—too function call intensive • And all vertex data must flow through CPU
    116. 116. 116 Example Scenario • An OpenGL application has to render a set of rectangles • Rectangle with its parameters • x, y, height, width, left color, right color, depth (x,y) depth order 0.0 1.0 left side color right side color height width
    117. 117. 117 Scene Representation • Each rectangle specified by following RectInfo structure: • Array of RectInfo structures describes “scene” • Simplistic scene for sake of teaching typedef struct { GLfloat x, y, width, height; GLfloat depth_order; GLfloat left_side_color[3]; // red, green, then blue GLfloat right_side_color[3]; // red, green, then blue } RectInfo;
    118. 118. 118 Example Scene and Rendering Result • Scene of 4 rectangles: RectInfo rect_list[4] = { { 10, 20, 180, 140, 0.5, { 1, 1, 1 }, { 1, 0, 1 } }, { 30, 40, 100, 60, 0.5, { 1, 0, 0 }, { 0, 0, 1 } }, { 140, 60, 100, 80, 0.5, { 0, 0, 1 }, { 0, 1, 0 } }, { 70, 120, 80, 60, 0.7, { 1, 1, 0 }, { 0, 1, 1 } }, }; • OpenGL-rendered result
    119. 119. 119 Immediate Mode Rectangle Rendering • Given sized RectInfo array, render vertices of quads 1st vertex 2nd vertex 3rd vertex 4th vertex void drawRectangles(int count, const RectInfo *list) { glBegin(GL_QUADS); for (int i=0; i<count; i++) { const RectInfo *r = &list[i]; glColor3fv(r->left_side_color); glVertex3f(r->x, r->y, r->depth_order); glColor3fv(r->right_side_color); glVertex3f(r->x+r->width, r->y, r->depth_order); // right_side_color “sticks” glVertex3f(r->x+r->width, r->y+r->height, r->depth_order); glColor3fv(r->left_side_color); glVertex3f(r->x, r->y+r->height, r->depth_order); } glEnd(); } For each rectangle
    120. 120. 120 Critique of Immediate Mode • Advantages • Straightforward to code and debug • Easy-to-understand conceptual model • Building stream of vertices with OpenGL commands • Avoids driver & application copies of vertex data • Flexible, allowing totally dynamic vertex generation • Disadvantages • Rendering continuously streams attributes through CPU • Pollutes CPU cache with vertex data • Function call intensive • Unable to saturate fast graphics hardware • CPUs just too slow • Contrast with vertex array approach…
    121. 121. 121 Vertex Array Approach • Step 1: Copy vertex attributes into vertex arrays • From: RectInfo array (CPU memory) • To: interleaved arrays of vertex attributes (CPU memory) • Step 2: To render • Configure OpenGL vertex array client state • Use glEnableClientState, glVertexPointer, glColorPointer • Render quads based on indices into vertex arrays • Use glDrawArrays
    122. 122. 122 Vertex Array Format • Interleave vertex attributes in color & position arrays color position float = 4 bytes vertex 0 vertex 1 red green blue x y z red green blue x y z color position 24 bytes per vertex
    123. 123. 123 Step 1: Copy Rectangle Attributes to Vertex Arrays void *initVarrayRectangles(int count, const RectInfo *list) { void *varray = (char*) malloc(sizeof(GLfloat)*6*4*count); GLfloat *p = varray; for (int i=0; i<count; i++, p+=24) { const RectInfo *r = &list[i]; // quad vertex #1 memcpy(&p[0], r->left_side_color, sizeof(GLfloat)*3); p[3] = r->x; p[4] = r->y; p[5] = r->depth_order; // quad vertex #2 memcpy(&p[6], r->right_side_color, sizeof(GLfloat)*3); p[9] = r->x+r->width; p[10] = r->y; p[11] = r->depth_order; // quad vertex #3 memcpy(&p[12], r->right_side_color, sizeof(GLfloat)*3); p[15] = r->x+r->width; p[16] = r->y+r->height; p[17] = r->depth_order; // quad vertex #4 memcpy(&p[18], r-> left_side_color, sizeof(GLfloat)*3); p[21] = r->x; p[22] = r->y+r->height; p[23] = r->depth_order; } return varray; }
    124. 124. 124 Step 2: Configure & Render from Vertex Arrays void drawVarrayRectangles(int count, const RectInfo *list) { char *varray = initVarrayRectangles(count, list); const GLfloat *p = (const GLfloat*) varray; const GLsizei stride = sizeof(GLfloat)*6;//3 RGB floats,3 XYZ floats glColorPointer(/*rgb*/3, GL_FLOAT, stride, p+0); glVertexPointer(/*xyz*/3, GL_FLOAT, stride, p+3); glEnableClientState(GL_COLOR_ARRAY); glEnableClientState(GL_VERTEX_ARRAY); glDrawArrays(GL_QUADS, /*firstIndex*/0, /*indexCount*/count*4); free(varray); }
    125. 125. 125 Critique of Simplistic Vertex Array Rendering • Advantages • Far fewer OpenGL commands issued • Disadvantages • Every render with drawVarrayRectangles calls initVarrayRectangles • Allocates, initializes, & frees vertex array memory every render • Improve by separating vertex array construction from rendering
    126. 126. 126 Initialize Once, Render Many Approach • This routine expects base pointer returned by initVarrayRectangles void drawInitializedVarrayRectangles(int count, const void *varray) { const GLfloat *p = (const GLfloat*) varray; const GLsizei stride = sizeof(GLfloat)*6; // 3 RGB floats, 3 XYZ floats glColorPointer(/*rgb*/3, GL_FLOAT, stride, p+0); glVertexPointer(/*xyz*/3, GL_FLOAT, stride, p+3); // Assume GL_COLOR_ARRAY and GL_VERTEX_ARRAY are already enabled! glDrawArrays(GL_QUADS, /*firstIndex*/0, /*indexCount*/count*4); }
    127. 127. 127 Client Memory Vertex Attribute Transfer GPU Processor command processor vertex puller hardware rendering pipeline CPU command queue CPU writes of command + vertex data GPU DMA transfer of command + vertex data application (client) memory vertex array vertex data travels through CPU memory reads CPU
    128. 128. 128 Vertex Buffer Object Vertex Attribute Pulling OpenGL (vertex) buffer object GPU command processor vertex puller hardware rendering pipeline CPU command queue CPU writes of command + vertex indices vertex array GPU DMA transfer of command data application (client) memory memory reads CPU GPU DMA transfer of vertex data—CPU never reads data
    129. 129. 129 Initializing Vertex Buffer Objects (VBOs) • Once using vertex arrays, easy to switch to VBOs • Make the vertex array as before • Then bind to buffer object and copy data to the buffer void initVarrayRectanglesInVBO(GLuint bufferName, int count, const RectInfo *list) { char *varray = initVarrayRectangles(count, list); const GLsizei stride = sizeof(GLfloat)*6; // 3 RGB floats, 3 XYZ floats const GLint numVertices = 4*count; const GLsizeiptr bufferSize = stride*numVertices; glBindBuffer(GL_ARRAY_BUFFER, bufferName); glBufferData(GL_ARRAY_BUFFER, bufferSize, varray, GL_STATIC_DRAW); free(varray); }
    130. 130. 130 Rendering from Vertex Buffer Objects • Once initialized, glBindBuffer to bind to buffer ahead of vertex array configuration • Send offsets instead of points void drawVarrayRectanglesFromVBO(GLuint bufferName, int count) { const char *base = NULL; const GLsizei stride = sizeof(GLfloat)*6; // 3 RGB floats, 3 XYZ floats glBindBuffer(GL_ARRAY_BUFFER, bufferName); glColorPointer(/*rgb*/3, GL_FLOAT, stride, base+0*sizeof(GLfloat)); glVertexPointer(/*xyz*/3, GL_FLOAT, stride, base+3*sizeof(GLfloat)); // Assume GL_COLOR_ARRAY and GL_VERTEX_ARRAY are already enabled! glDrawArrays(GL_QUADS, /*firstIndex*/0, /*indexCount*/count*4); }
    131. 131. 131 Understanding glBindBuffer • Buffer object bindings are frequent point of confusion for programmers • What does glBindBuffer do really? • Lots of buffer binding targets: • GL_ARRAY_BUFFER target—for vertex attribute arrays • Query with GL_ARRAY_BUFFER_BINDING • GL_ARRAY_ELEMENT_BUFFER target—for vertex indices, effectively topology • Query with GL_ELEMENT_ARRAY_BUFFER_BINDING • Each vertex array has its own buffer, query with • GL_VERTEX_ARRAY_BUFFER_BINDING • GL_COLOR_ARRAY_BUFFER_BINDING • GL_TEXCOORD_ARRAY_BUFFER_BINDING, etc.
    132. 132. 132 Bind and Query Buffer Targets Buffer Bind Tokens • GL_ARRAY_BUFFER • GL_ELEMENT_ARRAY_BUFFER Buffer Query Tokens • GL_ARRAY_BUFFER_BINDING • GL_ELEMENT_ARRAY_BUFFER_BINDING • GL_COLOR_ARRAY_BUFFER_BINDING • GL_VERTEX_ARRAY_BUFFER_BINDING • GL_FOGCOORD_ARRAY_BUFFER_BINDING • GL_TEXCOORD_ARRAY_BUFFER_BINDING • GL_VERTEX_ATTRIB_ARRRAY_BUFFER_BINDING Target tokens for glBindBuffer Query tokens to glGetIntegerv Query tokens to glGetVertexAttribiv
    133. 133. 133 Latched Vertex Array Buffer Bindings • Here’s the confusing part: glBindBuffer(GL_ARRAY_BUFFER, 34); glColorPointer(3, GL_FLOAT, color_stride, (void*)color_offset); • The glBindBuffer doesn’t change any vertex array binding • The GL_ARRAY_BUFFER_BINDING state that glBindBuffer sets does not itself affect rendering • It is the glColorPointer call that latches the array buffer binding to change the color array’s buffer binding! • Same with all vertex array buffer bindings
    134. 134. 134 Binding Buffer Zero is Special • By default, vertex arrays don’t access buffer objects • Instead client memory is accessed • This is because • The initial buffer binding for a context is zero • And zero is special • Zero means access client memory • You can always resume client memory vertex array access for a given array like this glBindBuffer(GL_ARRAY_BUFFER, 0); // use client memory glColorPointer(3, GL_FLOAT, color_stride, color_pointer); • Different treatment of the “pointer” parameter to vertex array specification commands • When the current array buffer binding is zero, the pointer value is a client memory pointer • When the current array buffer binding is non-zero (meaning it names a buffer object), the pointer value is “recast” as an offset from the beginning of the buffer • Once again • The glBindBuffer(GL_ARRAY_BUFFER,0) call alone doesn’t change any vertex array buffer bindings • It takes a vertex array specification command such as glColorPointer to latch the zero ensures compatibility with pre-VBO OpenGL
    135. 135. 135 Texture Coordinate Set Selector • A selector in OpenGL is • A state variable that controls what state a subsequent command updates • Examples of commands that modify selectors • glMatrixMode, glActiveTexture, glClientActiveTexture • A selector is different from latched state • Latched state is a specified value that is set (or “latched”) when a subsequent command is called • Pitfall warning: glTexCoordPointer both • Relies on the glClientActiveTexture command’s selector • And latches the current array buffer binding for the selected texture coordinate vertex array • Example glBindBuffer(GL_ARRAY_BUFFER, 34); glClientActiveTexture(GL_TEXTURE3); glTexCoordPointer(2, GL_FLOAT, uv_stride, (void*)buffer_offset); buffer value glTexCoordPointer latches selector glTexCoordPointer uses
    136. 136. 136 OpenGL’s Modern Buffer-centric Processing Model Vertex Array Buffer Object (VaBO) Transform Feedback Buffer (XBO) Parameter Buffer (PaBO) Pixel Unpack Buffer (PuBO) Pixel Pack Buffer (PpBO)Bindable Uniform Buffer (BUB) Texture Buffer Object (TexBO) Vertex Puller Vertex Shading Geometry Shading Fragment Shading Texturing Array Element Buffer Object (VeBO) Pixel Pipeline vertex data texel data pixel data parameter data (not ARB functionality yet) glBegin, glDrawElements, etc. glDrawPixels, glTexImage2D, etc. glReadPixels, etc. Framebuffer
    137. 137. 137 Usages of OpenGL Buffers Objects • Vertex uses (VBOs) • Input to GL: Vertex attribute buffer objects • Color, position, texture coordinate sets, etc. • Input to GL: Vertex element buffer objects • Indices • Output from GL: Transform feedback • Streaming vertex attributes out • Texture uses (TexBOs) • Texturing from: Texture buffer objects • Pixel uses (PBOs) • Output from GL: Pixel pack buffer objects • glReadPixels • Input from GL: Pixel unpack buffer objects • glDrawPixels, glBitmap, glTexImage2D, etc. • Shader uses (PaBOs, UBOs) • Input to assembly program: Parameter buffer objects • Input to GLSL program: Bind-able uniform buffer objects Key point: OpenGL buffers are containers for bytes; a buffer is not tied to any particular usage
    138. 138. 138 Continuum of OpenGL Usage Tweak-able Performance Immediate mode Client vertex arrays Vertex buffer objects (VBOs) Display lists
    139. 139. 139 Mid-session break 15 minutes
    140. 140. 140 Implementing OpenGL Mark Kilgard Principal System Software Engineer NVIDIA
    141. 141. 141 Topics in OpenGL Implementation • Dual-core OpenGL driver operation • What goes into a texture fetch? • You give me some texture coordinates • I give you back a color • Could it be any simpler?
    142. 142. 142 OpenGL Drivers for Multi-core CPUs • Today dual-core processors in PCs is nearly ubiquitous • 4, 6, 8, and more cores are clearly coming • How does OpenGL implementation exploit this trend? • Answer: develop dual-core OpenGL driver
    143. 143. 143 Dual-core OpenGL Driver Architecture Application thread … Application thread D Context 1 Application thread A Application rendering thread App ICD ICD’s app thread (tokenize thread) Worker thread 1 (server thread) Application thread C Application audio thread (no OpenGL) Context 2 Application thread B Application rendering thread ICD’s app thread (tokenize thread) Worker thread 2 (server thread) Circular command FIFO Circular command FIFO
    144. 144. 144 Dual-core Performance Results • A well-behaved OpenGL application benefiting from a dual-core mode of OpenGL driver operations 0 50 100 150 200 250 Single core Dual core Null driver Frames per second Mode of OpenGL driver operation
    145. 145. 145 Good Dual-core Driver Practices • General advice • Display lists execute on the driver’s worker thread! • You want to avoid situations where the application thread must “sync” with the driver thread • Specific advice • Avoid OpenGL state queries • More on this later • Avoid querying OpenGL errors in production code • Bad behavior is detected automatically and leads to exit from the dual-core mode • Back to the standard single-core driver mode of operation • “Do no harm”
    146. 146. 146 Consider an OpenGL texture fetch • Seems very simple • Input: texture coordinates (s,t,r,q) • Output: some color (r,g,b,a) • Just a simple function, written in Cg/HLSL: uniform sampler2D decal : TEXUNIT2; float4 texcoord : TEXCOORD3; float4 rgba = tex2D(decal, texcoordset.st); • Compiles to single instruction: TEX o[COLR], f[TEX3], TEX2, 2D; • Implementation is much more involved!
    147. 147. 147 Anatomy of a Texture Fetch Filtered texel vector Texel Selection Texel Combination Texel offsets Texel data Texture images Combination parameters Texture coordinate vector Texture parameters
    148. 148. 148 Texture Fetch Functionality (1) • Texture coordinate processing • Projective texturing (OpenGL 1.0) • Cube map face selection (OpenGL 1.3) • Texture array indexing (OpenGL 2.1) • Coordinate scale: normalization (ARB_texture_rectangle) • Level-of-detail (LOD) computation • Log of maximum texture coordinate partial derivative (OpenGL 1.0) • LOD clamping (OpenGL 1.2) • LOD bias (OpenGL 1.3) • Anisotropic scaling of partial derivatives (SGIX_texture_lod_bias) • Wrap modes • Repeat, clamp (OpenGL 1.0) • Clamp to edge (OpenGL 1.2), Clamp to border (OpenGL 1.3) • Mirrored repeat (OpenGL 1.4) • Fully generalized clamped mirror repeat (EXT_texture_mirror_clamp) • Wrap to adjacent cube map face • Region clamp & mirror (PlayStation 2)
    149. 149. 149 Texture Fetch Functionality (2) • Filter modes • Minification / magnification transition (OpenGL 1.0) • Nearest, linear, mipmap (OpenGL 1.0) • 1D & 2D (OpenGL 1.0), 3D (OpenGL 1.2), 4D (SGIS_texture4D) • Anisotropic (EXT_texture_filter_anisotropic) • Fixed-weights: Quincunx, 3x3 Gaussian • Used for multi-sample resolves • Detail texture magnification (SGIS_detail_texture) • Sharpen texture magnification (SGIS_sharpen_texture) • 4x4 filter (SGIS_texture_filter4) • Sharp-edge texture magnification (E&S Harmony) • Floating-point texture filtering (ARB_texture_float, OpenGL 3.0)
    150. 150. 150 Texture Fetch Functionality (3) • Texture formats • Uncompressed • Packing: RGBA8, RGB5A1, etc. (OpenGL 1.1) • Type: unsigned, signed (NV_texture_shader) • Normalized: fixed-point vs. integer (OpenGL 3.0) • Compressed • DXT compression formats (EXT_texture_compression_s3tc) • 4:2:2 video compression (various extensions) • 1- and 2-component compression (EXT_texture_compression_latc, OpenGL 3.0) • Other approaches: IDCT, VQ, differential encoding, normal maps, separable decompositions • Alternate encodings • RGB9 with 5-bit shared exponent (EXT_texture_shared_exponent) • Spherical harmonics • Sum of product decompositions
    151. 151. 151 Texture Fetch Functionality (4) • Pre-filtering operations • Gamma correction (OpenGL 2.1) • Table: sRGB / arbitrary • Shadow map comparison (OpenGL 1.4) • Compare functions: LEQUAL, GREATER, etc. (OpenGL 1.5) • Needs “R” depth value per texel • Palette lookup (EXT_paletted_texture) • Thresh-holding • Color key • Generalized thresh-holding
    152. 152. 152 Texture Fetch Functionality (5) • Optimizations • Level-of-detail weighting adjustments • Mid-maps (extra pre-filtered levels in-between existing levels) • Unconventional uses • Bitmap textures for fonts with large filters (Direct3D 10) • Rip-mapping • Non-uniform texture border color • Clip-mapping (SGIX_clipmap) • Multi-texel borders • Silhouette maps (Pardeep Sen’s work) • Shadow mapping • Sharp piecewise linear magnification
    153. 153. 153 Phased Data Flow • Must hide long memory read latency between Selection and Combination phases Texel Selection Texel Combination Texel offsets Texel data Texture images Combination parameters Texture coordinate vector Texture parameters Memory reads for samples FIFOing of combination parameters
    154. 154. 154 What really happens? • Let’s consider a simple tri-linear mip-mapped 2D projective texture fetch • Logically just one instruction TXP o[COLR], f[TEX3], TEX2, 2D; • Logically • Texel selection • Texel combination • How many operations are involved?
    155. 155. 155 Medium-Level Dissection of a Texture Fetch Convert texel coords to texel offsets integer / fixed-point texel combination texel offsets texel data texture images combination parameters interpolated texture coords vector texture parameters Convert texture coords to texel coords filtered texel vector texel coords floor / frac integer coords & fractional weights floating-point scaling and combination integer / fixed-point texel intermediates
    156. 156. 156 Interpolation • First we need to interpolate (s,t,r,q) • This is the f[TEX3] part of the TXP instruction • Projective texturing means we want (s/q, t/q) • And possible r/q if shadow mapping • In order to correct for perspective, hardware actually interpolates • (s/w, t/w, r/w, q/w) • If not projective texturing, could linearly interpolate inverse w (or 1/w) • Then compute its reciprocal to get w • Since 1/(1/w) equals w • Then multiply (s/w,t/w,r/w,q/w) times w • To get (s,t,r,q) • If projective texturing, we can instead • Compute reciprocal of q/w to get w/q • Then multiple (s/w,t/w,r/w) by w/q to get (s/q, t/q, r/q) Observe projective texturing is same cost as perspective correction
    157. 157. 157 Interpolation Operations • Ax + By + C per scalar linear interpolation • 2 MADs • One reciprocal to invert q/w for projective texturing • Or one reciprocal to invert 1/w for perspective texturing • Then 1 MUL per component for s/w * w/q • Or s/w * w • For (s,t) means • 4 MADs, 2 MULs, & 1 RCP • (s,t,r) requires 6 MADs, 3 MULs, & 1 RCP • All floating-point operations
    158. 158. 158 Texture Space Mapping • Have interpolated & projected coordinates • Now need to determine what texels to fetch • Multiple (s,t) by (width,height) of texture base level • Could convert (s,t) to fixed-point first • Or do math in floating-point • Say based texture is 256x256 so • So compute (s*256, t*256)=(u,v)
    159. 159. 159 Mipmap Level-of-detail Selection • Tri-linear mip-mapping means compute appropriate mipmap level • Hardware rasterizes in 2x2 pixel entities • Typically called quad-pixels or just quad • Finite difference with neighbors to get change in u and v with respect to window space • Approximation to ∂u/∂x, ∂u/∂y, ∂v/∂x, ∂v/∂y • Means 4 subtractions per quad (1 per pixel) • Now compute approximation to gradient length • p = max(sqrt((∂u/∂x)2 +(∂u/∂y)2 ), sqrt((∂v/∂x)2 +(∂v/∂y)2 )) one-pixel separation
    160. 160. 160 Level-of-detail Bias and Clamping • Convert p length to power-of-two level-of-detail and apply LOD bias • λ = log2(p) + lodBias • Now clamp λ to valid LOD range • λ’ = max(minLOD, min(maxLOD, λ))
    161. 161. 161 Determine Mipmap Levels and Level Filtering Weight • Determine lower and upper mipmap levels • b = floor(λ’)) is bottom mipmap level • t = floor(λ’+1) is top mipmap level • Determine filter weight between levels • w = frac(λ’) is filter weight
    162. 162. 162 Determine Texture Sample Point • Get (u,v) for selected top and bottom mipmap levels • Consider a level l which could be either level t or b • With (u,v) locations (ul,vl) • Perform GL_CLAMP_TO_EDGE wrap modes • uw = max(1/2*widthOfLevel(l), min(1-1/2*widthOfLevel(l), u)) • vw = max(1/2*heightOfLevel(l), min(1-1/2*heightOfLevel(l), v)) • Get integer location (i,j) within each level • (i,j) = ( floor(uw* widthOfLevel(l)), floor(vw* ) ) border edge s t
    163. 163. 163 Determine Texel Locations • Bilinear sample needs 4 texel locations • (i0,j0), (i0,j1), (i1,j0), (i1,j1) • With integer texel coordinates • i0 = floor(i-1/2) • i1 = floor(i+1/2) • j0 = floor(j-1/2) • j1 = floor(j+1/2) • Also compute fractional weights for bilinear filtering • a = frac(i-1/2) • b = frac(j-1/2)
    164. 164. 164 Determine Texel Addresses • Assuming a texture level image’s base pointer, compute a texel address of each texel to fetch • Assume bytesPerTexel = 4 bytes for RGBA8 texture • Example • addr00 = baseOfLevel(l) + bytesPerTexel*(i0+j0*widthOfLevel(l)) • addr01 = baseOfLevel(l) + bytesPerTexel*(i0+j1*widthOfLevel(l)) • addr10 = baseOfLevel(l) + bytesPerTexel*(i1+j0*widthOfLevel(l)) • addr11 = baseOfLevel(l) + bytesPerTexel*(i1+j1*widthOfLevel(l)) • More complicated address schemes are needed for good texture locality!
    165. 165. 165 Initiate Texture Reads • Initiate texture memory reads at the 8 texel addresses • addr00, addr01, addr10, addr11 for the upper level • addr00, addr01, addr10, addr11 for the lower level • Queue the weights a, b, and w • Latency FIFO in hardware makes these weights available when texture reads complete
    166. 166. 166 Phased Data Flow • Must hide long memory read latency between Selection and Combination phases Texel Selection Texel Combination Texel offsets Texel data Texture images Combination parameters Texture coordinate vector Texture parameters Memory reads for samples FIFOing of combination parameters
    167. 167. 167 Texel Combination • When texels reads are returned, begin filtering • Assume results are • Top texels: t00, t01, t10, t11 • Bottom texels: b00, b01, b10, b11 • Per-component filtering math is tri-linear filter • RGBA8 is four components • result = (1-a)*(1-b)*(1-w)*b00 + (1-a)*b*(1-w)*b*b01 + a*(1-b)*(1-w)*b10 + a*b*(1-w)*b11 + (1-a)*(1-b)*w*t00 + (1-a)*b*w*t01 + a*(1-b)*w*t10 + a*b*w*t11; • 24 MADs per component, or 96 for RGBA • Lerp-tree could do 14 MADs per component, or 56 for RGBA
    168. 168. 168 Total Texture Fetch Operations • Interpolation • 6 MADs, 3 MULs, & 1 RCP (floating-point) • Texel selection • Texture space mapping • 2 MULs (fixed-point) • LOD determination (floating-point) • 1 pixel difference, 2 SQRTs, 4 MULs, 1 LOG2 • LOD bias and clamping (fixed-point) • 1 ADD, 1 MIN, 1 MAX • Level determination and level weighting (fixed-point) • 1 FLOOR, 1 ADD, 1 FRAC • Texture sample point • 4 MAXs, 4 MINs, 2 FLOORs (fixed-point) • Texel locations and bi-linear weights • 8 FLOORs, 4 FRACs, 8 ADDs (fixed-point) • Addressing • 16 integer MADs (integer) • Texel combination • 56 fixed-point MADs (fixed-point)
    169. 169. 169 Observations about the Texture Fetch • Lots of ways to implement the math • Lots of clever ways to be efficient • Lots more texture operations not considered in this analysis • Compression • Anisotropic filtering • sRGB • Shadow mapping • Arguably TEX instructions are “world’s most CISC instructions” • Texture fetches are incredibly complex instructions • Good deal of GPU’s superiority at graphics operations over CPUs is attributable to TEX instruction efficiency • Good for compute too
    170. 170. 170 OpenGL’s Future Evolution Mark Kilgard Principal System Software Engineer NVIDIA
    171. 171. 171 What drives OpenGL’s future? • GPU graphics functionality • Tessellation & geometry amplification • Ratio of GPU to single-core CPU performance • Compatibility • Direct3Disms • OpenGLisms • Deprecation • Compute support • OpenCL, CUDA, Stream processing • Unconventional graphics devices
    172. 172. 172 Better Graphics Functionality • Expect more graphics performance • Easy prediction • Rasterization nowhere near peaked • Ray tracing fans—GPUs make rays and triangles faster – Market still values triangles more than rays • Expect more generalized graphics functionality • Trend for texture enhancements likely to continue
    173. 173. 173 Geometry Amplification • Tessellation • Programmable hardware support coming • True market demand probably not tessellation per se • Games want visual richness • Texture and shading have created much richness – Often “pixel richness” as substitute for geometry richness • Increasingly “visual richness” means geometric complexity • Geometry Amplification may be better term • Tessellation is one way to improve tessellation – Recognize the limits of bi-variate patches for representing geometry
    174. 174. 174 Programmable Tessellation • Stunning real-time geometric detail + animation possible • Programmable tessellation + vertex textured displacements
    175. 175. 175 Continuous Level-of-detail for Tessellation Increasing tessellation level-of-detail • Same patch mesh for all 3 scenes
    176. 176. 176 Adaptive Programmable Tessellation Programmable level-of-detail determination allows more tessellation along silhouette edges
    177. 177. 177 Limits of Patch Tessellation • What games tend to want • Here’s 8 vertices (bounding box), go draw a fire truck • Here’s a few vertices, go draw a tree
    178. 178. 178 Tessellation Not New to OpenGL • At least three different bi-variate patch tessellation schemes have been added to OpenGL • Evaluators (OpenGL 1.0) • NV_evaluators (GeForce 3) • water-tight • adaptive level-of-detail • forward differencing approach • ATI_pn_triangles Curved PN Triangles (Radeon) • tessellated triangle based on positions+normals • None succeeded • Hard to integrate into art pipelines • Didn’t offer enough performance advantage GLUT’s wire-frame teapot [Moreton 20001] [Vlachos 20001]
    179. 179. 179 Ratio of CPU core-to-GPU Performance • Well known computer architecture trends now • Single-threaded CPU performance trends are stalled • Multi-core is CPU designer response • GPU performance continues on-trend • What does this mean for graphics API design? • CPUs must generate more visually rich API command streams to saturate GPUs • Can’t just send more commands faster • Single-threaded CPUs can only do so much • So must send more powerful commands
    180. 180. 180 Déjà vu • We’ve been here before • Early 1980s: Graphics terminals used to be connected to minicomputers by slow speed interconnects • CPUs themselves far too slow for real-time rendering • Resulting rendering model • Download scene database to graphics terminal • Adjust viewing and modeling parameters • Send “redraw scene” command
    181. 181. 181 What Happened • Such “scene processor” hardware not very flexible • Difficult to animate anything beyond rigid dynamics • Eventually SGI and others matched CPUs and interconnects to graphics performance • Result was IRIS GL’s immediate mode • CPU fast enough to send geometry every frame • OpenGL took this model • Over time added vertex arrays, vertex buffers, texturing, programmable shading, and more performance • CPU performance became limiter still • Better graphics driver tuning helped • Dual-core drivers help some more
    182. 182. 182 OpenGL’s Most Powerful Command • Available since OpenGL 1.0 • Can render essentially anything OpenGL can render! • Takes just one parameter • The command glCallList(GLuint displayListName); • Power of display lists comes from • Playing back arbitrary compiled commands • Allowing for hierarchical calling of display list • A display list can contain glCallList or glCallLists • Ability of application to re-define display lists • No editing, but can be re-defined
    183. 183. 183 Enhanced Display Lists • OpenGL 1.0 display lists are too inflexible • Pixel & vertex data “compiled into” display lists • Binding objects always “by name” • Rather than “by reference • These problems can be fixed • Modern OpenGL supports buffers for transferring vertices and pixels • Compile commands into display lists that defer vertex and pixel transfers until execute-time – Rather than compile-time • Allow objects (textures, buffers, programs) to be bound “by reference” or “by name”
    184. 184. 184 Other Display List Enhancements • Conditional display list execution • Relaxed vertex index and command order • Parallel construction of display lists by multiple threads General insight: Easier for driver to optimize application’s graphics command stream if it gets to 1) see the repetition in the command stream clearly 2) take time to analyze and optimize usage
    185. 185. 185 Conditional Display List Execution • Today’s occlusion query • Application must “query” to learn occlusion result • Latency too great to respond • Application can use OpenGL 3.0’s conditional render capability • But just skips vertex pulling, not state changes • Conditional display list execution • Allow a glCallList to depend on the occlusion result from an occlusion query object • Allows in-band occlusion querying • Skip both vertex pulling and state changes
    186. 186. 186 Relaxed Vertex Index and Command Order • OpenGL today always executes commands “in order” • Sequentially requirement • Provide compile-time specification of re-ordering allowances • Allows GL implementation to re-order • Vertex indices within display list’s vertex batch • Commands within display list • Key rule: state vector rendering command executes in must match the state if command was rendered sequentially • Allow static or dynamic re-ordering • Static re-ordering needed for multi-pass invariances • Past practice • IRIS Performer would sort rendering by state changes for performance • [Sander 2007] show substantial benefit for vertex ordering
    187. 187. 187 Parallel Display List Construction • Today’s model • Single thread makes all OpenGL rendering calls • Minimizes GPU context switch overhead • Ties command generation rate to single core’s CPU performance • Enhanced display list model • Multiple threads can build display lists in parallel • Single thread still executes display lists • Countable semaphore objects used to synchronize hand-off of display lists built by other threads with main rendering thread
    188. 188. 188 Rethinking Display Lists • Display lists have been proposed for deprecation • Right as we really need them! • Much more interesting to enhance display lists • Dual-core driver already off-loads display list traversal to driver’s thread • Multi-core driver could scan frequently executed display lists to optimize their order and error processing • Includes adding pre-fetching to avoid stalling CPU on cache misses for object accesses
    189. 189. 189 Direct3Disms • Developing a shader-rich game title costs $$$ • For top titles, often US$ 5,000,000+ • Investment typically amortized over multiple platforms • Consoles are primary target, then PCs • PC version typically developed for Direct3D • Reality: OpenGL is often 3rd or worse priority • API differences = porting & performance pitfalls • Stops or slows Direct3D-developed 3D content from working easily on OpenGL platforms
    190. 190. 190 Supporting Direct3D: Not New • OpenGL has always supported multiple formats well • OpenGL’s plethora of pixel and vertex formats • Very first OpenGL extension: EXT_bgra • Provides a pixel component ordering to match the color component ordering of Windows for 2D GDI rendering • Made core functionality by OpenGL 1.3 • Many OpenGL extensions have embraced Direct3Disms • Secondary color • Fog coordinate • Point sprites
    191. 191. 191 Direct3D vs. OpenGL Coordinate System Conventions • Window origin conventions • Direct3D = upper-left origin • OpenGL = lower-left origin • Pixel center conventions • Direct3D9 = pixel centers at integer locations • OpenGL (and Direct3D 10) = pixel centers at half-pixel locations • Clip space conventions • Direct3D = [-1,+1] for XY, [0,1] for Z • OpenGL = [-1,+1] range for XYZ • Affects • How projection matrix is loaded • Fragment shaders that access the window position • Point sprites have upper-left texture coordinate origin • OpenGL already lets application choose lower-left or upper-left
    192. 192. 192 Direct3D vs. OpenGL Provoking Vertex Conventions • Direct3D uses “first” vertex of a triangle or line to determine which color is used for flat shading • OpenGL uses “last” vertex for lines, triangles, and quads • Except for polygons (GL_POLYGON) mode that use the first vertex Direct3D 9 pDev->SetRenderState( D3DRS_SHADEMODE, D3DSHADE_FLAT); OpenGL glShadeModel(GL_FLAT); Input triangle strip with per-vertex colors
    193. 193. 193 BGRA Vertex Array Order • Direct3D 9’s most common usage for sending per-vertex colors is 32-bit D3DCOLOR data type: • Red in bits 16:23 • Green in bits 8:15 • Blue in bits 0:7 • Alpha in bits 24:31 • Laid in memory, looks like BGRA order • OpenGL assumes RGBA order for all vertex arrays • Direct3Dism EXT_vertex_array_bgra extension allows: glColorPointer(GL_BGRA, GL_UNSIGNED_BYTE, stride, pointer); glSecondaryColorPointer(GL_BGRA, GL_UNSIGNED_BYTE, stride, pointer); glVertexAttribPointer(GL_BGRA, GL_UNSIGNED_BYTE, stride, pointer); 8-bit red 8-bit alpha 8-bit green 8-bit blue bit 31 bit 0
    194. 194. 194 OpenGLisms • Things about OpenGL’s operation that make it hard for non-OpenGL applications to port to OpenGL • Examples • Selectors • Linked GLSL program objects
    195. 195. 195 Eliminating Selectors from OpenGL • OpenGL has lots of selectors • Selectors set state that indicates what state subsequent commands will update • Already mentioned selectors: glClientActiveTexture • Other examples: glActiveTexture, glMatrixMode, glBindTexture, glBindBuffer, glUseProgram, glBindProgramARB • OpenGL is full of selectors – Partly OpenGL’s extensibility strategy – Partly because objects are bound into context » Bind-to-edit objects » Rather than edit-by-name • Direct State Access extension: EXT_direct_state_access • Provides complete selector-free additional API for OpenGL • Shipping in NVIDIA’s 180.43 drivers
    196. 196. 196 Reasons to Eliminate Selectors • Direct3D has an “edit-by-name” model of operation • Means Direct3D has no selectors • Having to manage selectors when porting Direct3D or console code to OpenGL is awkward • Requires deferring updates to minimize selector and object bind changes • Layered libraries can’t count of selector state • To be safe when updating sate controlled by selectors, such libraries must use idiom • Save selector, Set selector, Update state, Restore selector • Bad for performance, particularly bad for dual-core drivers since queries are expensive
    197. 197. 197 GLSL Program Object Linking • GLSL requires shader objects from different domains (vertex, geometry, fragment) to be linked into single GLSL program object • Means you can’t mix-and-match shaders easily • Other APIs don’t have this limitation • Direct3D • Prior OpenGL assembly language extensions • Consoles • Have a “separate shader objects” extension could fix this problem
    198. 198. 198 Separate Shader Objects Example • Combining different GLSL shaders at once Specular brick bump mapping Red diffuse Wobbly torus Smooth torus Different GLSL vertex shaders Different GLSL fragment shaders
    199. 199. 199 Deprecation • Part of OpenGL 3.0 is a marking of features for deprecation • LOTS of functionality is marked for deprecation • I contend no real application today uses the non-deprecated subset of OpenGL—all apps would have to change due to deprecation • Some vendors believe getting rid of features will make OpenGL better in some way • NVIDIA does not believe in abandoning API compatibility this way • OpenGL is part of a large ecosystem so removing features this way undermines the substantial investment partners have made in OpenGL over years • API compatibility and stability is one of OpenGL’s great strengths
    200. 200. 200 Synergy between OpenGL and OpenCL • Complimentary capabilities • OpenGL 3.0 = state-of-the-art, cross-platform graphics • OpenCL 1.0 = state-of-the-art, cross-platform compute • Computation & Graphics should work together • Most natural way to intuit compute results is with graphics • When Compute is done on a GPU, there’s no need to “copy” the data to see it visualized • Appendix B of OpenCL specification • Details with sharing objects between OpenGL and OpenCL • Called “GL” and “CL” from here on…
    201. 201. 201 Four Kinds of Shared Objects OpenCL 3D image object cl_mem OpenGL renderbuffer object GLuint renderbuffer OpenGL buffer object GLuint bufferobj OpenCL buffer object cl_mem OpenGL texture 2D object GLenum target GLuint texture GLint miplevel OpenGL texture 3D object GLenum target GLuint texture GLint OpenCL 2D image object cl_mem 2D image object cl_mem clCreateFromGLBuffer clCreateFromGLTexture2D clCreateFromGLTexture3D clCreateFromGLRenderbuffer OpenGL OpenCL
    202. 202. 202 OpenGL / OpenCL Sharing • Requirements for GL object sharing with CL • CL context must be created with an OpenGL context • Each platform-specific API will provide its appropriate way to create an OpenGL-compatible CL context • For WGL (Windows), CGL (OS X), GLX (X11/Linux), EGL (OpenGL ES), etc. • Creating cl_mem for GL Objects does two things 1.Ensures CL has a reference to the GL objects 2.Provides cl_mem handle to acquire GL object for CL’s use • clRetainMemObject & clReleaseMemObject can create counted references to cl_mem objects
    203. 203. 203 Acquiring GL Objects for Compute Access • Still must “enqueue acquire” GL objects for compute kernels to use them • Otherwise reading or writing GL objects with CL is undefined • Enqueue acquire and release provide sequential consistency with GL command processing • Enqueue commands for GL objects • clEnqueueAcquireGLObjects • Takes list of cl_mem objects for GL objects & list of cl_events that must complete before acquire • Returns a cl_event for this acquire operation • clEnqueueReleaseGLObjects • Takes list of cl_mem objects for GL objects & list of cl_events that must complete before release • Returns a cl_event for this release operation
    204. 204. 204 Unconventional OpenGL Deployments • Workstation PCs—Quadro • Consumer PCs—GeForce • High-end Visualization—QuadroPlex Visual Computing Solution (VCS) • Embedded Applications • Handheld Devices • Game Consoles Conventional PC OpenGL Products Unconventional
    205. 205. 205 OpenGL in Context A facilitated conversation with Dr. Marc Levoy, Stanford University
    206. 206. 206 Questions?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×