Advanced Game Development with the Mobile 3D Graphics API

Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center

Gain an insight to the Mobile 3D Graphics API and take your games to the next level

Prerequisites
Fundamentals of 3D graphics
OpenGL or some other modern 3D API
Java programming, preferably MIDP

Objectives
Get a quick overview of the whole API
Gain a deeper understanding of selected topics
Learn practical tricks not found in the spec

Agenda
Why should I care?
Getting started
Low level features
Bitmaps and texturing
Scene graph
Dynamic meshes
Animation

Mobile 3D Graphics API (M3G)
Also known as JSR-184
Designed for mobile devices
Primarily CLDC / MIDP
But also CDC
Mobile 3D Graphics API Java Virtual Machine (CLDC 1.1) MIDP Midlets Graphics Hardware

Overcome the performance barrier Benchmarked on an ARM9 processor Native (C/C++) vs. Java on mobiles

Leverage hardware acceleration
Benefit for developers
Mask hardware differences (3D HW, FPU, DSP,…)
Exploit any new hardware automatically
Benefit for hardware vendors
Gives a concrete target which features to offer

Speed up development cycles
Separate content from code
Quick prototyping without programming
Artists create looks and behavior
Export all in a binary file format
Load the file, populate scene graph
Commonly needed functionality built in
High level functionality such as scene graph, keyframe animation

Why a new standard?
OpenGL (ES) (and D3D) are too low-level
Lots of Java code needed for simple things
Java 3D is too bloated
A hundred times larger than M3G
Does not fit together with MIDP
Tried and failed, but…
Now knew what we wanted!
Basic Java 3D ideas: nodes, scene graph
Add file format, keyframe animation
Remain compatible with OpenGL ES

Agenda
Why should I care?
Getting started
Low level features
Scene graph
Dynamic meshes
Animation

Key classes World Graphics3D Loader 3D graphics context Performs all rendering Scene graph root node Can load individual objects and entire scene graphs (M3G and PNG files)

Graphics3D
Contains global state
Target surface, viewport, depth buffer
Camera, light sources
Rendering quality hints
Each renderable object has its own local state
Geometry and appearance (material, textures, etc.)
Transformation relative to parent or world

Graphics3D: Rendering modes
Retained mode
Render a scene graph, rooted by the World
Take the Camera and Lights from the World
Immediate mode
Render a branch or an individual node at a time
Explicitly give the Camera and Lights to Graphics3D

Graphics3D: How-To-Use
Bind a target to it, render, release the target
Tip: Never mess with a bound target
Tip: Graphics3D is a singleton (threads!)
void paint(Graphics g) { myGraphics3D.bindTarget(g); myGraphics3D.render(world); myGraphics3D.releaseTarget(); }

Graphics3D: Rendering targets Graphics Canvas Image CustomItem Graphics3D Image2D World Can render to textures also M3G MIDP

Everything is synchronous
A method returns only when it’s done
No separate thread for renderer or loader
There are no callbacks
No abstract methods, interfaces, or events
Tip: Do not even try to override any methods
Scene update is decoupled from rendering
animate : updates the scene to the given time
align : applies auto-alignments, e.g., billboards
render : draws the scene, no side-effects
Things to keep in mind

“Hello, World” import javax.microedition.midlet.MIDlet; import javax.microedition.lcdui.Display; import javax.microedition.lcdui.game.GameCanvas; import javax.microedition.m3g.*; public class Player extends MIDlet { public void pauseApp() {} public void destroyApp(boolean b) {} public void startApp() { PlayerCanvas player = new PlayerCanvas(true); Display.getDisplay(this).setCurrent(player); try { player.run(); } catch (Exception e) {} notifyDestroyed(); } } A simplified animation player

“ Hello, World” class PlayerCanvas extends GameCanvas { PlayerCanvas(boolean suppress){super(suppress);} public void run() throws Exception { Graphics3D g3d = Graphics3D.getInstance(); World w = (World) Loader.load("/file.m3g")[0]; long start, elapsed, time = 0; while (getKeyStates() == 0) { start = System.currentTimeMillis(); g3d.bindTarget(getGraphics()); try { w.animate(time); g3d.render(w); } finally { g3d.releaseTarget(); } flushGraphics(); elapsed = System.currentTimeMillis()-start; time += (elapsed < 100) ? 100 : (int)elapsed; if (elapsed < 100) Thread.sleep(100-elapsed); } } }

Demo
Animation playback

Agenda
Why should I care?
Getting started
Low level features
Scene graph
Dynamic meshes
Animation

Renderable Objects Mesh Sprite3D Made of triangle strips Base class for meshes (three types exist) 2D image placed in 3D space Good for labels, etc.

Mesh
Common buffer of vertex data
Submesh has triangle strip indices to vertices
One Appearance for each submesh
Mesh VertexBuffer coordinates normals colors texcoords Composed of submeshes IndexBuffer Appearance

Appearance components CompositingMode Material colors for lighting Can track per-vertex colors PolygonMode Fog Texture2D Material Blending, depth buffering Alpha testing, masking Winding, culling, shading Perspective correction hint Fades colors based on distance Linear and exponential mode Texture matrix, blending, filtering Multitexturing: One Texture2D for each unit

Sprite3D
Scaled mode for billboards, trees, etc.
Unscaled mode for text labels, icons, etc.
Image2D 2D image with a position in 3D space Sprite3D Appearance Image2D CompositingMode Fog

Rendering tricks
Use layers to impose rendering order
Appearance contains a layer index (integer)
Defines a global ordering for submeshes & sprites
Tip: Disable z-buffering for sky boxes, use layers
Tip: Maximize triangle strip length
Even if it requires adding degenerate triangles
Better benefits from vertex caching

Example: Rotating cube // Corners of a cube as (X,Y,Z) triplets static short[] cubeVertices = { -1, -1, 1, 1, -1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, -1, -1, -1, 1, -1, 1, 1, -1 }; // A color for each corner as an (R,G,B) triplet static byte[] cubeColors = { (byte) 255, (byte) 0, (byte) 0, // red … (byte) 0, (byte) 255, (byte) 0, // green }; // Define the cube as a single triangle strip static int[] indices = { 6,7,4,5,1,7,3,6,2,4,0,1,2,3 }; static int[] stripLen = { 14 }; Define raw data for a cube

Example: Rotating cube // Fill in VertexArrays from short[] and byte[] int numVertices = vertices.length/3; VertexArray pos = new VertexArray(numVertices, 3, 2); VertexArray col = new VertexArray(numVertices, 3, 1); pos.set(0, numVertices, cubeVertices); col.set(0, numVertices, cubeColors); // Attach the VertexArrays to a VertexBuffer // Note the scale (1.0) and bias (0,0,0) vertices = new VertexBuffer(); vertices.setPositions(pos, 1.0f, null); vertices.setColors(col); // Fill in the triangle strip triangles = new TriangleStripArray(cubeIndices, stripLen); // Create a Mesh with default Appearance cube = new Mesh(vertices, triangles, new Appearance()); Copy raw data into Objects

Example: Rotating cube Camera cam = new Camera(); // 60-degree field of view, screen aspect ratio // Near clipping plane at 1.0, far plane at 1000 float aspect = (float) getWidth() / (float) getHeight(); cam.setPerspective(60.0f, aspect, 1.0f, 1000.0f); // Place the camera at z=5.0 in world space // View vector is along the negative Z axis transform = new Transform(); transform.postTranslate(0.0f, 0.0f, 5.0f); g3d = Graphics3D.getInstance(); g3d.setCamera(cam, transform); Set up a Camera

Example: Rotating cube g3d = Graphics3D.getInstance(); g3d.bindTarget(getGraphics()); try { g3d.clear(null); g3d.render(cube, transform); } finally { g3d.releaseTarget(); } xAngle += 1; yAngle += 2; zAngle += 3; transform.setIdentity(); transform.postRotate(xAngle, 1.0f, 0.0f, 0.0f); transform.postRotate(yAngle, 0.0f, 1.0f, 0.0f); transform.postRotate(zAngle, 0.0f, 0.0f, 1.0f); Clear the buffers, render, animate

Demo
Rotating cube

Agenda
Why should I care?
Getting started
Low level features
Scene graph
Dynamic meshes
Animation

Add bitmap graphics
3D graphics is mostly vector graphics
mathematical shape definitions
define camera, lights, simulate
“ computer graphics look”, pretty boring
Bitmap graphics adds visual richness
background images
texture maps on surfaces
sprites

Texturing tricks
Tip: Use light maps for lighting effects
Usually faster than per-vertex lighting
Use luminance textures, not RGB
Multitexturing is your friend
Tip: Use the perspective correction hint
Almost always faster than adding more vertices
But first check if the implementation supports it!

More bitmap tricks
Tip: Use background images
Can be scaled, tiled and scrolled very flexibly
Generally much faster than sky boxes or similar
Tip: Use sprites as impostors, particles
Generally (much) faster than textured quads
Unscaled mode is (much) faster than scaled
Tip: Load your images with the Loader
Loader.load(“/img.png”) outputs Image2D
Going through MIDP Image is a waste of memory

Demo
Backgrounds and texturing

Agenda
Why should I care?
Getting started
Low level features
Scene graph
Dynamic meshes
Animation

The scene graph SkinnedMesh Group Group Group Mesh Sprite Light World Group Camera Group MorphingMesh Actually, it’s just a tree Not allowed!

How to find an object in the scene VertexArray Background Image2D VertexBuffer IndexBuffer Appearance Image2D Texture2D Mesh Group Mesh Mesh Group Light Camera World UserID defaults to 0 find(4) Diagram courtesy of Sean Ellis, Superscape Mesh 4 0 1 2 3 4 4 5 6 1 0 6 7 8 9 10 11 UserIDs not necessarily unique

From this node to the parent node
Ignored for the root node
Composed of four parts
Translation T
Orientation R
Non-uniform scale S
Generic 3x4 matrix M
Composite: C = T R S M
Node transformations Group Group Group Mesh Sprite C C C C C World C

Node transformations
Tip: Keep the transformations simple
Favor the T R S components over M
Avoid non-uniform scales in S
Tip: How to scale along an arbitrary axis
There is no direct support ( C = T R N -1 S N M)
Method 1: R’ = R N -1 and M’ = N M
Method 2: Use an extra Group node
Tip: How to rotate about an arbitrary point
Again, no direct support ( C = T P -1 R P S M )
Method 1: Use an extra Group node
Method 2: If S is identity, combine into T and M

Example: Set up a hierarchy private Group group1, group2; private Mesh cube1, cube2, cube3; … cube1 = new Mesh(cubeVB, triangles, appearance); cube2 = new Mesh(cubeVB, triangles, appearance); cube3 = new Mesh(cubeVB, triangles, appearance); group1 = new Group(); group2 = new Group(); group1.addChild(cube1); group1.addChild(cube2); group1.addChild(group2); group2.addChild(cube3); group2 group1 cube1 cube3 cube2

Animate and render the hierarchy g3d.render(group1, null); … cube1.setOrientation( yAngle, 0.0f, 1.0f, 0.0f ); cube1.setTranslation( 0.0f, 2.0f, 0.0f ); cube2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f ); cube2.setTranslation( 0.0f,-2.0f, 0.0f ); group2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f ); group2.setTranslation( -2.0f, 0.0f, 0.0f ); cube3.setOrientation( zAngle, 0.0f, 0.0f, 1.0f ); cube3.setTranslation( -2.0f, 0.0f, 0.0f ); cube3.setScale( 0.5f, 0.5f, 0.5f );

Demo
Hierarchic transformations

Agenda
Why should I care?
Getting started
Low level features
Scene graph
Dynamic meshes
Animation

MorphingMesh
A base mesh and several morph targets
Result = weighted sum of morph deltas
Can morph any vertex attribute
Vertex positions
Colors
Normals
Texture coordinates
For vertex morphing animation

MorphingMesh Example: Animating a rabbit’s face Base Target 1 eyes closed Target 2 mouth closed Animate eyes and mouth independently

SkinnedMesh
Stretch a mesh over a hierarchic “skeleton”
The skeleton consists of scene graph nodes
Each node (“bone”) defines a transformation
Each vertex is linked to one or more bones
Articulated characters without cracks at joints

SkinnedMesh
Neutral pose, bones at rest
Weighted skinning

SkinnedMesh
Bone B rotated 90 degrees
Weighted skinning

SkinnedMesh Example: Animating an arm No skinning Local skinning one bone per vertex Smooth skinning two bones per vertex

Agenda
Why should I care?
Getting started
Low level features
Scene graph
Dynamic meshes
Animation

Animation: Relevant classes KeyframeSequence AnimationController AnimationTrack A link between sequence, controller and target Object3D Base class for all objects that can be animated Controls the playback of one or more animations Storage for keyframes Defines interpolation mode

KeyframeSequence Keyframe is a time and the value of a property at that time Can store any number of keyframes Several keyframe interpolation modes Can be open or closed (looping) sequence time t v KeyframeSequence Diagram courtesy of Sean Ellis, Superscape

AnimationController Can control several keyframed animations together Determines relationship between world time and sequence time world time AnimationController Diagram courtesy of Sean Ellis, Superscape 0 d sequence time t 0 t s 0 d sequence time 0 d sequence time

Animation Relates animation controller, keyframe sequence, and object property together. Identifies animated property on this object Call to animate(worldTime) s v Calculate sequence time from world time Look up value at this sequence time Apply value to animated property Diagram courtesy of Sean Ellis, Superscape Object3D AnimationTrack AnimationController KeyframeSequence 0 d sequence time

Animation
Tip: You can read back the animated values
Use the animation engine for anything you want
Much faster than doing interpolation in Java
Especially in case of quaternions and splines

Example: Set up keyframes KeyframeSequence createKeyframeSequence() { // 10 keys, 3 components, spline interpolation KeyframeSequence ks; ks = new KeyframeSequence(10,3,KeyframeSequence.SPLINE); // x grows linearly, y wiggles up-down, z remains 0.0f float[] tb = { 0.0f, 0.0f, 0.0f }; tb[0] = -4.0f; tb[1] = 0.0f; ks.setKeyframe(0, 0, tb); tb[0] = -3.0f; tb[1] = 2.0f; ks.setKeyframe(1, 10, tb); … tb[0] = 5.0f; tb[1] = 0.0f; ks.setKeyframe(9, 90, tb); ks.setDuration(100); // sequence duration in local time ks.setValidRange(0, 9); // all 10 keyframes are valid ks.setRepeatMode(KeyframeSequence.LOOP); return ks; }

Example: Set up controller controller = new AnimationController(); // The animation is active between these world times controller.setActiveInterval(0, 1500); // Set speed, scale “around” given time (0) controller.setSpeed(2.0f, 0); // Create animation track with keyframes & controller ks = createKeyframeSequence(); at = new AnimationTrack(ks,AnimationTrack.TRANSLATION); at.setController(controller); // Attach it to our cube cube.addAnimationTrack(at);

Demo
Keyframe animation

Summary
M3G is the standard 3D API for J2ME ™
Will be supported in millions of devices
Not just an API
M3G also defines a binary file format
M3G is flexible
Using the immediate mode gives you full control
High-level features make your code faster & smaller

Demo
3D snowboarding

For More Information
M3G specification
www.forum.nokia.com/java/jsr184
SDKs, additional documentation
www.forum.nokia.com/ (click “Series 40”)
developer.superscape.com/ (registration needed)
M3G content creation tools
www.discreet.com/
www.superscape.com/

Q&A
Thanks: Kari Kangas, Sean Ellis, Nokia M3G team, JSR-184 Expert Group

Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center

Advanced Game Development with the Mobile 3D Graphics API

by Tomi Aarnio

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

Advanced Game Development with the Mobile 3D Graphics API Presentation Transcript