Nearly all of today's PC and console games use 3D graphics. With mobile phones getting more and more powerful and some devices even boasting a hardware 3D acceleration, development using three-dimensional graphics is an important topic for mobile computing. This module covers the JSR 184 (Mobile 3D Graphics) and explores the possibilities offered by its retained mode (high level) API for defining a scene graph and modifying objects in the world. A detailed example walks you through the basics of the free, open source 3D package Blender. Using this powerful software, you can create a sample 3D scene that you can display on your mobile phone with just a few lines of code. The challenge is about extending a sample 3D game framework, featuring an exciting car racing game! Contents: * Mobile 3D - Overview * JSR 184 (m3g) * Scene graph * Your first m3g file with Blender * Display, load and modify the 3D scene * Objects and materials * Challenge: 3D car racing game

1. Java™ Platform, Micro Edition Part 8 – Mobile 3D Graphics v3.0a – 18 April 2009 1 Andreas Jakl, 2009

2. Disclaimer These slides are provided free of charge at http://www.symbianresources.com and are used during Java ME courses at the University of Applied Sciences in Hagenberg, Austria at the Mobile Computing department ( http://www.fh-ooe.at/mc ) Respecting the copyright laws, you are allowed to use them: for your own, personal, non-commercial use in the academic environment In all other cases (e.g. for commercial training), please contact andreas.jakl@fh-hagenberg.at The correctness of the contents of these materials cannot be guaranteed. Andreas Jakl is not liable for incorrect information or damage that may arise from using the materials. This document contains copyright materials which are proprietary to Sun or various mobile device manufacturers, including Nokia, SonyEricsson and Motorola. Sun, Sun Microsystems, the Sun Logo and the Java™ Platform, Micro Edition are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. Andreas Jakl, 2009 2

3. Agenda Mobile 3D – Overview JSR 184 Scene graph Your first m3g file with Blender Display, load and modify the 3D scene Objects and materials Andreas Jakl, 2009 3

4. Introduction Mobile 3D Andreas Jakl, 2009 4

5. 3D Java Game Examples 5 Brothers in Arms: Earned in Blood (Gameloft) Planet Riders 3D (Fishlabs) Massive Snowboarding 3D (Gameloft) Tornado Mania! 3D (Digital Chocolate) For comparison: Current C++ / Open GL ES based N-Gage games (Symbian OS)(This is what is currently possible with mid-/higher class mobile phones) Blades & Magic 3D (Fishlabs) Hooked On: Creatures of the Deep (Infinite Dreams) Star Wars: The Force Unleashed (Universomo / LucasArts) One (Digital Legends) Andreas Jakl, 2009

6. Benchmark Your Phone FuturemarkSPMarkJava JSR 184http://www.futuremark.com/products/spmark/spmarkjavajsr184/ JBenchmark 3D/HD:http://www.jbenchmark.com/ Andreas Jakl, 2009 6

7. 3D APIs for Java ME Mobile 3D Graphics API (JSR 184) Java based 3D graphics library Supports immediate and retained mode (low and high level) Optional package Open GL ES (JSR 239) Provide Java bindings to the Open GL ES native 3D graphics library OpenGL ES can also be used in native code (Symbian OS / C++) Mascot Capsule Like JSR 184, additionally supported by e.g. SonyEricsson (Java 3D from Java SE is too bloated for mobile phones: ~40 MB) 7 Andreas Jakl, 2009

8. 3D APIs M3G builds on the feature set of OpenGL ES Both APIs are designed concurrently Andreas Jakl, 2009 8 Native C / C++ Applications Java Applications M3G (JSR 184) OpenGL ES Graphics Hardware

9. Performance Java is slow on mobile phones KVM most widely used today Nokia (and others) migrating to HotSpot VM from Sun But HotSpot takes a lot of RAM  problematic in real life 9 Andreas Jakl, 2009 Benchmarked on an ARM926EJ-S processor with hand-optimized Java and assembly code Diagram from TomiAarnio (see sources slide at the end)

10. The Basics of JSR 184 Get started Andreas Jakl, 2009 10

11. JSR 184 Hardware independent Uses newer hardware + 3D accelerators if present Separate code from content m3g file format contains models, animation, appearance, textures, ... Control logic in source code Tightly integrated with LCDUI (from MIDP, requires floating point support of CLDC 1.1+) Built in high level & low level API Scene graphs, keyframe animation, bones, ... 11 Andreas Jakl, 2009

12. Modes Immediate Mode Create objects programmatically Uses triangles as primitives Low level Retained mode Create objects in 3D app Scene graphs Keeps graphics processing code in native code (no Java)! Mixing is possible (e.g. insert immediate object into a scene graph) 12 Andreas Jakl, 2009

13. Tools Blender (Freeware, http://www.blender.org/) m3g exporter plug-in: http://www.nelson-games.de/bl2m3g/default.html Blender: Powerful, but difficult to learn UI Written in Python Commercial 3DS Max (integrated) Maya Lightwave 13 Andreas Jakl, 2009

14. JSR 184 – Synchronous Structure All APIs are synchronous Calls return when completed Create own thread for loading larger scenes! Structure No callbacks (interfaces, events, abstract methods) Do not override any methods 14 Andreas Jakl, 2009

15. m3g File Format Compact binary format Stores complete scene graph in file Objects are serialized Includes camera(s), lightning and objects Can be compressed Viewer:http://www.mascotcapsule.com/toolkit/m3g/en/index.php 15 Andreas Jakl, 2009

16. Scene Graph Andreas Jakl, 2009 16 World is a top-level node containing the whole scene Background defines the background against which the 3D scene is rendered World Background Groups allow the application to treat multiple nodes as a single unit Group Camera Light Groups can be nested inside other groups Sprite3D is a 2D image with a 3D position Light defines a light source in the scene Camera defines a viewpoint Sprite3D Group Group Mesh User object Mesh defines the 3D geometry of a visible object An arbitrary user object can be associated with any scene object MorphingMesh SkinnedMesh Morphing and skinned meshes are animated geometry objects Based on the scene graph image from the JSR 184 API documentation

17. Graph vs. Tree Scene graph has a tree structure Node can belong to at most one group at a time But components (VertexArrays, textures, ...) can be shared No cyclic structures allowed Andreas Jakl, 2009 17 Node Node Component Group Group Component Node

18. Scene Graph – Example Andreas Jakl, 2009 18 World root node Camera and light The car The landscape scene group Tree The tree consists of two meshes The tree has no texture image Grass block Grass texture

19. Node Abstract base class for all scene graph nodes Contains information about: Transformation Parent Alignment (also automatic!) Visibility ... Andreas Jakl, 2009 19 Node Camera Light Group Mesh Sprite3D World

20. Key Classes 3D graphics context Handles all rendering Graphics3D Utility to load m3g and png files (entire scene graphs or single objects) Loader Root node of the scene graph World 20 Andreas Jakl, 2009

21. Graphics3D Stores global state Rendering target, viewport, depth buffer, back buffer Camera, light sources Rendering quality hints (Antialiasing, dithering, true color rendering) Each node has its own local state Geometry & appearance (material, textures, ...) Transformation relative to the parent or world 21 Andreas Jakl, 2009 Graphics3D

22. Graphics3D – Rendering Modes Retained mode Render entire world (scenegraph) Uses lights and camera nodes from the world Immediate mode Render scene graph nodes or submesh Uses lights and camera from Graphics3D object Andreas Jakl, 2009 22

23. Graphics3D – Render Targets Graphics3D can render to any Graphics object or Image2D  possible to render to textures (eg. environment mapping) Andreas Jakl, 2009 23 World M3G(JSR 184) Graphics3D Image2D Graphics Canvas Image MIDP CustomItem

24. Graphics3D – Example Target has to be bound and released Do not modify the target from the outside in between Andreas Jakl, 2009 24 publicclassMyCanvasextendsCanvas { Graphics3D myG3D = Graphics3D.getInstance(); publicvoidpaint(Graphics g) { try { myG3D.bindTarget(g); // ... update thescene ... // ... render thescene ... } finally { myG3D.releaseTarget(); } }

25. World Node Top level node for a scene graph Contains Other nodes that compose the scene Background Camera, Light, Fog Requires an active camera for rendering Andreas Jakl, 2009 25 World

26. Example – Goal Create an m3g file with Blender Load and display the file in a MIDlet Rotate the object Andreas Jakl, 2009 26

27. Blender The basics of Andreas Jakl, 2009 27

28. Blender – Interface Andreas Jakl, 2009 28 3D Viewport Light Sample Cube Camera Buttons Window

29. Blender – Windows Andreas Jakl, 2009 29 The highlighted window receiveskeyboard inputs: Highlighted (slightly brightercolour) Change the window type: ... you can split, join and resize windows anddrag mini windows (with the )

30. Blender – 3D Viewport Andreas Jakl, 2009 30 Click + hold middle mouse button  Rotate viewport Shift + middle mouse button  Pan view (move sideways) Mouse wheel  Zoom Z-key  Toggle wireframe or solid mode NUM5 (with NUM LOCK on)  Toggle Ortographic / Perspective NUM0 (with NUM LOCK on)  Camera view (MMB will exit it) Choose viewport shading (more options than Z) NUM refers to the number pad – if you pressed the numbers above the normal keys instead, press 1 to return to the normal view (for viewing layer 1)

31. The 3D Cursor Andreas Jakl, 2009 31 Task: place the 3D cursor between the camera and the cube Make sure you are in object mode (press TAB to toggle) Disable the “3d transform manipulator” to make sure you can move the cursor without selecting the cube by accident Hit NUM7 to get to the top view Click with LMB between cube and camera Choose different view (NUM1 – front view or NUM3 – side view) Click between cube and camera with LMB Rotate the view to see if it was positioned correctly 1 2 3 4 Note: we’re working in a 3D space but only have a 2D screen – therefore you need two views to set all three coordinates of the cursor! 5

32. Deleting and Adding Objects Andreas Jakl, 2009 32 Task: replace the cube with a monkey Set “Object Mode”; switch off “3d transform manipulator” (see previous slide) Move the cursor to the center (Shift+CKEY) RMB on the cube to select it DELKEY to delete the object. Confirm in the prompt window (“Erase Selected”) Use the Add menu to add a new object. Here: Add  Mesh  Monkey Blender automatically switches to edit mode. Go to object mode (TAB), press CKEY to center the cursor on the screen, ZKEY to toggle solid and wireframe mode. Zoom in (Mouse wheel) 3 4 5 6

33. Material Andreas Jakl, 2009 33 2 3 Task: change the material of the monkey Select the monkey object (RMB); set “Object Mode” Press the shading button In contrast to the cube, the monkey doesn’t have a default material. Click the button next to “Add New” and choose “O Material” (same that was originally on the cube) In the “Material” tab, set any color you want for “Col” and “Spe” (specularcolor, for highlights) Press the button with the small car in the “Links and Pipeline” tab for an automatic material name Set the draw type of the 3D Viewport to “Shaded” for a more accurate preview 4 5 6

34. IDs and m3g Export Andreas Jakl, 2009 34 Task: to access the nodes from the m3g file in source code, they need IDs. Set “Object Mode” Select the monkey with the RMB Add #01 at the end of the object name to give it the ID 1 in the m3g file Do the same for the light (#02) and the camera (#03) Task: export the m3g file (requires the m3g exporter plug-in) File  Export  M3G You can safely disable texturing

35. m3g File Andreas Jakl, 2009 35 Open the m3g file in the m3g viewer* and choose Display  Scene Graph View Our IDs have been saved in the file * http://www.mascotcapsule.com/toolkit/m3g/en/index.php

36. Display the 3D scene Switching to the Java side Andreas Jakl, 2009 36

37. Loading the m3g File Create a simple MIDlet and Canvas-class (sample start project is provided) Add the monkey.m3g to the root of the .jar Init code snippetof the Canvas: Object3D[] roots = null; try { // Load the m3g file // The loader will return all root level objects of the file, // which are not referenced by any other objects. // Always state an absolute path ("/")! roots = Loader.load("/monkey.m3g"); } catch (IOException ex) { // couldn't open the file, or invalid data in the file ex.printStackTrace(); } // Usually the world node is the only and first node. // If loading unknown files, you should check it though. // Save the world node to an instance variable for (inti = 0; i < roots.length; ++i) { if (roots[i] instanceof World) { iWorld = (World) roots[i]; } } Andreas Jakl, 2009 37

38. Game loop in a GameCanvas Andreas Jakl, 2009 38 public void run() { // Get the singleton Graphics3D instance that is associated with this midlet. Graphics3D g3d = Graphics3D.getInstance(); // Measure the time that has passed since the prev. frame long start, elapsed = 0; int time = 0; while (iIsActive) { start = System.currentTimeMillis(); try { // Bind the 3D graphics context to the given MIDP Graphics object. g3d.bindTarget(getGraphics()); // Update the world [...] iWorld.animate(time); // Animate the world // Render the view from the active camera g3d.render(iWorld); } finally { // Release the graphics context g3d.releaseTarget(); } flushGraphics(); // Flush the rendered image to the screen // Give other threads a chance to run Thread.sleep(20); // This sample omits try and catch elapsed = System.currentTimeMillis() - start; time += elapsed; // Time that has passed since the last frame } }

39. MIDP Relationship Andreas Jakl, 2009 39 Mobile 3D MIDP 2 binds Graphics3D Graphics renders to renders renders World Screen

40. The Camera The camera was too far away in the Blender scene Move it towards the object after loading the world: Andreas Jakl, 2009 40 Camera cam = iWorld.getActiveCamera(); cam.setTranslation(-5.0f, 5.0f, -4.0f);

41. Node Transformations Each node: local coordinate system Node transformation: from this node to the parent node (ignored for root) Translation T Rotation R Non-uniform scale S Generic matrix M Composite C = TRSM Andreas Jakl, 2009 41 C World C Group C C Sprite3D Group C C Group Mesh

42. Modifying Objects Rotate the monkey Andreas Jakl, 2009 42 private static final int ID_MONKEY = 1; // Find searches through all children of this node Node monkey = (Node) iWorld.find(ID_MONKEY); switch (getKeyStates()) { case LEFT_PRESSED: // Parameters: angle, x / y / z component of // the rotation axis. Here: all on z axis monkey.postRotate(5.0f, 0.0f, 0.0f, -1.0f); break; case RIGHT_PRESSED: monkey.postRotate(5.0f, 0.0f, 0.0f, 1.0f); break; }

43. Objects and Materials What is actually rendered? Andreas Jakl, 2009 43

44. Textures Add visual richness Backgrounds Surface structure Sprites for “cheating”(no complex 3D models) Andreas Jakl, 2009 44 Character images copyright David Dang http://www.chi-3d.co.uk/ Sprites in “Doom” by id Software

45. Perspective Correction Perspective correction Expensive, avoid if possible But still better than a lot more polygons Nokia: 2% fixed overhead, 20% in the worst case Andreas Jakl, 2009 45 Perspective correction http://en.wikipedia.org/wiki/Texture_mapping

46. Light Maps Lightning Use light maps instead of real light Pre-calculate light for non-textured (colored) objects: vertex coloring Andreas Jakl, 2009 46 light map(unfiltered) light map(filtered) w/o light maps with light maps Images from lecture slides of Stephen Chenney (see sources at the end)

47. Mobile Textures II Backgrounds Use background images, no skybox or real 3D Sprites Much faster than real geometry or textured quads But too much overhead for particle effects Images Load through Loader class (Image2D) – going through MIDP Image wastes memory Andreas Jakl, 2009 47 Sprites in a 3D scene Playman World Soccer (Mr.Goodliving) 3D Sprites can be useful for 2D games as well (rotation!) Tower Bloxx (Digital Chocolate)

48. IndexBuffer IndexBuffer Appearance Appearance Mesh How is the object defined? (vertices, material, ...) Andreas Jakl, 2009 48 Mesh Vertex positions, normals, texture coordinates, ... VertexBuffer Defines a submesh (consisting of triangle strips) – reference data from the vertex buffer IndexBuffer Appearance 1 Appearance per submesh. Defines color, texture, ...

49. Sprite3D 2D image with 3D position Fast, but functionally restricted alternative to textured geometry Scaled mode: billboards, trees, ... Unscaled mode: text labels, icons, ... Andreas Jakl, 2009 49 Sprite3D Appearance Contains compositing and fogging attributes Image2D

50. Animation Short overview of Andreas Jakl, 2009 50

51. Animations Defined in the m3g file Automatically played corresponding to the time by animate() method Andreas Jakl, 2009 51 Defines time within the animation, speed, weight for blending, ... Animated Object (Object3D) AnimationTrack AnimationController AnimationTrack KeyframeSequence Each track associates a property (e.g. position) with a controler & keyframe data. Multiple tracks may be blended. Values of the animated property (keyframes) + the interpolation mode. KeyframeSequence

52. KeyframeSequence Andreas Jakl, 2009 52 KeyframeSequence Keyframe: time & property value at that time Sequence: Can store multiple keyframes Several interpolation methods Can be looping property value sequence time v t Diagram based on Sean Ellis, Superscape

53. KeyframeSequence Keyframe: time & property value at that time Sequence: Can store multiple keyframes Several interpolation methods Can be looping Andreas Jakl, 2009 53 KeyframeSequence property value sequence time Diagram based on Sean Ellis, Superscape

54. KeyframeSequence Keyframe: time & property value at that time Sequence: Can store multiple keyframes Several interpolation methods Can be looping Andreas Jakl, 2009 54 KeyframeSequence property value sequence time Diagram based on Sean Ellis, Superscape

55. KeyframeSequence Andreas Jakl, 2009 55 KeyframeSequence Keyframe: time & property value at that time Sequence: Can store multiple keyframes Several interpolation methods Can be looping property value sequence time v t Diagram based on Sean Ellis, Superscape

56. AnimationController Controls position, speed and weight of an animation sequence Usually controls multiple AnimationTracks (properties) at the same time Defines mapping from world time to sequence time Andreas Jakl, 2009 56 AnimationController world time sequence time 0 s d t1 t2 Diagram based on Sean Ellis, Superscape

57. Morphing Andreas Jakl, 2009 57 Base Target 1eyes closed Target 2 mouth closed Animate eyesand mouth independently Unfortunately, the animations are not exported to the pdf Mesh MorphingMesh Images from Aarnio and Pulli (see sources at the end)

58. Skinning Andreas Jakl, 2009 58 No skinning Local skinning one bone per vertex Smooth skinning two bones per vertex Unfortunately, the animations are not exported to the pdf Mesh SkinnedMesh Images from Aarnio and Pulli (see sources at the end)

59. Summary M3G enables real-time 3D on mobile phones Works both with software and hardware 3D Supported by millions of devices Mixture of low- and high level Allows easy creation of scenes But still gives the developer full control Flexible architecture Based on OpenGL ES features Defines a convenient file format Andreas Jakl, 2009 59

60. Sources TomiAarnio and Kari Pulli: Advanced Game Development with the Mobile 3D Graphics API. JavaOne Conference, 2004. Andrew Davison: Special Topics in Info. Eng.: Introductionto J2ME Programming. Lectureslides, 2007 Qusay H. Mahmoud: Getting Started With the Mobile 3D Graphics API for J2ME. http://developers.sun.com/mobility/apis/articles/3dgraphics/ Carol Hamer: Creating a basic M3G file with Blender. http://bittyjava.wordpress.com/2007/01/04/creating-a-basic-m3g-file-with-blender/ David Millet, Arthur Tombs et al.: Blender 3D: Noob to Pro. http://en.wikibooks.org/wiki/Blender_3D:_Noob_to_Pro MikaelBaros: 3D programming tutorial for mobile devices using M3G (JSR 184). http://developer.sonyericsson.com/site/global/techsupport/tipstrickscode/mobilejava3d/p_java3d_tutorial_part1_compliments_redikod.jsp Kari Pulli et al.: Developing Mobile 3D Applications With OpenGL ES and M3G. Eurographics 2006. http://people.csail.mit.edu/kapu/mobile_3D_course/index.html JanneHellsten: Building scalable 3D apps - Tips & tricks for developers. http://www.khronos.org/developers/library/seoul_04_2005/Hybrid_Tips-and-Tricks.ppt Stephen Chenney: Computer Game Technology. Lecture slides, 2001. http://www.cs.wisc.edu/graphics/Courses/638-f2001/ Andreas Jakl, 2009 60

61. Thanks for your attention That’s it! Andreas Jakl, 2009 61