Your SlideShare is downloading. ×
Real-Time Non-Photorealistic Shadow Rendering
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Real-Time Non-Photorealistic Shadow Rendering

5,454
views

Published on

Master of Science Thesis: Real-Time Non-Photorealistic Shadow Rendering

Master of Science Thesis: Real-Time Non-Photorealistic Shadow Rendering

Published in: Education, Art & Photos, Business

0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,454
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
220
Comments
0
Likes
8
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Master of Science Thesis Real-Time Non-Photorealistic Shadow Rendering by Tam´s Martinec a Supervisor: Jan Westerholm Tampere, 2007 ˚bo Akademi University A Master’s programme in Software Engineering Department of Information Technologies Information Faculty
  • 2. To Ildik´, Marcell and Tam´s o a
  • 3. Contents Contents i 1 Non-Photorealistic Rendering 7 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.1 Two-dimensional techniques . . . . . . . . . . . . . . . . . . 8 1.2.2 2.5D NPR techniques . . . . . . . . . . . . . . . . . . . . . 10 1.2.3 Three-dimensional NPR techniques . . . . . . . . . . . . . . 10 1.3 Styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3.1 Toon-shading, Cel-shading, Outlining, Stylized rendering . 12 1.3.2 Pen and Ink, engraving and line art . . . . . . . . . . . . . 12 1.3.3 Sketching and hatching . . . . . . . . . . . . . . . . . . . . 13 1.3.4 Painterly style . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.3.5 Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2 Real-time Shadow Rendering 17 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 Shadow concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.3 Shadow maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.3.1 Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 Non-linear shadow maps . . . . . . . . . . . . . . . . . . . . 20 2.4 Shadow volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4.1 Silhouette edges . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4.2 Inside-outside test . . . . . . . . . . . . . . . . . . . . . . . 22 2.4.3 Stencil shadow volumes . . . . . . . . . . . . . . . . . . . . 23 2.4.4 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.5 Hybrid algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.6 Soft shadows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.6.1 Image based soft shadows . . . . . . . . . . . . . . . . . . . 25 2.6.2 Geometry based soft shadows . . . . . . . . . . . . . . . . . 28 i
  • 4. ii CONTENTS 3 Hardware accelerated computer graphics 31 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Modern graphics pipelines . . . . . . . . . . . . . . . . . . . . . . . 31 3.2.1 Modeling transformations . . . . . . . . . . . . . . . . . . . 32 3.2.2 Illumination . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.3 Viewing transformations . . . . . . . . . . . . . . . . . . . . 32 3.2.4 Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.5 Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.6 Rasterization . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.7 Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.3 Shader programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3.1 Vertex shaders . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3.2 Pixel shaders . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4 Shader developing environments . . . . . . . . . . . . . . . . . . . 34 3.4.1 Rendermonkey . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.4.2 FX Composer . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4 A real-time NPR shadow rendering example 39 4.1 Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.2 Developing vertex and pixel shaders . . . . . . . . . . . . . . . . . 40 4.2.1 A general frame application using shader programs . . . . . 40 4.2.2 Hatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.2.3 Shadow hatching . . . . . . . . . . . . . . . . . . . . . . . . 46 4.2.4 Distance based shading of a transparent shadow texture . . 49 5 Conclusion and future work 51 5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Bibliography 53 A NPR shadow pass vertex shader source code 59 B NPR shadow pass pixel shader source code 63 List of Symbols and Abbreviations 67 List of Figures 68
  • 5. Abstract Non-photorealistic rendering (NPR) is a branch of computer graphics where im- ages are purposefully created to be different than real images. Early NPR work mostly focuses on recreating artistic styles like pen and ink, watercolor or paint- ings, but NPR possibilities are a lot more diverse than just recreating human artistic styles. By modifying or completely redesigning photorealistic graphics pipelines, completely new, unseen visual worlds can be created. A relatively rarely seen area in NPR is NPR shadows. Since shadow rendering requires substantial computing power, it is rarely in focus of NPR work. With more and more powerful computer hardware today it is possible to satisfy the resource needs of non-photorealistic renderings complete with shadows. This thesis presents a real-time non-photorealistic shadow rendering solu- tion to emphasize the importance of shadows in NPR. Two Rendermonkey demo shaders are reused to combine a hatching shader with a shader that uses the shadow map algorithm to generate hard shadows. The shadowed area is repre- sented with a different hatching direction and the shadow is transparently blended to the base hatching on the receiver surface. 1
  • 6. Acknowledgements I would like to thank Jan Westerholm for being my guide through the long journey of writing this thesis. His remarks and advice were essential to complete my work. I would also like to thank him for introducing me to the world of computer graphics and choosing me as course assistant as this opportunity provided me insight in areas where I would have not gone by myself. I would also like to thank Jukka Arvo for his remarks on the initial material and for his help at choosing this interesting topic. Furthermore I would like to thank Roy Mickos, Jarno Juntunen, Anne Aal- tonen and Seppo Kolari to support my studies besides working. Finally I thank my family: Ildik´, Tam´s, Marcell and my parents for their o a support throughout the times of writing the thesis. 3
  • 7. Introduction Throughout human history there has been an urge to reproduce the images we perceive with our eyes. Starting from early cave paintings [Cav07] to recording motion pictures with high-end digital cameras we capture moments of the visual world around us. The longing to recreate what we see made us drafting them long before we could really create the images exactly. We made drawings that reminded us of what we saw as well as we could. The results are usually far from the image that was formed on our retinas when we observed them. Sometimes it would take an intelligent being to be able to recognize the original. The image is different because either the tools we used are not capable of recreating the image, we do not remember well enough what we saw or our drawing technique is not perfect. It is also possible that we do not want to recreate the exact image. We might want to emphasize different aspects of the object of our drawing, make an abstraction of it, or depict features that are not visible otherwise but which we know are there because for example we examined it from several angles. The resulting differences are visible throughout the history of art. Human art therefore – possibly except for the 1960s movement called photorealism – does not recreate exactly real world images. Human art is not photorealistic. [HF05] Computer graphics is similar to human art in the sense that in the beginning it did not have the means for the exact recreation of the visual world around us. Early computers had low-resolution displays and scarce resources to achieve the several orders of magnitude higher resolution of the images drawn by photons arriving at our eyes. Early computer images were therefore as far from photo- realism as prehistoric cave paintings. Nevertheless computer graphics had the theoretical possibility that one day, if computers kept getting better at the same rate they were always doing – at an exponential rate – we will have the means for an exact recreation of the visual world around us. Today we are very close to that goal. We now have computer generated images that can fool the untrained eye to think that the image we see could be part of our reality. If computers get a magnitude faster still and if we solve the problem of content creation then an ancient dream will become true for us: photorealistic virtual reality. 5
  • 8. 6 CONTENTS A subtle but very important part of generating images by computers is the rendering of shadows. Shadows give the viewer important guidance about the position of objects. Calculating and displaying correct shadows in computer graphics images requires an amount of resources comparable to the image ren- dering itself. For this reason shadows were not displayed on rendered images for a long time. It also took substantial time to achieve hardware speeds where shad- ows could be calculated real-time. First hard shadows – with sharp transitions between shadowed and not shadowed areas – then soft shadows – with smooth transitions – become possible to be rendered fast enough. Similarly to photorealistic rendering, non-photorealistic (NPR) rendering can be enhanced substantially by adding shadows to the rendered images. Since re- sources just recently achieved sufficient speeds for photorealistic shadow render- ing, non-photorealistic shadows can be found quite rarely in applications today. As NPR itself, NPR shadow rendering is an area of research still in its infancy. This thesis will sum up the basics of the research done in non-photorealistic rendering and shadow rendering until today and present a real-time non-photo- realistic shadow rendering example: • Chapter 1 summarizes non-photorealistic image rendering processes: possi- ble software techniques – series of image generation or alteration algorithms – that can generate such images and styles that are assembled from these techniques and resemble some real artistic styles or are completely new. • Chapter 2 describes shadow rendering approaches with a special focus on the two major hard shadow algorithms: shadow mapping and shadow vol- umes. The chapter also introduces current soft shadow algorithms. • Chapter 3 introduces the reader to the possibilities of hardware accelerated computer graphics. The chapter includes a short summary of graphics pipeline components and description of the possibilities of pixel and vertex shader programs. • Chapter 4 describes the development of a non-photorealistic shader pro- gram using two existing demonstration shaders. The shader programs will be gradually built up from a skeleton program until the required effect is achieved. • Chapter 5 draws conclusions about the work done and presents some di- rections for possible future work in this area.
  • 9. Chapter 1 Non-Photorealistic Rendering 1.1 Introduction On the road of getting from images depicted by character mosaics to picture- like, photorealistic computer generated images, the renditions were different from realistic images for a long time. Wire frame images, flat shaded three dimensional renditions, textured ray-traced images were all closer and closer to reality, but the common basic goal of the mainstream of computer graphics was to finally achieve photorealism. Apart from this general approach there were attempts to create images that are not like photographs but for some reason deviate from them. The reason could be the recreation of artistic effects, special artistic styles, some technical reason like displaying several views of an object on one drawing, or to accent some aspects of it. Those rendering techniques that purposefully differ from photorealism are collectively called non-photorealistic rendering. Non-photorealistic rendering is an area of computer graphics where images and animations are purposefully created to be different from real images. The term rightfully received a lot of criticism for its awkwardness of defining a subject as not being something. Better sounding names like ”artistic rendering” and ”art-based rendering” have appeared but they either did not cover the whole spectrum of possibilities or did not get accepted throughout the field. Non-photorealistic rendering creates images or animations that are not like reality. To achieve this several software techniques exist. These techniques are algorithms or series of algorithms that generate images that are different from real images. These techniques or several of them applied after each other can be used to achieve visual effects that are called styles because many of them resemble existing artistic styles like paintings and engravings. There are styles 7
  • 10. 8 CHAPTER 1. NON-PHOTOREALISTIC RENDERING that use only one technique to achieve the desired effect, but many styles use several techniques applied after each other. 1.2 Techniques According to the process of image creation, NPR images can be created using either two-dimensional or three-dimensional techniques. 2-D techniques usually work on photos or pre-rendered image pixels and apply a process that creates the desired effect. Three-dimensional techniques use data that describe positions and materials of objects in three-dimensional space and use processes that create an image of a view of the given objects from a certain location. Some styles use both two- and three-dimensional techniques as it is certainly possible to mix them. For example a two-dimensional painterly style can be applied after a three-dimensional toon-shading phase. The same or close to the same effect can be achieved with several techniques. More complex techniques give better or more versatile results but they also require more resources. The possibilities are almost endless. By changing the parameters of the processes a virtually infinite number of effects can be achieved and only a very small fragment of these have been discovered and used in human arts. Some of the yet unused effects that were discovered during the creation of NPR effects can be seen in some experimental animations [WFH+ 97]. In the following subsections we will consider two and three dimensional NPR techniques and evaluate their properties and merit. 1.2.1 Two-dimensional techniques Two-dimensional techniques are performed on pixels of photos or pre-rendered images. Such techniques are used for example to produce the painterly style or engravings. These techniques can be grouped according to the type of algorithms that are usually used to generate them. These groups will be described in the followings. 1.2.1.1 Halftoning, Dithering and Artistic Screening In most of these techniques shades are created with methods used in printing. Printers create images with tiny dots of ink of different sizes that looks as contin- uous shades and shapes from a distance. The patterns used for these processes can be also reused to achieve different artistic or illustrational effects. Halftoning uses dot patterns to achieve continuous shading effects. Since printers use only black, ink shades were created with arrangements of ink dots of different size so small that they could be only seen with a magnifying glass. Dithering was used to display a wider color range with devices with a limited color production capa- bilities. Using special arrangements of pixels of different colors, the effects of new
  • 11. 1.2. TECHNIQUES 9 Figure 1.1: Artistic screening with non-linear screen element mapping [SS02] colors or smoother shades can be achieved. These effects can be used for artistic or illustrational effects in NPR. In artistic screening meaningful motifs or pat- terns are used to generate the required shades and figures. Non-linear mapping of these motives can result in special warping effects of the perceived surface of the image (see Figure 1.1). Another technique belonging to this category is the method of assembling pictures from images highly reduced in size used as picture elements in a mosaic-like way. 1.2.1.2 Strokes In the beginning the purpose of non-photorealistic rendering was to automati- cally recreate artistic effects on different images. Most of human art is created by painting colors onto a two-dimensional medium with free hand strokes of a tool. These strokes are different from mathematically correct curves as the freehand drawing capabilities of humans are limited by anatomy and technique. The dif- ferences can be summarized with the following features that can be implemented in algorithms that mimic the freehand style: • Wiggliness: The curve randomly deviates from a mathematical smooth curve. • Length: The start and endpoint is different from the intended. • Thickness and brightness: Random changes of width and intensity through- out the line.
  • 12. 10 CHAPTER 1. NON-PHOTOREALISTIC RENDERING • Line ends: The shape of line ends can be different than the rest of the line. (Small hooks or dots.) Besides these main features several other attributes can be considered when drawing artistic lines. There are a number of mathematical methods of describ- ing the lines as well. Between applying the traditional line drawing algorithms with random deviations several times and strokes approximated by parametric polynomial curves and projected onto three-dimensional bodies there are several methods that can be used depending on our purpose and the resources available. The line width and features of the line drawing tool also introduce additional pa- rameters. Using wavelet-theory it is possible to arbitrarily extend the resolution of lines during magnification [FS94]. It also enables some new special effects in line drawing. Besides applying these artistic strokes in automatic image gener- ation these strokes directed by an artist in interactive painting applications can help the artistic process remarkably. 1.2.1.3 Simulating natural media and tools The look of hand drawn and painted works of art can be best approximated by simulating the interaction of natural materials and tools. To describe the inter- action of paper and pencil, canvas and brush, etc. there has been a number of models developed by researchers that can be implemented with different level of detail. The simulation can be realized with cellular automata, modeling the be- havior of the ink solvent, fluid simulation algorithms, or by modeling the behavior of small fragments of tool-material on the media [SS02]. 1.2.2 2.5D NPR techniques 2.5 dimensional techniques use some extra information in addition to the two- dimensional picture pixels that will be modified. Such information can be an infrared or x-ray image of the same subject or additional geometric information stored in an image (also known as g-buffer), like pixel-viewpoint distance (z- buffer), object selection or shading enhancing graphical masks. The additional information can be used to highlight areas of the image that are special for some reason (for example local temperature deviations from the neighboring areas in a car engine can highlight problems). This technique also usable for edge highlighting, shading, shadowing and algorithmic texturing among many others [SS02]. 1.2.3 Three-dimensional NPR techniques Three-dimensional NPR effects can be seen in video games and films most of- ten. These techniques render images of subjects based upon their geometrical
  • 13. 1.3. STYLES 11 and material information, mostly by modifying subprocesses of traditional three- dimensional rendering. Many NPR styles use one or several three-dimensional techniques to achieve their desired final look like toon-shading, illustrations in general and the hatching style. 1.2.3.1 Edges Compared to traditional rendering, non-photorealistic rendering introduces some new problems in geometry processing. In art the artist depicts the subject with the most characteristic strokes that are usually object outlines and intersections. When images are created from three dimensional models these prominent strokes will be located at positions where triangles that form the models meet. Some of these meeting lines or edges form the outlines of objects. Calculating edges are not needed in photorealistic rendering since every visible geometric information is displayed. In NPR however sometimes only these prominent edges are drawn which require their calculation [SS02]. 1.2.3.2 NPR lighting models Photorealistic rendering uses the lighting model – a simplified way of calculating the brightness of pixels in the generated images – that approximates the physi- cal behavior of light in the best possible way considering the available resources. With non-photorealistic rendering we have the opportunity to use completely different models to achieve the required similar or completely novel effect. The purpose of the lighting effect can be emphasizing, communication of more infor- mation, or to achieve artistic effects. One possibility for example is to compress and shift shading color to only a part of the shading range and use the unused colors for outlining or to describe other information. Instead of using shades that are the result of illumination, other information can be used also, like one that describes object temperature. Such lighting enables for example the analysis of object temperatures on a three dimensional model of the object from angles that are not possible in reality [SS02]. 1.3 Styles A style in NPR graphics refers to the similarity to a real artistic style or artistic technique of the image generated. Depending on what real artistic style the result is similar to, NPR styles were grouped to sets named after real artistic styles. There is an unlimited number of non-photorealistic ways an image can be created. Most early NPR styles were efforts to recreate real artistic styles algorithmically. Some real styles were created with extremely tedious work that can be greatly
  • 14. 12 CHAPTER 1. NON-PHOTOREALISTIC RENDERING Figure 1.2: Objects shaded with real-time hatching effect. [PHWF01] accelerated now with the help of computer automation. Let us examine the basic non-photorealistic rendering style types. 1.3.1 Toon-shading, Cel-shading, Outlining, Stylized rendering The earliest NPR styles were trying to achieve a cartoon-like style. This style is referred to with many names such as toon-shading, cel-shading or stylized render- ing. The image is rendered with traditional three-dimensional rendering methods but during shading only a few discreet colors are used instead of the continuous shading of photorealistic rendering. Generally outlines of objects – just like in cartoons – are emphasized at the same time for which several algorithms exist with different image quality and resource need. Although the result seems simple the process requires more resources than traditional rendering since besides nor- mal rendering done with a limited number of colors, emphasizing edges requires extra computational power [SS02]. 1.3.2 Pen and Ink, engraving and line art Images that are created by straight and curved lines placed against a usually plain background without gradations of shade or hue are called line art. Types of line art include drawings made by pen and ink and also engravings. Engravings are designs incised onto a hard, flat surface – traditionally a copper plate – by cutting grooves into it. The hardened steel tool that is used to cut the designs into the surface is called a burin. The similarly named NPR styles reproduce the image world of these artistic techniques. The intensity of lines is given by the tool used: it does not change or it can take only specific distinct values just as their real counterparts. Basic line drawing algorithms can be used to recreate art with fixed width and color lines. More sophisticated line drawing algorithms also exist to better mimic lines drawn by these tools considering for example tool shape and tool angle with respect to the surface [Gas04].
  • 15. 1.3. STYLES 13 Figure 1.3: A frame from the Oscar winning film for best visual effects using painterly style: ”What Dreams May come” [REV07] (Copyright 1998 PolyGram Filmed Entertainment Distribution, Inc.) 1.3.3 Sketching and hatching Sketching and hatching is considerably similar to the pen and ink style with the distinction of the usage of continuous shades that are achieved by hatching meth- ods. The sketching style reproduces handmade pencil or pen sketches. Uncertain, broken, redrawn lines, uneven hatching and shading, simple motifs and drawing of only the main features characterize them. The direction and style of lines conveys the pictured material, tone and form. Hatching can be used to create shading with tools that otherwise would not support the creation of continuous shades like ink tools or engraving tools. Hatching in computer graphics can be implemented with line drawing methods that require more resources but give better and more controllable results or with texturing that uses prepared images with a hatching pattern. (Figure 1.2) 1.3.4 Painterly style The painterly style imitates the effect of painting on natural media – such as paper, canvas, wood or glass – with a brush. The procedure usually takes stroke speed, pressure, type and angle of the brush and canvas properties into consid- eration. Some techniques model the interaction of paint particles and the canvas simulating the drying of the paint on the uneven surface of the canvas. Besides general painting effects, specific styles and techniques can be recreated to simu-
  • 16. 14 CHAPTER 1. NON-PHOTOREALISTIC RENDERING Figure 1.4: Technical illustration with ghosting view (Copyright 1994-2006 Kevin Hulsey Illustration) late the style of a given painter like Monet, Van Gogh or John Mallord William Turner. (Figure 1.3) 1.3.5 Illustrations Illustrations are images that are created to decorate, describe or help to under- stand written text. Since the subject is more important than photorealism with these images, computer generated pictures are quite frequently used as illustra- tions. 1.3.5.1 Scientific and medical Scientific and medical illustrations were among the first types of illustrations. The methods used in medical illustrations are so commonly used nowadays that sometimes we do not even notice them consciously, for example marking arteries with red and veins with blue. NPR rendering techniques can accelerate the cre- ation of these extremely detailed images considerably. Besides different coloring schemes several shading methods, and projection types can be used, sometimes even within the same image. Special projection types enable more views of the same object from different angles on the same picture. 1.3.5.2 Technical illustrations Technical illustrations include cutaway, ghosting (Figure 1.4), section and ex- ploded views of objects. Several computer aided design (CAD) tools support the automatic creation of such technical illustrations. Cutaway views let us see the inner parts of object as if they were cut open. Ghosting images display outer layers of objects transparent thus giving a better understanding of the inside contents of objects. Exploded views display subjects taken apart, helping to un-
  • 17. 1.3. STYLES 15 derstand how the specific object is put together from smaller components. Since these illustration are rarely photorealistic their creation belong to NPR styles as well. The features of these images are nowadays controlled by ISO standards. Some parts of these figures can be omitted, some can be emphasized for better understanding or for highlighting features. The placement of symbols and labels can also be automated with these tools.
  • 18. Chapter 2 Real-time Shadow Rendering 2.1 Introduction Shadows play an important role in our vision. The intensity and location of shad- owed areas help the viewer identify positions and spatial relations of objects in three-dimensional space (see Figure 2.5). Calculating physically correct shadows within the time frame of change perceivable by a human eye is far out of the reach of current computer hardware. For applications of computer graphics where hu- man or other outside interaction changes the scene of the images rendered require that a complete image generation is executed within this time limit. Generally if an algorithm has to finish within a pre-set time limit it is called a real-time algorithm. In computer graphics real-time rendering means that the rendering cycle has to finish within the time limit of human perception of change – roughly 15ms. Physically correct shadows require tracing light after its emission from the light source, its reflection and absorption on all objects that the light falls on in the scene until all of its energy is exhausted. Implementing such algorithms that model this behavior results in hours of render times with currently used image resolutions on available hardware. To be able to render images with shadows within the limits given in real-time computer graphics, our model has to be largely simplified. In this chapter the two most common hard shadow rendering methods – shadow maps and shadow volumes – will be presented after describing basic shadow concepts. Also a method called hybrid algorithm is described that use features of both basic algorithms. At the end of the chapter a short summary of soft shadow rendering algorithms will also be given. 17
  • 19. 18 CHAPTER 2. REAL-TIME SHADOW RENDERING 2.2 Shadow concepts Shadowed areas are sections of space where objects partially or fully block the path of light to that specific area. Point lights create hard shadows since there are no such areas in lit surfaces where only a part of the light source is seen, therefore lit by only a section of the light source. If the light source is extended it creates soft shadows where transitions from shadowed to lit areas are continuous. (Figure 2.6) The object blocking the path of light is called blocker or occluder, the object where shadowed and lit regions are created because of the occluder is called the receiver. The outline shape of an object with a featureless interior is called silhouette. Shadow shapes created on receivers are projected silhouettes of blockers. 2.3 Shadow maps One of the most widely used shadow algorithms is shadow maps, a concept in- troduced by Williams in 1978 [Wil78]. The shadow map is a two-pass algorithm. In the first pass depth values – the distance between the light source and oc- cluders – are rendered into a buffer called the shadow map of the scene from the point of view of the light source. In the second pass the image is rendered from the camera’s point of view using the shadow map to decide whether a pixel is in shadow or not. The decision can be made using the depth values. Williams transformed the rendered pixel distance value into the light source coordinate system and compared the distance to the depth map value to decide if the pixel is in the shadow (later called backward shadow mapping). Shadow maps are images that describe the distances of object pixels from the light source. By transforming these maps to camera space it can be decided whether rendered pixels are lit or are in shadow. It is also possible to transform the shadow map into the cameras coordinate system and make the same comparison (forward shadow mapping [Zan98]). By transforming the shadow map to the camera coordinate space the depth values of the shadow map get comparable to the camera point of view depth map and a comparison gives as a result whether the pixel perceived by the observer is in the shadow or not. (Figure 2.1). The shadow map transformation can be executed us- ing hardware acceleration since most current hardware support projective texture mapping. Being the computationally cheapest real shadow rendering algorithm shadow maps are widely used in computer games and other real-time rendering engines.
  • 20. 2.3. SHADOW MAPS 19 Figure 2.1: Geometric configuration of the shadow map algorithm. Sample A is considered to be in shadow because the point on the receiving plane is farther from the light source than the distance stored in the shadow map (which belongs to the blocker). In contrast, sample B is illuminated. [Cha04] 2.3.1 Artifacts Since shadow mapping is created using a very limited model of real shadows it has its artifacts (visible differences from real shadows). Two major problems are present when using the shadow map algorithm: aliasing and self-shadowing artifacts. 2.3.1.1 Aliasing The resolution of the shadow map introduces aliasing along shadow edges that can be highly distracting when looked at closely (Figure 2.3). The shadow maps are bitmaps with usually smaller resolution than the screen, therefore when they are projected their pixels may cover several screen pixels resulting in blocky shadow outlines. To smooth these shadow edges several partial and full solutions have been published that require more or less resources [Bra03]. Rendering shadow maps with higher resolution decreases aliasing but - since the shadow map is projected to observer space - at certain screen configurations aliasing is quite
  • 21. 20 CHAPTER 2. REAL-TIME SHADOW RENDERING Figure 2.2: Bias related artifacts. (a.) Too little bias: incorrect self shadowing effect - pixels falsely determined to be in shadow (b.) Too much bias applied: shadow boundary problem. (c.) Correct bias. [Cha04] severe even with high resolution shadow maps. 2.3.1.2 Self-shadowing artifacts The discreet nature of pixels also lead to the problem that the projected shadow map pixels and camera viewpoint rendered pixels have different sizes. This causes incorrect self shadowing effects where pixels are falsely determined to be in the shadow. This is known as surface acne (Figure 2.2.). This artifact is also the result of that the projected shadow map pixels and the calculated image pixels don’t have a one to one mapping: some shadowed pixels are calculated not to be in the shadow. To correct this a constant value called bias is usually added to the shadow map value that shifts the shadow map so that the comparison will not result in shadowed pixels when they are not there. Selecting the correct bias is not easy since too much bias can result in the removal of certain parts of the shadow [Cha04]. 2.3.2 Non-linear shadow maps The original shadow mapping algorithm covers a limited volume of the scene where correct shadows can be calculated since the shadow map is a rectangle. Therefore the procedure is not usable or is only usable in a limited way for point lights. One solution to the problem is to use several shadows maps (6 to cover all directions from a point light source [Die01]) or to use non-linearly projected for example dual paraboloid shadow maps that can cover the whole sphere of a point light with only two shadow maps [Bra03]. Non-linear shadow maps are only presented in research papers at the moment, their usage is expected in the future as using them requires more resources than the use of linear shadow maps.
  • 22. 2.4. SHADOW VOLUMES 21 Figure 2.3: Shadow map aliasing: (a.) 256x256, (b.) 512x512, (c.) 1024x1024 shadow maps. [Cha04] 2.4 Shadow volumes A more accurate shadow rendering method is the shadow volume algorithm. Shadow volumes enable precise shadow calculations but they also require a lot more computational power than shadow maps. Shadow volumes describe shad- owed volumes of the scene using calculated geometry, instead of pixels. Shadow volumes are geometrically determined sections of the scene space that are used to determine whether objects and thus pixels of the rendered images lie in shadow. After creating shadow volumes it is possible to exactly decide about each pixel of the rendered image whether the surface it depicts is in the shadow or not. The original shadow volume algorithm [Cro77] had numerous limitations that mostly had been resolved since then [Bra03]. Also several suggestions have been made to accelerate shadow volume calculations with current hardware features [Bra03]. 2.4.1 Silhouette edges Creating the geometry of shadow volumes is done by identifying silhouette edges from the point of view of the light source and extending the silhouette to infinity away from the light. The result of the operation is a semi-infinite volume that describes the section of space that the given object creates a shadow in. There are several published methods of deciding whether an edge in an object is a silhouette edge or not. The most simple way is to check if the silhouette edges are connecting front and back facing triangles with respect to the light source. The orientation of the triangles can be decided by calculating the dot product of the face normal and a vector to the light source. A positive product indicates front facing, and a negative product indicates back facing triangles. By repeating this calculation for all connecting triangles a set of silhouette edges can be obtained that form a closed loop. These calculations require that the objects are such polygonal models that are oriented (the face normals point away from the object), and
  • 23. 22 CHAPTER 2. REAL-TIME SHADOW RENDERING Figure 2.4: Inside-outside test: Counting intersections with shadow volumes (in- crementing at front facing polygons and decrementing at back facing polygons) give zero for non-shadowed pixels and non-zero for shadowed pixels. watertight implying that any point in space is unambiguously inside or outside the object. By extending the silhouettes far away to infinity from the light source semi- infinite volumes can be formed that describe regions in space which are in the shadow with respect to the given light source. 2.4.2 Inside-outside test After shadow geometry has been calculated we have to decide about each rendered pixel if it lies within a shadow volume or not. The simplest algorithm for this is to follow a ray from the camera to each rendered pixel and count the crossings of shadow volumes [Cha04]. If a front facing shadow polygon is crossed a counter is incremented and if a back facing shadow polygon is crossed it is decremented. If the ray intersects an object the counter is tested for the value zero which implies that it lies outside of all shadow volumes and therefore is not in shadow. If the counter is non-zero then the pixel is part of a shadowed area of an object. (Figure 2.4.) This method works in most scene configurations but there are cases when the
  • 24. 2.4. SHADOW VOLUMES 23 Figure 2.5: Shadow volumes: Final image (left), Silhouette edges (middle), Shadow volumes (right) [Bra03] result is incorrect. If the camera is within a shadow volume or is clipped against the near plane of a shadow volume some regions will be calculated with inverted shadows. 2.4.3 Stencil shadow volumes The inside-outside test for shadow volumes can be accelerated using hardware stencil buffers [Hei91] enabling the method to be used in real-time applications. There are several approaches of the implementation: depth pass, depth fail, split plane and exclusive-or. 2.4.3.1 Depth pass or z-pass In the depth pass method only shadow volume polygons between the camera and object are counted. Depth pass means that for these polygons the depth test will pass and they will be used. In modern graphics hardware a bitmap called the stencil – generally used for stencil-like effects – can be used for operations in the process of calculating the final images. In the depth pass method the stencil image is calculated to describe shadowed and non-shadowed areas of the image. The stencil is calculated in two passes. First the front faces of the shadow volumes are rendered, each incrementing the stencil buffer values, then in the second pass back faces are rendered decrementing the buffer values. After the operations the stencil buffer will have zeroes at non-shadowed pixels. This method fails in the same situations as the inside-outside test, resulting in incorrect stencil values when the camera is in shadow or if a shadow volume is clipped against the near plane.
  • 25. 24 CHAPTER 2. REAL-TIME SHADOW RENDERING 2.4.3.2 Depth fail or z-fail The depth fail method solves the problems associated with the depth pass method by inverting the stencil count scheme so that volumes behind the object belonging to the given pixel are counted, (for these polygons the depth test will fail during rendering). To accomplish this successfully the rear end of shadow volumes have to be capped. This method is also known as Carmack’s reverse [Bra03]. 2.4.3.3 Split plane The split plane algorithm [Lai05] uses either the depth pass or the depth fail method for each tile of the rendered image based on comparing the contents of a low-resolution depth buffer and an automatically constructed plane called the split plane. The location of a split plane is calculated from the scene configura- tion. It was shown that this method results in considerable fillrate savings on a software rasterizer, however, its hardware acceleration would require new hard- ware features namely being able per image tile to test against the selected split plane. 2.4.3.4 Exclusive-Or Instead of having the stencil buffer value as a counter it is possible to describe the inside-outside counter ray with a one bit value as being in or outside a shadow volume. Using exclusive-or on a stencil buffer bit when crossing a shadow volume polygon instead of addition and subtraction results in this bit describing whether he pixel is in the shadow or not. This method gives incorrect results when there are intersecting shadow volumes but saves fillrate in case of a scene without intersecting shadows. 2.4.3.5 Stencil value saturation For all stencil methods it is possible that the stencil value saturates if there are not enough bits to describe it, resulting in incorrect stencil values. The value when stencil value saturation occurs depend on the number of bits used to describe the value. With the 8-bit stencil buffers used with current graphics hardware the problem is unlikely. 2.4.4 Performance With increasing geometric complexity shadow volume calculations require more and more computational resources since shadow volumes are usually large, over- lapping polygons and each of them requires stencil updates. Several papers have been published that try to decrease the number of shadow volumes processed based on for example discarding volumes outside a rectangle or a given depth
  • 26. 2.5. HYBRID ALGORITHMS 25 range. The most successful fillrate saving techniques – described in the following section – are hybrid algorithms that use shadow maps to determine shadow edge pixels for shadow volume calculations [Cha04]. 2.5 Hybrid algorithms Hybrid shadow algorithms use the advantages of both shadow maps and shadow volumes. During the execution of these methods shadow maps are used to de- termine what areas in the picture contain shadow edges and shadow volume calculations are executed only for those pixels of the image. With these methods highly accurate shadows can be calculated considerably faster than using only shadow volumes. The first hybrid algorithm [McC00] used the shadow map to identify occluder silhouette edges with an edge detection algorithm. Edge detection algorithms process bitmaps and mark points in a resulting image where the intensity of the image changes sharply. Based on the edges detected shadow volumes are reconstructed and used to render precise shadow edges. Govindaraju et al. [GLY+ 03] first used special techniques to decrease the number of potential occluders and receivers and then rendered shadow edges using both shadow maps and object-space clipping. Chan [Cha04] used edge detection to identify shadow edge pixels from a com- putational mask – a bitmap with information about the location of edge pixels in the image – to calculate only shadow edge pixels on the stencil buffer. 2.6 Soft shadows The previous methods are capable of calculating only shadows of points lights, or hard shadows. If light is emitted from a source with nonzero area, shadow silhouettes become blurry and also significantly harder to render since between the lit and the shadowed areas (umbra) there is an area that is lit by only a part of the light source (penumbra), see Figure 2.6. Most models for generating these soft shadows have been so complex that including them in a graphics pipeline would certainly make them so slow that real-time applications would be impossible. However with the ever continuing acceleration of graphics hardware some models have become feasible. It is possible to extend both shadow maps and shadow volumes to generate soft shadows, each method inheriting the advantages and disadvantages of the original. 2.6.1 Image based soft shadows Image based methods are soft shadow generation procedures that are based on the shadow map algorithm. All of these methods create a shadow map that describe
  • 27. 26 CHAPTER 2. REAL-TIME SHADOW RENDERING Figure 2.6: Soft shadows [HLHS03] shadowed areas and are used to calculate the color of the shadowed surfaces. Shadow maps for soft shadows contain several values per shadow map pixel to be able to represent continuous transitions from shadowed to non-shadowed areas. The following sections describe the main image base methods. 2.6.1.1 Combination of several point-based shadow maps If hard shadows are calculated using sample points of the extended light source and the resulting shadows combined to a final image an approximation can be achieved that models real soft shadows accurately if the number of samples is sufficiently high. This method requires that the shadow algorithm is executed as many times as samples are required resulting in very high rendering times, [HLHS03] (Figure 2.7). 2.6.1.2 Layered Shadow Maps The layered method is an extension of the previous method. Instead of calculating a shadow map for each object, one layered attenuation map is calculated for the whole image that contains all objects that are visible from at least one light sample point and describes the distance to the light source and the percentage of sample points seeing this object. The shadow map in this method contains as many layers of information as many samples are taken from the light source. The layers describe blocking entities between the light source and the receivers. During rendering the rendered pixel is checked if it is in the layered shadow map,
  • 28. 2.6. SOFT SHADOWS 27 Figure 2.7: Combining several hard shadows. Left: 4 samples. Right: 1024 samples [HLHS03] and if it is there its color is modulated according to the visibility percentage value in the map. Since the algorithm has to run as many times as the number of samples with this method also, rendering times are not acceptable for real-time applications on current hardware. [KM99] 2.6.1.3 Quantitative Information in the Shadow Map Heidrich [HBS00] extended the shadow map method by calculating the percentage value of the light source visible for each shadow map pixel. Compared to the layered shadow map method, there is only one shadow map in this approach, and this method does not involve the depth information in the shadow map calculation. Therefore this method calculates less accurate but faster computable shadows. The original algorithm worked only for linear light sources but later it was extended by Ying [YTD02] to be usable for polygonal light sources. 2.6.1.4 Single Sample Soft Shadows With the single sample soft shadow method a standard shadow map is calculated using a point light source and during rendering the depth map is analyzed to identify occluder and receiver distances. These distances are used then to cal- culate a light attenuation coefficient that help determine the level of shadowing at a given location thus the size of inner and outer penumbra (see Figure 2.8). This way effects of area light sources are approximated using a point light. The method approximates soft shadow effects nicely considering the required amount of resources.
  • 29. 28 CHAPTER 2. REAL-TIME SHADOW RENDERING Figure 2.8: Inner and outer penumbra calculated from point light. [HLHS03] 2.6.1.5 Convolution technique Soler et al. [SS98] observed that soft shadow calculations can be approximated with a convolution operation on the image of the light source and the image of the occluder. The technique use the fast Fourier transform to efficiently convolute an image of the light source with an image of the blocker, and the resulting blurred blocker image is projected onto receiver objects. This method works well with arbitrary shaped light sources and produces high quality soft shadows at a limited operation complexity. 2.6.2 Geometry based soft shadows Geometry based soft shadow methods use modifications of the shadow volume algorithm to calculate soft shadows. Similarly to the shadow map based soft shad- ows it is possible to combine several shadow volumes taken from point samples of the light source. Other approaches extend the shadow volume using a specific heuristic: Plateaus or Smoothies. It is also possible to compute a penumbra vol- ume for each edge of the shadow silhouette. In the following subsections these methods will be described in more detail. 2.6.2.1 Combining several hard shadows Similarly to the combination of images calculated with the shadow map algorithm it is possible to average images calculated with shadow volumes. The number of samples required for good quality images are quite high requiring multiple times the resources of the original shadow volume algorithm. 2.6.2.2 Soft Planar Shadows Using Plateaus With the plateaus method an attenuation map is generated using shadow vol- umes. The silhouette vertices of the objects are transformed into cones and
  • 30. 2.6. SOFT SHADOWS 29 surfaces and the resulting volumes are projected onto the receiver and penumbra volumes are colored using Gouraud shading. The method is limited to planar receivers, it assumes a spherical light source and computes outer penumbra (see Figure 2.8) only resulting in limited accuracy. 2.6.2.3 Smoothies Smoothies are planar surfaces calculated from object silhouettes that describe soft shadow edges. The method can be fully accelerated with graphics hardware. Similarly to the plateaus method smoothies compute only the outer penumbra. Occluders therefore always project an umbra, even with large light sources with respect to the occluders [Cha04]. 2.6.2.4 Penumbra wedges Another soft shadow method that is designed to be computed with the use of shader programs is penumbra wedges. For each silhouette edge a silhouette wedge is calculated. In a first pass normal shadow volumes are used to render a hard shadow. Then the wedges are used to calculate the percentage of the light source visible for soft shadow pixels. The resulting visibility buffer is used when ren- dering the scene with standard methods. Penumbra wedges result in physically exact shadow in most cases[AAM03]. Compared to smoothies the difference is in the algorithm of calculating the soft shadow (or transparency) values of the penumbra: smoothies calculate these values by extending hard shadows with pla- nar surfaces while penumbra wedges calculate these values by projecting the light source shape to the shadowed surface. Since the penumbra shape is more exact with penumbra wedges this method provides even more exact results.
  • 31. Chapter 3 Hardware accelerated computer graphics 3.1 Introduction The computational power of computer hardware has been accelerating exponen- tially since the invention of the integrated chip. In order to achieve faster and better looking three-dimensional graphics applications graphics cards were ex- tended in the middle of the 1990s by a dedicated processing unit: the Graphical Processing Unit (GPU). Before that graphics applications were handled entirely in the main processor unit. Graphical Processing Units and their programming interfaces have been continuously evolving. Three-dimensional image generation is divided into separate stages generally described by Foley et al. [FvDFH96] which is usually referred to as the graphics pipeline. Applications using the graphics hardware usually access its features via a standardized application pro- gramming interfaces (API). The APIs provide uniform methods to access various graphics hardware thus supporting fast application development. Current widely accepted APIs for graphics hardware are DirectX and OpenGL. Each API has its own implementation of the graphics pipeline. 3.2 Modern graphics pipelines Data structures that describe three-dimensional scenes are fed to the graphics pipeline as its input and a two-dimensional view of the described scene is pre- sented as output. The stages of data processing can be summarized as follows: 31
  • 32. 32 CHAPTER 3. HARDWARE ACCELERATED COMPUTER GRAPHICS 3.2.1 Modeling transformations The modeling transformations put objects to their designated place in the scene with the correct scaling and rotations. Objects generally are described by trian- gles with the coordinates of their vertices. The output of the modeling transfor- mation are the same objects but with their vertex coordinates (spatial positions of the points that define the triangles) changed according to the required trans- formations. 3.2.2 Illumination The illumination phase calculates vertex colors according to the given lighting model. Current hardware graphics pipelines calculate lighting for vertices only, per pixel lighting values can be calculated with approximations by pixel shader programs in the rasterization phase. 3.2.3 Viewing transformations The viewing transformations changes the position, rotation and scale of the ob- jects as they would be perceived from the camera position. This also includes perspective or orthographic projection according to the given settings. 3.2.4 Clipping The clipping phase removes objects hidden from the camera from the process in order to accelerate the remaining stages. This stage may include view frustrum culling, face culling or hidden object removal from the view frustrum. 3.2.5 Projection Projection transforms all geometry to their 2D position on the camera screen. 3.2.6 Rasterization The rasterization stage calculates the color of every pixel in the image based on the previously calculated geometry and lighting information. 3.2.7 Display The display phase deals with putting the rendered image to its final place (coor- dinates) on the display device - for example a window - and also handling possible low level issues of displaying the image like handling the set-up and special re- quirements of the display device.
  • 33. 3.3. SHADER PROGRAMS 33 Figure 3.1: A possible hardware rendering architecture including vertex and pixel shaders. [SL04] 3.3 Shader programs Early 3D accelerator hardware functionality was fixed, using a simple subset of the graphics hardware available today. The possibilities ended with setting parameters of the predefined graphics pipeline and specifying what textures to use on the geometry. Later as graphics cards became more and more sophisticated arbitrary (though limited length) programmability became possible for some of the pipeline stages. Several different languages were proposed for programming these aspects of the graphics pipeline. Currently programmers can choose from HLSL (High Level Shader Language), Cg (C for Graphics), or GLSL (OpenGL Shading Language) as higher level languages or directly use the assembly-like low-level instruction sets of the pipeline APIs. 3.3.1 Vertex shaders Current parts of the graphics pipeline that deal with geometry modifications are programmable with vertex shader programs. Modeling, viewing and pro- jection transformations along with any special geometry modification are freely programmable today.
  • 34. 34 CHAPTER 3. HARDWARE ACCELERATED COMPUTER GRAPHICS Vertex shader programs specify how graphics cards process geometry data. Having free access to the geometry processing of the pipeline enables the pro- grammer to achieve visual effects that were not possible before. These effects in- clude procedural geometry, animation key-frame interpolation, particle systems, real-time modification of the perspective view and special lighting models. 3.3.2 Pixel shaders Vertex shader functionality was in a limited way possible with fixed pipelines. Pixel shaders however enable possibilities that were previously unimaginable in real-time applications. The word shader originates from the non-real time com- puter graphics industry. In ray-tracing applications shaders are separate software modules that are passed all available geometrical and environmental information about a pixel and they return the color of that pixel. In order to achieve a similar level of realism as ray tracing graphics hardware adopted the same functionality with the name of pixel shaders. Pixel shaders are programs that calculate the color for each displayed pixel. Pixel shaders enable per-pixel non-standardized lighting effects that bring a level of realism to real-time graphics that was not possible before. With pixel shaders non-photorealistic rendering also entered the real-time rendering world. Effects like procedural textures, volumetric effects, anisotropic lighting, among others, became possible. 3.4 Shader developing environments Early shader development required extreme low-level understanding of the graph- ics pipeline APIs. Shader programs were written in an assembly like shader lan- guage that usually were compiled using simple tools provided with the APIs. Later higher level, C-like programming languages appeared and also the first In- tegrated Development Environments were developed by major hardware vendors. Also more visual tools appeared on the marked enabling shader development and experimentation by connecting shader program elements visually as boxes on a graphical interface. Both of the major vendors ATI and NVIDIA has its own shader developer IDE, that works best with their own products. With these tools the user can focus on shader development only, the rest of the pipeline API set up is done by the IDE. It also accelerates work with shaders as the components of shaders are visually presented and can be changed easily. In the following subchapters the IDEs of the market leaders in the graphics card industry will be described.
  • 35. 3.4. SHADER DEVELOPING ENVIRONMENTS 35 Figure 3.2: Screenshot of ATI’s Rendermonkey shader development environment. 3.4.1 Rendermonkey The shader developer IDE of ATI is Rendermonkey [RMD07]. Rendermonkey version 1.6 was used for the example shader development in this thesis. It sup- ports all shader models provided with DirectX 9.0c (including HLSL) and for shading using OpenGL GLSL shading language. The motivation for developing RenderMonkey is given by ATI as follows [RMD07]: • A powerful programmer’s development environment for creating shaders. • A standard delivery mechanism to facilitate the sharing of shaders amongst developers. • A flexible, extensible framework that supports the integration of custom components and provides the basis for future tools development. • An environment where not just programmers but artists and game designers can work to create ”mind-blowing” special effects.
  • 36. 36 CHAPTER 3. HARDWARE ACCELERATED COMPUTER GRAPHICS • A tool that can easily be customized and integrated into a developer’s regular workflow. 3.4.2 FX Composer FX Composer is the shader developer IDE developed by NVidia [FXC07]. FX Composer empowers developers to create high performance shaders in an in- tegrated development environment with a real-time preview and optimization features. FX Composer was designed with the goal of making shader develop- ment and optimization easier for programmers while providing an intuitive GUI for artists customizing shaders for a particular scene. FX Composer supports all the standard features one would expect in an integrated development environment for high performance shaders [FXC07]: 1. Creating shaders in a high powered development environment: • Sophisticated text editing with intellisense (auto-complete) and syntax highlighting • Work directly with HLSL .FX files, creating multiple techniques and passes • Use of the .FX files created with FX Composer directly in applications • Convenient, artist-friendly graphical editing of shader properties • Supports Microsoft DirectX standard HLSL semantics and annota- tions • Support for all DirectX 9.0 shader profiles • Developing shaders on models with lighting and animation 2. Debugging shaders with visual shader debugging features: • Interactive compiler helps finding and fixing problems • Visible preview of source and intermediate textures targets • Interactive jump-to-error feature helping to locate problems quickly 3. Tuning shader performance with advanced analysis and optimization: • Enables performance tuning workflow for vertex and pixel shaders • Simulates performance for the entire family of GeForce FX GPUs • Capture of pre-calculated functions to texture look-up table • Provides empirical performance metrics such as GPU cycle count, reg- ister usage, utilization rating, and FPS. • Optimization hints notifying of performance bottlenecks
  • 37. 3.4. SHADER DEVELOPING ENVIRONMENTS 37 Both Rendermonkey and FX Composer provide a completely set up 3D engine for displaying shaders so developers can focus on shader development without working through the tedious graphics pipeline setup and the quite steep learning curve of fully programming the pipeline APIs.
  • 38. Chapter 4 A real-time NPR shadow rendering example In this chapter the development of a non-photorealistic shadow rendering shader will be presented. The set of possible NPR techniques is vast, compared to photo realistic rendering - the extremely various visual world as we see it. The basic details and possibilities of 3D rendering and shader programming will not be described in this thesis. Instead, these can be located for example in the book Shaders for Game Programmers and Artists by Sebastien St-Laurent [SL04]. Using shader programs running in graphics hardware for our example effect, render times well within human perception can be achieved with current desktop computers. Therefore our example will be a real-time rendered NPR shadow effect. 4.1 Concept The NPR effect chosen is a hatching shader with the extension of presenting the shadows in the scene with a different hatching pattern than the object hatching just like an artist drawing the scene would present shadows with another direction of hatching lines. The geometry the effect will be presented on is a trefoil knot object on a disc. The scene will be lit by a point light from above therefore the trefoil knot will cast its shadow on the disc. To implement the effect the following three rendering passes will be required: • Object rendering pass: an NPR hatching pass 39
  • 39. 40 CHAPTER 4. A REAL-TIME NPR SHADOW RENDERING EXAMPLE Figure 4.1: Screenshots of the reused Rendermonkey example shaders NPR/Hatch (left) and Shadows/Shadowmapping (right). • Depth map rendering pass: rendering the shadow map from the point of view of the light source. • Shadowed area rendering pass: using the projected shadow map to decide if a pixel is in shadow. Unshadowed areas will be rendered with normal hatching and shadowed areas with normal hatching combined with a dif- ferent angle hatching pattern. The first two passes can be reused from Rendermonkey example shaders called NPR/Hatch and Shadows/Shadowmapping. The rest of the chapter will focus on the development of the third pass shaders. 4.2 Developing vertex and pixel shaders 4.2.1 A general frame application using shader programs When creating a new pass in Rendermonkey four items are created automatically in the workspace tree: • Model • Vertex shader • Pixel shader • Stream mapping The model can be any geometry in .3ds, .nms, .obj, or .x data formats [SL04]. Let us choose the disc.3ds object from the Rendermonkey example objects as the platform where the shadow of other objects will fall on. The file contains geometry information about the disc: location of its points in space described by
  • 40. 4.2. DEVELOPING VERTEX AND PIXEL SHADERS 41 their spatial coordinates (x,y,z), and what points are connected by the surfaces that define the object. 4.2.1.1 Skeleton vertex shader A skeleton vertex shader is created with the new pass: float4x4 m a t V i e w P r o j e c t i o n ; struct VS INPUT { f l o a t 4 P o s i t i o n : POSITION0 ; }; struct VS OUTPUT { f l o a t 4 P o s i t i o n : POSITION0 ; }; VS OUTPUT vs ma in ( VS INPUT I n p u t ) { VS OUTPUT Output ; Output . P o s i t i o n = mul( m a tV ie w P ro je c t i on , I n p u t . P o s i t i o n ) ; return ( Output ) ; } The skeleton vertex shader consists of the declaration of the input and output data structures and the main function that applies the projection matrix to the input geometry. 4.2.1.2 Skeleton pixel shader A skeleton pixel shader is also created: f l o a t 4 ps main ( ) : COLOR0 { return ( f l o a t 4 ( 1 . 0 f , 0 . 0 f , 0 . 0 f , 1 . 0 f ) ) ; } The pixel shader returns constant red color for all pixels that the shader is run on. (The float values in the returned data structures stand for the red, green, blue and alpha values in the range of 0 to 1.) Using a trefoil knot object the assembled 3 pass shader will result the image presented in figure 4.2. The disc object is intersecting the trefoil knot object that can be fixed in the vertex shader by offsetting the geometry with constants. The values of the required constants can be easily determined since RenderMonkey enables the user to dynamically adjust parameters during preview rendering. Two variables are added and the value can be determined by adjusting them while the shaders are running: float posShiftY ; float objectScale ;
  • 41. 42 CHAPTER 4. A REAL-TIME NPR SHADOW RENDERING EXAMPLE Figure 4.2: Disc object intersecting the trefoil knot object (left), fixed disc scale and position (right). // s c a l i n g model In . Pos . xyz = o b j e c t S c a l e ∗ I n . Pos . xyz ; // 1 . 6 i s f i n e // s h i f t i n g p l a t f o r m model under t h e o b j e c t I n p u t . P o s i t i o n . y −= p o s S h i f t Y ; // 40 i s f i n e 4.2.1.3 Stream mapping Stream mapping defines in what form geometry information is passed to the vertex shaders. The data is passed in data structures. It is possible to select in Rendermonkey what information to pass to which hardware registers and how the data is represented. The default stream mapping contains position information only, passed in a data type that holds 3 float values (FLOAT3). 4.2.2 Hatching To add a hatching effect to the disc the same code can be added as for the hatched object. 4.2.2.1 Vertex shader In the vertex shader, based on a diffuse lighting values, weights are calculated for each of the hatching textures. There are 6 hatching textures with different shades. The calculated weights are passed to the pixel shader. To be able to calculate the diffuse lighting a normal vector of the geometry is required as an input parameter of the vertex shader. Also to be able to address the hatching texture pixels in the pixel shader, a texture input parameter is also added to the shaders and the unmodified value is passed to the pixel shader. float posShiftY ; float objectScale ; float t i m e 0 X : r e g i s t e r ( c11 ) ;
  • 42. 4.2. DEVELOPING VERTEX AND PIXEL SHADERS 43 float4 l i g h t P o s : r e g i s t e r ( c4 ) ; float4 lightDir ; float4x4 matViewProjection ; float4x4 p r o j m a t r i x : r e g i s t e r ( c6 ) ; struct VS INPUT { float4 Pos : POSITION0 ; float3 Normal : NORMAL0; float2 TexCoord : TEXCOORD0; }; struct VS OUTPUT { f l o a t 4 Pos : POSITION0 ; float3 lightVec : : TEXCOORD0; // t o be used l a t e r f l o a t 4 shadowCrd : : TEXCOORD1; // t o be used l a t e r f l o a t 2 TexCoord : TEXCOORD2; f l o a t 3 HatchWeights0 : TEXCOORD3; f l o a t 3 HatchWeights1 : TEXCOORD4; }; VS OUTPUT vs ma in ( VS INPUT In ) { VS OUTPUT Out ; // F l i p , s c a l e and t r a n s l a t e our model t o s u i t our s c e n e // In r e a l a p p l i c a t i o n s , t h i s work s h o u l d n o r m a l l y be // done a t l o a d time , a l t e r n a t i v e l y i f animati on i s d e s i r e d , // be a l t e r e d by a w o r l d m a t r i x . I n . Pos . xyz = o b j e c t S c a l e ∗ In . Pos . xyz ; // was 1 . 6 I n . Pos . y −= p o s S h i f t Y ; // was 40 // I n p u t . Normal = I n p u t . Normal . x y z ; Out . Pos = mul( m a t V ie wP ro je c ti on , I n . Pos ) ; Out . TexCoord = In . TexCoord ; // Animate t h e l i g h t p o s i t i o n . // Comment o u t t h i s code t o u s e a s t a t i c l i g h t . // In r e a l a p p l i c a t i o n s t h i s work i s b e t t e r done on // t h e CPU as i t ’ s c o n s t a n t f o r t h e w h o l e s c e n e . float3 lightPos ; l i g h t P o s . x = cos ( 1 . 3 2 1 ∗ t i m e 0 X ) ; l i g h t P o s . z = sin ( 0 . 9 2 3 ∗ time 0 X ) ; l i g h t P o s . xz = 100 ∗ normalize ( l i g h t P o s . xz ) ; lightPos . y = 100; // C r e a t e v i e w v e c t o r s f o r t h e l i g h t , l o o k i n g a t ( 0 , 0 , 0 ) // In r e a l a p p l i c a t i o n s t h i s work i s b e t t e r done on // t h e CPU as i t ’ s c o n s t a n t f o r t h e w h o l e s c e n e . f l o a t 3 d i r Z = −normalize ( l i g h t P o s ) ; f l o a t 3 up = f l o a t 3 ( 0 , 0 , 1 ) ; f l o a t 3 dirX = cross ( up , d i r Z ) ; f l o a t 3 dirY = cross ( d irZ , dirX ) ; // Transform i n t o l i g h t ’ s v i e w s p a c e . // In r e a l a p p l i c a t i o n s we would be b e t t e r o f f u s i n g a 4 x4 // matrix instead , but f o r t h i s shader i t ’ s cheaper to // j u s t t r a n s p o s e and r o t a t e i n t o l i g h t ’ s v i e w s p a c e .
  • 43. 44 CHAPTER 4. A REAL-TIME NPR SHADOW RENDERING EXAMPLE f l o a t 4 pos ; I n . Pos . xyz −= l i g h t P o s ; pos . x = dot ( dirX , In . Pos ) ; pos . y = dot ( dirY , In . Pos ) ; pos . z = dot ( d irZ , I n . Pos ) ; pos . w = 1 ; f l o a t 3 normalW = −normalize ( pos ) ; // C a l c u l a t e d i f f u s e l i g h t i n g float d i f f u s e = min ( 1 . 0 ,max( 0 , dot(− l i g h t D i r , normalW ) ) ) ; diffuse = diffuse ∗ diffuse ; diffuse = diffuse ∗ diffuse ; // C a l c u l a t e t e x t u r e w e i g h t s float hatchFactor = d i f f u s e ∗ 6 . 0 ; float3 weight0 = 0 . 0 ; float3 weight1 = 0 . 0 ; i f ( h a t c h F a c t o r >5.0) { weight0 . x = 1 . 0 ; } // End i f e l s e i f ( h a t c h F a c t o r >4.0) { weight0 . x = 1.0 − ( 5 . 0 − hatchFactor ) ; weight0 . y = 1.0 − weight0 . x ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >3.0) { weight0 . y = 1.0 − ( 4 . 0 − hatchFactor ) ; weight0 . z = 1.0 − weight0 . y ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >2.0) { weight0 . z = 1.0 − ( 3 . 0 − hatchFactor ) ; weight1 . x = 1.0 − weight0 . z ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >1.0) { weight1 . x = 1.0 − ( 2 . 0 − hatchFactor ) ; weight1 . y = 1.0 − weight1 . x ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >0.0) { weight1 . y = 1.0 − ( 1 . 0 − hatchFactor ) ; weight1 . z = 1.0 − weight1 . y ; } // End e l s e i f Out . HatchWeights0 = w e i g h t 0 ; Out . HatchWeights1 = w e i g h t 1 ; return Out ; }
  • 44. 4.2. DEVELOPING VERTEX AND PIXEL SHADERS 45 The diffuse lighting value is calculated based on the direction of the light and the surface normal in the diffuse variable. Then for each texture sample a weight value is calculated based on the diffuse value and stored in the weight parameters. The weight parameters are the output of the vertex shaders and are used in the pixel shader to generate the colors of pixels. 4.2.2.2 Pixel shader The pixel shader takes the weighted samples from each texture and returns their sum: sampler Hatch0 ; sampler Hatch1 ; sampler Hatch2 ; sampler Hatch3 ; sampler Hatch4 ; sampler Hatch5 ; struct PS INPUT { float2 TexCoord : TEXCOORD0; float3 HatchWeights0 : TEXCOORD1; float3 HatchWeights1 : TEXCOORD2; }; struct PS OUTPUT { float4 Color : COLOR0; }; PS OUTPUT ps main ( PS INPUT In ) { PS OUTPUT Out ; // Sample each t e x t u r e and w e i g h t t h e sample . f l o a t 4 hatchTex0 = tex2D ( Hatch0 , I n . TexCoord ) ∗ I n . HatchWeights0 . x ; f l o a t 4 hatchTex1 = tex2D ( Hatch1 , I n . TexCoord ) ∗ I n . HatchWeights0 . y ; f l o a t 4 hatchTex2 = tex2D ( Hatch2 , I n . TexCoord ) ∗ I n . HatchWeights0 . z ; f l o a t 4 hatchTex3 = tex2D ( Hatch3 , I n . TexCoord ) ∗ I n . HatchWeights1 . x ; f l o a t 4 hatchTex4 = tex2D ( Hatch4 , I n . TexCoord ) ∗ I n . HatchWeights1 . y ; f l o a t 4 hatchTex5 = tex2D ( Hatch5 , I n . TexCoord ) ∗ I n . HatchWeights1 . z ; //Sum t h e w e i g h t e d t e x t u r e v a l u e s f l o a t 4 h a t c h C o l o r = hatchTex0 + hatchTex1 + hatchTex2 + hatchTex3 + hatchTex4 + hatchTex5 ; Out . C o l o r = h a t c h C o l o r ; return Out ; }
  • 45. 46 CHAPTER 4. A REAL-TIME NPR SHADOW RENDERING EXAMPLE Figure 4.3: Hatching applied to platform (left), shadow hatched with different texture (right). The pixel shader receives texture coordinates, and texture weights – calculated by the vertex shader – in the PS INPUT data structure. The tex2D function reads pixel color from textures specified by texture coordinates and a texture. To generate the hatching effect 6 hatching textures are used and a weighted sum of all the six textures results in the output color of the pixels. The result can be seen in figure 4.3 on the left. 4.2.2.3 Stream mapping To be able to use the textures in the shader programs a texture coordinate pa- rameter has to be added to the stream map. Also for calculating diffuse lighting a surface normal is added to the map. 4.2.3 Shadow hatching To draw shadowed areas with a different hatching pattern the shadow map have to be taken in use. The projected shadow map values are compared to the distance between the platform geometry and the camera in order to decide whether a pixel is in the shadow. The pixel color is masked by the shadow variable so that shadowed pixels use the differently hatched texture. 4.2.3.1 Vertex shader . . . // p a r t s e x p l a i n e d b e f o r e o m i t t e d // f o r shadow float objectLightDistance ; f l o a t d i s t a n c e S c a l e : r e g i s t e r ( c10 ) ; struct VS INPUT
  • 46. 4.2. DEVELOPING VERTEX AND PIXEL SHADERS 47 { f l o a t 4 Pos : POSITION0 ; f l o a t 3 Normal : NORMAL0; f l o a t 2 TexCoord : TEXCOORD0; }; struct VS OUTPUT { f l o a t 4 Pos : POSITION ; float3 lightVec : TEXCOORD0; f l o a t 4 shadowCrd : TEXCOORD1; f l o a t 2 TexCoord : TEXCOORD2; f l o a t 3 HatchWeights0 : TEXCOORD3; f l o a t 3 HatchWeights1 : TEXCOORD4; }; VS OUTPUT vs ma in ( VS INPUT In ) { . . . // p a r t s e x p l a i n e d b e f o r e o m i t t e d // P r o j e c t i t . For t h i s sample u s i n g t h e normal p r o j e c t i o n // m a t r i x s u f f i c e s , however , r e a l a p p l i c a t i o n s w i l l t y p i c a l l y // u s e a s e p a r a t e p r o j e c t i o n m a t r i x f o r each l i g h t d e p e n d i n g // on i t s p r o p e r t i e s such as FOV and r ange . f l o a t 4 sPos = mul( p r o j m a t r i x , pos ) ; // Use p r o j e c t i v e t e x t u r i n g t o map t h e p o s i t i o n o f each f r a g m e n t // t o i t s c o r r e s p o n d i n g t e x e l i n t h e shadow map . sPos . z += 1 0 ; Out . shadowCrd . x = 0 . 5 ∗ ( sPos . z + sPos . x ) ; Out . shadowCrd . y = 0 . 5 ∗ ( sPos . z − sPos . y ) ; Out . shadowCrd . z = 0 ; Out . shadowCrd . w = sPos . z ; o b j e c t L i g h t D i s t a n c e=l i g h t P o s . z−sPos . z ; . . . // p a r t s e x p l a i n e d b e f o r e o m i t t e d return Out ; } The shadow map is projected to the image seen and the projected shadow map value is passed to the pixel shader in the shadowCrd data structure. 4.2.3.2 Pixel shader f l o a t shadowBias ; float shadowShadeOffset ; sampler Hatch0 ; sampler Hatch1 ; sampler Hatch2 ; sampler Hatch3 ; sampler Hatch4 ; sampler Hatch5 ; sampler Hatch20 ;
  • 47. 48 CHAPTER 4. A REAL-TIME NPR SHADOW RENDERING EXAMPLE sampler Hatch21 ; sampler Hatch22 ; sampler Hatch23 ; sampler Hatch24 ; sampler Hatch25 ; sampler ShadowMap ; struct PS OUTPUT { float4 Color : COLOR0; }; PS OUTPUT ps main ( f l o a t 3 lightVec : TEXCOORD0, float4 shadowCrd : TEXCOORD1, float2 TexCoord : TEXCOORD2, float3 HatchWeights0 : TEXCOORD3, float3 HatchWeights1 : TEXCOORD4 ) : COLOR { PS OUTPUT Out ; // R a d i a l d i s t a n c e f l o a t depth = length ( l i g h t V e c ) ; // N o r m a l i z e s l i g h t v e c t o r l i g h t V e c /= depth ; // The d e p t h o f t h e f r a g m e n t c l o s e s t t o t h e l i g h t f l o a t shadowMap = tex2Dproj ( ShadowMap , shadowCrd ) ; // I f t h e d e p t h i s l a r g e r than t h e s t o r e d depth , t h i s f r a g m e n t // i s n o t t h e c l o s e s t t o t h e l i g h t , t h a t i s we a r e i n shadow . // Otherwise , we ’ r e l i t . Add a b i a s t o a v o i d p r e c i s i o n i s s u e s . f l o a t shadow = ( depth < shadowMap + shadowBias ) ; f l o a t 4 hatchTex0 = tex2D ( Hatch0 , TexCoord ) ∗ HatchWeights0 . x ; f l o a t 4 hatchTex1 = tex2D ( Hatch1 , TexCoord ) ∗ HatchWeights0 . y ; f l o a t 4 hatchTex2 = tex2D ( Hatch2 , TexCoord ) ∗ HatchWeights0 . z ; f l o a t 4 hatchTex3 = tex2D ( Hatch3 , TexCoord ) ∗ HatchWeights1 . x ; f l o a t 4 hatchTex4 = tex2D ( Hatch4 , TexCoord ) ∗ HatchWeights1 . y ; f l o a t 4 hatchTex5 = tex2D ( Hatch5 , TexCoord ) ∗ HatchWeights1 . z ; f l o a t 4 h a t c h C o l o r = hatchTex0 + hatchTex1 + hatchTex2 + hatchTex3 + hatchTex4 + hatchTex5 ; f l o a t 4 shadowHatchTex0 = tex2D ( Hatch20 , TexCoord ) ∗ ( HatchWeights0 . x−s h a d o w S h a d e O f f s e t ) ; f l o a t 4 shadowHatchTex1 = tex2D ( Hatch21 , TexCoord ) ∗ ( HatchWeights0 . y−s h a d o w S h a d e O f f s e t ) ; f l o a t 4 shadowHatchTex2 = tex2D ( Hatch22 , TexCoord ) ∗ ( HatchWeights0 . z−s h a d o w S h a d e O f f s e t ) ; f l o a t 4 shadowHatchTex3 = tex2D ( Hatch23 , TexCoord ) ∗ ( HatchWeights1 . x−s h a d o w S h a d e O f f s e t ) ; f l o a t 4 shadowHatchTex4 = tex2D ( Hatch24 , TexCoord ) ∗ ( HatchWeights1 . y−s h a d o w S h a d e O f f s e t ) ; f l o a t 4 shadowHatchTex5 = tex2D ( Hatch25 , TexCoord ) ∗ ( HatchWeights1 . z−s h a d o w S h a d e O f f s e t ) ; f l o a t 4 shadowHatchColor = shadowHatchTex0 +
  • 48. 4.2. DEVELOPING VERTEX AND PIXEL SHADERS 49 Figure 4.4: Distance based shading of a transparent shadow texture. shadowHatchTex1 + shadowHatchTex2 + shadowHatchTex3 + shadowHatchTex4 + shadowHatchTex5 ; Out . C o l o r = shadow ∗ h a t c h C o l o r + (1−shadow ) ∗ shadowHatchColor ; return Out ; } The pixel shader decides whether the current pixel is in the shadow. If it is not in the shadow it uses the texture described in the previous section. If the pixel is in the shadow a different texture (with a hatching rotated 45 degrees) is used. The result can be seen in Figure 4.3 to the right. 4.2.4 Distance based shading of a transparent shadow texture Though the result of the previous section is already acceptable, the shader can be refined further. The textures of the shadowed areas could be mixed with the base texture and the shadow shade would look much nicer if it would be lighter with growing distance from the object. The texture mixing can be achieved easily by calculating a weighted average of the diffuse and shadow textures. To achieve a lighter shadow shade further from the object the w coordinate of the shadowCrd parameter can be used. It holds the z value of the projected geometry which is an increasing value as we get away from the object. This value can be scaled to be used for weighting the shadow value. Based on the weighted value, texture weights are calculated the same way as the diffuse lighting value. The source code of the shaders can be found in the Appendix and the result in Figure 4.4.
  • 49. Chapter 5 Conclusion and future work 5.1 Conclusion The thesis set out to show that current real-time non-photorealistic rendering methods and shadow rendering techniques can be combined to result in more interesting and complete images. The present status of both fields was summarized and the development of a possible real-time non-photorealistic rendering shader program was presented. It was shown in the thesis that NPR shadows can be rendered easily with the use of existing methods resulting in visually more appealing shadow renditions. Current video hardware is quite capable of calculating real-time NPR shadows opening up an infinitely various area of computer graphics. Hopefully more and more applications will take use of this new possibility. 5.2 Future work Besides merely combining existing NPR methods with shadow rendering possi- bilities – a huge area of possible research already – changing the shape of the shadow to apply special artistic effects algorithmically could be one extremely interesting area of study. Numerous films play with the idea of objects casting a different shape of shadow than the observer would expect to add dramatic effect to a scene. Besides the exploration of achieving this effect algorithmically for artistic purposes the idea of adding information to shadow shapes or colors could be generalized and also applied to other NPR fields like scientific or technical illustrations. 51
  • 50. Bibliography [AAM03] Ulf Assarson and Tomas Akenine-M¨ller. A geometry-based soft shadow o volume algorithm using graphics hardware. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2003), 2003. [cited at p. 29] [Bra03] Stefan Brabec. Shadow Techniques for Interactive and Real-Time applica- tions. PhD thesis, Max-Planck-Institut fur Informatik Saarbrucken, Ger- many, 2003. [cited at p. 19, 20, 21, 23, 24, 68] [Cav07] The chauvet-pont-d’arc cave website, (http://www.culture.gouv.fr/ culture/arcnat/chauvet/en/), 2007. [cited at p. 5] [Cha04] Eric Chan. Efficient shadow algorithms on graphics hardware. Master’s thesis, Massachusetts Institute of Technology, 2004. [cited at p. 19, 20, 21, 22, 25, 29, 68] [Cro77] Franklin C. Crow. Shadow algorithms for computer graphics. In Proceedings of ACM SIGGRAPH, pages 242–248. ACM Press, 1977. [cited at p. 21] [Die01] Sim Dietrich. Shadow techniques. Nvidia Corporation, 2001. [cited at p. 20] [FS94] Adam Finkelstein and David H. Salesin. Multiresolution curves. Computer Graphics, 28(Annual Conference Series):261–268, 1994. [cited at p. 10] [FvDFH96] James D. Foley, Andries van Dam, Steven K. Feiner, and John F. Hughes. Computer Graphics - Principles and Practice. The Systems Programming Series. Addison-Wesley, second edition, 1996. [cited at p. 31] [FXC07] Nvidia’s fx composer webpage (http://developer.nvidia.com/object/ fx composer home.html), 2007. [cited at p. 36] [Gas04] Bamber Gascoigne. How to Identify Prints. Thames and Hudson, 2nd edi- tion, 2004. [cited at p. 12] [GLY+ 03] Naga K. Govindaraju, Brandon Lloyd, Sung-Eui Yoon, Avneesh Sud, and Dinesh Manocha. Interactive shadow generation in complex environments. In ACM Transactions on Graphics (TOG), 2003. [cited at p. 25] 53
  • 51. 54 BIBLIOGRAPHY [HBS00] Wolfgang Heidrich, Stefan Brabec, and Hans-Peter Seidel. Soft shadow maps for linear lights high-quality. In 11th Eurographics Workshop on Rendering, pages 269–280, 2000. [cited at p. 27] [Hei91] Tim Heidmann. Real shadows real time. In IRIS Universe, 1991. [cited at p. 23] [HF05] Hugh Honour and John Fleming. A World History of Art. Laurence King Publishing 7Rev Ed edition, 2005. [cited at p. 5] [HLHS03] J.-M. Hasenfratz, M. Lapierre, N. Holzschuch, and F.X. Sillion. A survey of real-time soft shadows algorithms. In Eurographics 2003, 2003. [cited at p. 26, 27, 28, 68] [KM99] Brett Keating and Nelson Max. Shadow penumbras for complex objects by depth-dependent filtering of multi-layer depth images. pages 197–212, 1999. [cited at p. 27] [Lai05] Samuli Laine. Split-plane shadow volumes. In Graphics Hardware, 2005. [cited at p. 24] [McC00] Michael D. McCool. Shadow volume reconstruction from depth maps. In ACM Transactions on Graphics (TOG), 2000. [cited at p. 25] [PHWF01] Emil Praun, Hugues Hoppe, Matthew Webb, and Adam Finkelstein. Real- time hatching. In Eugene Fiume, editor, SIGGRAPH 2001, Computer Graphics Proceedings, pages 579–584, 2001. [cited at p. 12, 68] [REV07] Re:vision effects website (http://www.revisionfx.com/company/about/), 2007. [cited at p. 13, 68] [RMD07] Ati rendermonkey documentation, (http://ati.amd.com/developer/ rendermonkey/downloads.html), 2007. [cited at p. 35] [SL04] Sebastien St-Laurent. Shaders for Game Programmers and Artists. Course Technology PTR, 2004. [cited at p. 33, 39, 40, 69] [SS98] Cyril Soler and Fran¸ois Sillion. Fast calculation of soft shadow textures c using convolution. In Computer Graphics Proceedings, pages 321–332, Jul 1998. Annual Con. [cited at p. 28] [SS02] Thomas Strothotte and Stefan Schlechtweg. Non-Photorealistic Computer Graphics: Modeling, Rendering and Animation. Morgan Kaufmann, 2002. [cited at p. 9, 10, 11, 12, 68] [WFH+ 97] Daniel N. Wood, Adam Finkelstein, John F. Hughes, Craig E. Thayer, and David H. Salesin. Multiperspective panoramas for cel animation. Computer Graphics, 31(Annual Conference Series):243–250, 1997. [cited at p. 8] [Wil78] Lance Williams. Casting curved shadows on curved surfaces. In Proceedings of ACM SIGGRAPH, pages 270–274. ACM Press, 1978. [cited at p. 18] [YTD02] Zhengming Ying, Min Tang, and Jinxiang Dong. Soft shadow maps for area light by area approximation. In 10th Pacific Conference on Computer Graphics and Applications, pages 442–443. IEEE, 2002. [cited at p. 27]
  • 52. 55 [Zan98] Hansong Zang. Forward shadow mapping. In Eurographics Rendering Work- shop 1998, pages 131–138, 1998. [cited at p. 18]
  • 53. Appendices 57
  • 54. Appendix A NPR shadow pass vertex shader source code float4 l i g h t D i r ; float objectLightDistance ; float4x4 v i e w m a t r i x ; float4x4 v i e w p r o j m a t r i x : r e g i s t e r ( c0 ) ; f l o a t 4 l i g h t P o s : r e g i s t e r ( c4 ) ; f l o a t 4 v i e w p o s i t i o n : r e g i s t e r ( c5 ) ; float4x4 p r o j m a t r i x : r e g i s t e r ( c6 ) ; f l o a t d i s t a n c e S c a l e : r e g i s t e r ( c10 ) ; f l o a t t i m e 0 X : r e g i s t e r ( c11 ) ; struct VS OUTPUT { f l o a t 4 Pos : POSITION ; float3 lightVec : TEXCOORD0; f l o a t 4 shadowCrd : TEXCOORD1; f l o a t 2 TexCoord : TEXCOORD2; f l o a t 3 HatchWeights0 : TEXCOORD3; f l o a t 3 HatchWeights1 : TEXCOORD4; f l o a t 3 ShadowHatchWeights0 :TEXCOORD5; f l o a t 3 ShadowHatchWeights1 :TEXCOORD6; }; struct VS INPUT { float4 Pos : POSITION0 ; float3 Normal : NORMAL0; float2 TexCoord : TEXCOORD0; }; VS OUTPUT main (VS INPUT I n ) { VS OUTPUT Out ; // Animate t h e l i g h t p o s i t i o n . // Comment o u t t h i s code t o u s e a s t a t i c light . 59
  • 55. 60 APPENDIX A. NPR SHADOW PASS VERTEX SHADER SOURCE CODE // In r e a l a p p l i c a t i o n s t h i s work i s b e t t e r done on // t h e CPU as i t ’ s c o n s t a n t f o r t h e w h o l e s c e n e . float3 lightPos ; l i g h t P o s . x = cos ( 1 . 3 2 1 ∗ t i m e 0 X ) ; l i g h t P o s . z = sin ( 0 . 9 2 3 ∗ time 0 X ) ; l i g h t P o s . xz = 100 ∗ normalize ( l i g h t P o s . xz ) ; lightPos . y = 100; // F l i p , s c a l e and t r a n s l a t e our model t o s u i t our s c e n e // In r e a l a p p l i c a t i o n s , t h i s work s h o u l d n o r m a l l y be // done a t l o a d time , a l t e r n a t i v e l y i f animatio n i s d e s i r e d , // be a l t e r e d by a w o r l d m a t r i x . I n . Pos . xyz = 1 . 6 ∗ In . Pos . xyz ; I n . Pos . y −= 4 0 ; I n . Normal = I n . Normal . xyz ; Out . Pos = mul( v i e w p r o j m a t r i x , I n . Pos ) ; // World−s p a c e l i g h t i n g Out . l i g h t V e c = d i s t a n c e S c a l e ∗ ( l i g h t P o s − I n . Pos . xyz ) ; // C r e a t e v i e w v e c t o r s f o r t h e l i g h t , l o o k i n g a t ( 0 , 0 , 0 ) // In r e a l a p p l i c a t i o n s t h i s work i s b e t t e r done on // t h e CPU as i t ’ s c o n s t a n t f o r t h e w h o l e s c e n e . f l o a t 3 d i r Z = −normalize ( l i g h t P o s ) ; f l o a t 3 up = f l o a t 3 ( 0 , 0 , 1 ) ; f l o a t 3 dirX = cross ( up , d i r Z ) ; f l o a t 3 dirY = cross ( d irZ , dirX ) ; // Transform i n t o l i g h t ’ s v i e w s p a c e . // In r e a l a p p l i c a t i o n s we would be b e t t e r o f f u s i n g a 4 x4 // m a t r i x i n s t e a d , b u t f o r t h i s s h a d e r i t ’ s c h e a p e r t o // j u s t t r a n s p o s e and r o t a t e i n t o l i g h t ’ s v i e w s p a c e . f l o a t 4 pos ; I n . Pos . xyz −= l i g h t P o s ; pos . x = dot ( dirX , In . Pos ) ; pos . y = dot ( dirY , In . Pos ) ; pos . z = dot ( d irZ , I n . Pos ) ; pos . w = 1 ; // P r o j e c t i t . For t h i s sample u s i n g t h e normal p r o j e c t i o n // m a t r i x s u f f i c e s , however , r e a l a p p l i c a t i o n s w i l l t y p i c a l l y // u s e a s e p a r a t e p r o j e c t i o n m a t r i x f o r each l i g h t d e p e n d i n g // on i t s p r o p e r t i e s such as FOV and range . f l o a t 4 sPos = mul( p r o j m a t r i x , pos ) ; // Use p r o j e c t i v e t e x t u r i n g t o map t h e p o s i t i o n o f each f r a g m e n t // t o i t s c o r r e s p o n d i n g t e x e l i n t h e shadow map . sPos . z += 1 0 ; Out . shadowCrd . x = 0 . 5 ∗ ( sPos . z + sPos . x ) ; Out . shadowCrd . y = 0 . 5 ∗ ( sPos . z − sPos . y ) ; Out . shadowCrd . z = 0 ; Out . shadowCrd . w = sPos . z ; o b j e c t L i g h t D i s t a n c e=l i g h t P o s . z−sPos . z ; Out . TexCoord = I n . TexCoord ; f l o a t 3 normalW = −normalize ( pos ) ; float d i f f u s e = min ( 1 . 0 ,max( 0 , dot(− l i g h t D i r , normalW ) ) ) ;
  • 56. 61 diffuse = diffuse ∗ diffuse ; diffuse = diffuse ∗ diffuse ; float hatchFactor = d i f f u s e ∗ 6 . 0 ; float3 weight0 = 0 . 0 ; float3 weight1 = 0 . 0 ; i f ( h a t c h F a c t o r >5.0) { weight0 . x = 1 . 0 ; } // End i f e l s e i f ( h a t c h F a c t o r >4.0) { weight0 . x = 1.0 − ( 5 . 0 − hatchFactor ) ; weight0 . y = 1.0 − weight0 . x ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >3.0) { weight0 . y = 1.0 − ( 4 . 0 − hatchFactor ) ; weight0 . z = 1.0 − weight0 . y ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >2.0) { weight0 . z = 1.0 − ( 3 . 0 − hatchFactor ) ; weight1 . x = 1.0 − weight0 . z ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >1.0) { weight1 . x = 1.0 − ( 2 . 0 − hatchFactor ) ; weight1 . y = 1.0 − weight1 . x ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >0.0) { weight1 . y = 1.0 − ( 1 . 0 − hatchFactor ) ; weight1 . z = 1.0 − weight1 . y ; } // End e l s e i f Out . HatchWeights0 = w e i g h t 0 ; Out . HatchWeights1 = w e i g h t 1 ; Out . ShadowHatchWeights0 = w e i g h t 0 ; Out . ShadowHatchWeights1 = w e i g h t 1 ; return Out ; }
  • 57. Appendix B NPR shadow pass pixel shader source code float backProjectionCut ; f l o a t shadowBias ; float objectLightDistance ; sampler ShadowMap ; f l o a t shadowDistM ; f l o a t shadowDistD ; float shadowShadeOffset ; sampler Hatch0 ; sampler Hatch1 ; sampler Hatch2 ; sampler Hatch3 ; sampler Hatch4 ; sampler Hatch5 ; sampler Hatch20 ; sampler Hatch21 ; sampler Hatch22 ; sampler Hatch23 ; sampler Hatch24 ; sampler Hatch25 ; struct PS OUTPUT { float4 Color : COLOR0; }; PS OUTPUT main ( float3 lightVec : TEXCOORD0, float4 shadowCrd : TEXCOORD1, float2 TexCoord : TEXCOORD2, float3 HatchWeights0 : TEXCOORD3, 63
  • 58. 64 APPENDIX B. NPR SHADOW PASS PIXEL SHADER SOURCE CODE f l o a t 3 HatchWeights1 : TEXCOORD4, f l o a t 3 ShadowHatchWeights0 : TEXCOORD5, f l o a t 3 ShadowHatchWeights1 : TEXCOORD6 ) : COLOR { PS OUTPUT Out ; // R a d i a l d i s t a n c e f l o a t depth = length ( l i g h t V e c ) ; // N o r m a l i z e s l i g h t v e c t o r l i g h t V e c /= depth ; // The d e p t h o f t h e f r a g m e n t c l o s e s t t o t h e l i g h t f l o a t shadowMap = tex2Dproj ( ShadowMap , shadowCrd ) ; // I f t h e d e p t h i s l a r g e r than t h e s t o r e d depth , t h i s f r a g m e n t // i s n o t t h e c l o s e s t t o t h e l i g h t , t h a t i s we a r e i n shadow . // Otherwise , we ’ r e l i t . Add a b i a s t o a v o i d p r e c i s i o n i s s u e s . f l o a t shadow = ( depth < shadowMap + shadowBias ) ; // Cut back−p r o j e c t i o n , t h a t i s , make s u r e we don ’ t l i t // a n y t h i n g b e h i n d t h e l i g h t . T h e o r e t i c a l l y , you s h o u l d j u s t // c u t a t w = 0 , b u t i n p r a c t i c e you ’ l l have t o c u t a t a // f a i r l y h i g h p o s i t i v e number t o a v o i d p r e c i s i o n i s s u e when // c o o r d i n a t e s a p p r o a c h e s z e r o . shadow ∗= ( shadowCrd . w > b a c k P r o j e c t i o n C u t ) ; float4 hatchTex0 = tex2D ( Hatch0 , TexCoord ) ∗ HatchWeights0 . x ; float4 hatchTex1 = tex2D ( Hatch1 , TexCoord ) ∗ HatchWeights0 . y ; float4 hatchTex2 = tex2D ( Hatch2 , TexCoord ) ∗ HatchWeights0 . z ; float4 hatchTex3 = tex2D ( Hatch3 , TexCoord ) ∗ HatchWeights1 . x ; float4 hatchTex4 = tex2D ( Hatch4 , TexCoord ) ∗ HatchWeights1 . y ; float4 hatchTex5 = tex2D ( Hatch5 , TexCoord ) ∗ HatchWeights1 . z ; f l o a t 4 h a t c h C o l o r = hatchTex0 + hatchTex1 + hatchTex2 + hatchTex3 + hatchTex4 + hatchTex5 ; float hatchFactor ; h a t c h F a c t o r = 6 ∗ ( b a c k P r o j e c t i o n C u t / ( shadowCrd . w−o b j e c t L i g h t D i s t a n c e ) ) ; float3 weight0 = 0 . 0 ; float3 weight1 = 0 . 0 ; i f ( h a t c h F a c t o r >5.0) { weight0 . x = 1 . 0 ; } // End i f e l s e i f ( h a t c h F a c t o r >4.0) { weight0 . x = 1.0 − ( 5 . 0 − hatchFactor ) ; weight0 . y = 1.0 − weight0 . x ; } // End e l s e i f
  • 59. 65 e l s e i f ( h a t c h F a c t o r >3.0) { weight0 . y = 1.0 − ( 4 . 0 − hatchFactor ) ; weight0 . z = 1.0 − weight0 . y ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >2.0) { weight0 . z = 1.0 − ( 3 . 0 − hatchFactor ) ; weight1 . x = 1.0 − weight0 . z ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >1.0) { weight1 . x = 1.0 − ( 2 . 0 − hatchFactor ) ; weight1 . y = 1.0 − weight1 . x ; } // End e l s e i f e l s e i f ( h a t c h F a c t o r >0.0) { weight1 . y = 1.0 − ( 1 . 0 − hatchFactor ) ; weight1 . z = 1.0 − weight1 . y ; } // End e l s e i f float4 shatchTex0 = tex2D ( Hatch20 , TexCoord ) ∗ weight0 . x ; float4 shatchTex1 = tex2D ( Hatch21 , TexCoord ) ∗ weight0 . y ; float4 shatchTex2 = tex2D ( Hatch22 , TexCoord ) ∗ weight0 . z ; float4 shatchTex3 = tex2D ( Hatch23 , TexCoord ) ∗ weight1 . x ; float4 shatchTex4 = tex2D ( Hatch24 , TexCoord ) ∗ weight1 . y ; float4 shatchTex5 = tex2D ( Hatch25 , TexCoord ) ∗ weight1 . z ; f l o a t 4 s h a t c h C o l o r = shatchTex0 + shatchTex1 + shatchTex2 + shatchTex3 + shatchTex4 + shatchTex5 ; Out . C o l o r = shadow ∗ h a t c h C o l o r + (1−shadow ) ∗ ( 4 ∗ s h a t c h C o l o r /5+ h a t c h C o l o r / 5 ) ; return Out ; }
  • 60. List of Symbols and Abbreviations Abbreviation Description Mentioned NPR Non-Photorealistic Rendering page 7 API Application Programming Interface page 31 CAD Computer Aided Design page 14 IDE Integrated Development Environment page 34 ISO International Organisation for Standardization page 15 GPU Graphics Processing Unit page 31 DirextX DirectX is a collection of APIs for handling mul- page 31 timedia tasks. OpenGL Open Graphics Library page 31 HLSL High Level Shader Language page 33 Cg C for Graphics is a high-level shading language page 33 created by NVIDIA for programming vertex and pixel shaders GLSL OpenGL Shading Language page 33 ATI ATI Technologies U.L.C. is a major designer of page 35 graphics processing units and video display cards and a wholly owned subsidiary of AMD NVIDIA NVIDIA Corporation is a major supplier of graph- page 33 ics processors, graphics cards, media and commu- nications devices for PCs and game consoles 67
  • 61. List of Figures 1.1 Artistic screening with non-linear screen element mapping [SS02] . . . 9 1.2 Objects shaded with real-time hatching effect. [PHWF01] . . . . . . . 12 1.3 A frame from the Oscar winning film for best visual effects using painterly style: ”What Dreams May come” [REV07] (Copyright 1998 PolyGram Filmed Entertainment Distribution, Inc.) . . . . . . . . . . 13 1.4 Technical illustration with ghosting view (Copyright 1994-2006 Kevin Hulsey Illustration) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.1 Geometric configuration of the shadow map algorithm. Sample A is considered to be in shadow because the point on the receiving plane is farther from the light source than the distance stored in the shadow map (which belongs to the blocker). In contrast, sample B is illumi- nated. [Cha04] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2 Bias related artifacts. (a.) Too little bias: incorrect self shadowing effect - pixels falsely determined to be in shadow (b.) Too much bias applied: shadow boundary problem. (c.) Correct bias. [Cha04] . . . . 20 2.3 Shadow map aliasing: (a.) 256x256, (b.) 512x512, (c.) 1024x1024 shadow maps. [Cha04] . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 Inside-outside test: Counting intersections with shadow volumes (in- crementing at front facing polygons and decrementing at back facing polygons) give zero for non-shadowed pixels and non-zero for shad- owed pixels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5 Shadow volumes: Final image (left), Silhouette edges (middle), Shadow volumes (right) [Bra03] . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.6 Soft shadows [HLHS03] . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.7 Combining several hard shadows. Left: 4 samples. Right: 1024 sam- ples [HLHS03] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.8 Inner and outer penumbra calculated from point light. [HLHS03] . . 28 68
  • 62. 69 3.1 A possible hardware rendering architecture including vertex and pixel shaders. [SL04] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2 Screenshot of ATI’s Rendermonkey shader development environment. 35 4.1 Screenshots of the reused Rendermonkey example shaders NPR/Hatch (left) and Shadows/Shadowmapping (right). . . . . . . . . . . . . . . . 40 4.2 Disc object intersecting the trefoil knot object (left), fixed disc scale and position (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.3 Hatching applied to platform (left), shadow hatched with different texture (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.4 Distance based shading of a transparent shadow texture. . . . . . . . . 49