Project Outline

Progress Update 1, March 7, 2001

Progress Update 2, April 11, 2001

Project Final Report

Project Description

The major novelty and benefit of such a system over most standard non-photorealistic rendering techniques is that it allows users to create a wide variety of effects, ranging from artistic styles like crosshatching to the more bizarre effects you will see demonstrated below, all by simply changing the palette of textures . The obvious drawback is that it doesn't have the ability to do some of the things available in  any custom implementation of single NPR techniques.  For example, some non-photorealistic rendering system that implement sketching techniques like crosshatching could align the direction of the crosshatches to the curvature of  the model. This is not possible within our system. We see the benefits of allowing users  to express their creativity and invent new and exciting NPR rendering styles greatly outweighs this drawback.

Using The System

2)    Once the renderer in initialized the render state can be set. This is done with calls to functions similar to the OpenGL render state functions. For example the function:
 

void Lightfv(int LightID, int ParamName, float *Value);

allows a user to set a lighting parameter just like in OpenGL. The important parameters that the renderer supports are for setting up the light and material properties.

The system also has a few custom state variables which can be set. These are used to specify how the system chooses the textures to map to the object.

ColorMatchMode - controls how the method used to match textures to colors.

- In MATCH_COLORS mode it tries to match over all color channels, so that a blue light triangle will be matched to a blue texture.
- In MATCH_INTENSITY mode it only looks at the overall brightness of the triangle and matches it to a texture of similar brightness.
- MATCH_ORDER mode is useful for texture palettes that do not vary in average intensity. In this mode it chooses textures based on the order they appear in the input text file, with brighter target triangles mapping to textures further down on the list.

IntensityRangeScale - allows scaling of the range covered by the texture palette. If the textures in the texture palette only hit a small range of intensities, the IntensityRangeScale can be used to spread them out.

3) Once the render state is set. The system behaves just like OpenGL. To render some triangles (the only supported primitives) call Begin(), then RenderTriangle(Tri) to send the triangles and finally End() to complete rendering.

Implementation

1)  The Begin() call, to begin rendering,  just clears the some internal data structure in the renderer to prepare for the next rendering frame.

 2) Each RenderTriangle(Tri) call sends one triangle to be rendered. In our renderer all rendering is done in a deferred fashion. When a triangle is sent to be rendered the first step is to determine what the lighting color of that triangle should be in the OpenGL lighting model. We choose the center of mass of the triangle and use a software implementation of the OpenGL lighting model to compute the lighting intensity at that point, given the state of the lights and the material properties. Once this is done we must decide which texture best matches that lighting color.

Texture matching is done in several ways. Keep in mind that as a preprocess we have already computed the average intensity of all of the textures in the texture palette. If the color matching mode is MATCH_COLORS we treat the lighting color of the triangle and the average color of each texture as vector in R³ . We then apply a norm to compute the distance between the textures color and the triangle's color and choose the texture that is closest. If the color matching mode is MATCH_INTENSITY we take the maximum value of the three channels of the triangle's color, multiply it by 3 and use that as it's intensity value. For the texture average color we take the the sum of the three color channels as its intensity. We look for the texture with the smallest difference. This may seem like a strange method of matching intensity, but in practice it works quite well. The result is that any triangle that is brightly lit with any color light is mapped to the bright textures in the texture palette. Finally, if the color mode is MATCH_ORDER we simple compute the sum of the three color channels for the triangle lighting and index that number into the texture set so that brighter triangles get assigned textures further down the list.

Once the a triangle is associated with one of the textures in the palette by the above steps we simply bucket the triangle into one of a series of buckets maintained by the renderer. There is one bucked for each palette texture. Once the last RenderTriangle(Tri) call is done every triangle has been assigned a texture by its lighting intensity and all triangles are sorted into the appropriate buckets to be rendered.

3) The End() call is where all of the the triangles are rendered to the screen. The actual rendering is done using OpenGL, but with state set by the renderer to create the non-photorealistic effect. The process works as follows:

• Render the background texture on a Quad that covers the entire screen width the OpenGL depth text disabled.
• For each palette texture bucket:

Disable all lighting and textures and enable the depth test and stencil function.

Render every triangle in the bucket setting the stencil buffer for pixels hit by the triangles.

Disable depth text and enable the stencil test. Render a full screen Quad with the background texture. This will only modify pixels who have their stencil bit enabled.

Enable alpha blending. Render a full screen Quad with the palette texture corresponding to that bucket. Alpha blending will blend the palette texture with the background wherever the palette texture is transparent. Again, only pixels with stencil bit enabled will be modified.

• Reset the render state to the way it was before the renderer was called.

There are some very important features of this rendering method which need mentioning:

Performance Features:

1. All rendering is deferred so state changes are minimal.
2. The triangles of the model are only passed to OpenGL once to render. When they are rendered, the lighting and texture states are turned off for maximum rendering speed. This can be done because we are only interested in the stencil buffer values for each pixel.
3.  The only textured primitives rendered by the non-photorealistic renderer are the full screen Quads, which have a very low triangle count. Also there are only two times as many full screen Quads rendered as palette textures. Thus the number of textured primitives rendered is independent of the triangle count of the model. Also, since full screen Quads are very cheap to render, the only real performance hit caused by adding palette textures is to the texture memory which must load all the textures in sequence for every frame.

Esthetics Features:

1. The renderer always renders the textures on a full screen Quad. Apart from the performance benefits, we also have the benefit that the textures are not warped by OpenGL's perspectively correct texturing. For example if we simply texture mapped the original triangles of the model with a crosshatch pattern then, if a triangle was tiled back away from the camera, we would see the crosshatch lines converging as depth increase (think of railway tracks converging to a point in the distance). This is a common effect we take for granted with texture mapping, which looks correct is we image that the texture is painted on the original triangle. But when we do non-photorealistic rendering we want the crosshatch pattern to always appear flat to the camera, so that depth variation in the original model don't effect how the texture appears. This is what you expect when you see a an artist draw a picture of the model using crosshatching.  This is exactly the effect we get using this method of rendering.

Screen Shots

In all cases I used a 3D model of a Tie Fighter to show the non-photorealistic effect.
 

Style	OpenGL Rendering	NPR Rendering
Crosshatch
Description:	The crosshatch texture palette is made up of 10 textures which start from completely transparent and with each texture gain more and more black crosshatch lines until the entire image is black. This gives good results for light colored background images, like the one shown. The best choice for color matching mode is MATCH_INTENSITY since we want to map the various colors of the lighting to black and white texture images.

Style	OpenGL Rendering	NPR Rendering
Speckle Trace
Description:	The speckle trace texture palette is made up of 16 textures which start from completely transparent and with each texture gain more and more black speckles until the entire image is black.

Style	OpenGL Rendering	NPR Rendering
Wood
Description:	The wood texture set contains 30 images of various wood grain textures taken from the internet.

Style	OpenGL Rendering	NPR Rendering
Rock
Description:	The rock texture set has 17 images of various rock patterns taken from the internet. This is the only set with which I used MATCH_COLORS color matching mode so that triangles lit in red would match reddish rock textures and triangles lit in blue would match blue rock textures.

Style	OpenGL Rendering	NPR Rendering
Mist
Description:	The mist texture set has 13 images of multicolored mist that vary in color and transparency across the images.

Style	OpenGL Rendering	NPR Rendering
Spray Paint
Description:	The spray paint palette has 14 images which start as black spray paint on a transparent background and evolve to include various shades of blue and pink spray paint over top.

Style	OpenGL Rendering	NPR Rendering
Palette Knife
Description:	This set produces subtle effects that are hard to see without a moving image. Here I used the Palette Knife filter for Adobe PhotoShop applied repeatedly to an image to make   the texture set. I used the MATCH_ORDER matching mode because the images do not vary in intensity. The result is a rendering style that blurs and warps the image for darker triangles and leaves it alone for lighter triangles.

Style	OpenGL Rendering	NPR Rendering
Offset
Description:	Again this is the result of a PhotoShop filter on a single image. Here I simply offset the image 2 pixels for each image in the texture palette. The result (using MATCH_ORDER color matching) is a style where darker parts of the Tie Fighter are offset more than lighter ones.

Style	OpenGL Rendering	NPR Rendering
Solerize
Description:	This again results from a PhotoShop filter. Here I applied the solerize operation multiple times to an image, so that darker parts of the Tie Fighter solerize the image more than lighter ones.

Results & Conclusions

The choice of texture set is also very important. It was observed that sets generated in PhotoShop tended to work better (in terms of smoothness of the overall appearance) than those gathered from the web. The reason for this is that texture sets work best if they vary along one parameter and fall evenly and fairly closely together in the parameter dimension. Texture sets that I compiled from the web tended to vary in both all color channels and be unevenly distributed among the color range. Using such a texture set and trying to match color is the same as trying to sample the entire space of RGB color with under 30 samples. With such a low sampling rate you invariable get large jumps between the colors. On the other hand texture sets that varied in one parameter could be matched with MATCH_INTENSITY or MATCH_ORDER modes quite well. This is because in these modes we project the colors of the lighting down to a one dimensional space (overall intensity). It is this space that we need to sample with our textures. Sampling a one dimensional space with under 30 samples is still difficult, but in is a much more manageable task than a 3 dimensional space. The results show much smoother transitions if we do this sampling using evenly spaced and close together samples like the ones we can generate in PhotoShop.

Future Work

• Implementation Using Pixel Shaders

This concept of using textures to define custom shading operations is an ideal candidate for implementation using pixels shaders. This would provide a dramatic increase in performance, with a possible drawback being the limitation on texture stages.
• Extension of Image Analysis

The image analysis preprocess used to gather information about the texture palette is currently limited to only computing the average color intensity of the textures. Adding more complex operations like determining the major directional components of the texture image could allow the system to control not only which texture is chosen but also how it it oriented. This would allow effects like alignment of crosshatches to curvature to be implemented.
• More Fun With Texture Sets

The more creative texture sets that people can come up with, the better the world will be.

References

Lake A., Marshall C., Harris M. Blackstein M., “Stylized Rendering Techniques For Scalable Real-Time 3D Animation” Intel Architecture Lab & University of North Carolina.

 
Salisbury M., Wong T., Hughes J., Salesin D., “Orientable Textures for Image-Based Pen-and-Ink Illustration”, University of Washington.

 
Lansdown, J. & Schofield, S. Expressive Rendering: A review of non-photorealistic  techniques, in IEEE Computer Graphics & Applications,

Volume 15(3) May 1995.
 
Teece, D., (Supervisor: Watt, A.), "Three-dimensional interactive non-photorealistic rendering", PhD thesis, University of Sheffield, 1998. 12.

OpenGL emulation code

• Kenneth E. Hoff III, "SoftGL System" University of North Carolina - Chapel Hill, 1999