Describe the process to make partman run. This includes listing the necessary input files, their function, and their contents. It also includes a description of the quirks partman exhibits while running. This HOW-TO does not describe HOW-DOES, i.e. it does not detail how partman works, but only the mechanical steps needed to run it.
Partman is a specialized renderer created for the limited task of rendering particles - LOTS of particles. The design goal was to render them fast with extremely low RAM footprint so that many particles could be rendered. As an example, The current record size for a partman render is 470 million "fully 3D" particles in less than 30 MB of RAM, in less than 60 cpu hours (1K image). And there is no limit to the number of particles that can be rendered. Partman employs special algorithms which allow an a-buffer and z-buffer to track all of the information necessary to properly hide one particle behind another with only one particle in memory at a time. Hence, the only cost of increasing the number of particles rendered is cpu time.
The formal structure of the partman command line is shown when you type just partman on the command line, with no options, and press return. The output usage is:
--------------------------------------------------------------------
PARTMAN - Your source for rendering LOTS of particles
usage:
partman -p <parameter-file> -e <edit-parameters> [-g <geometry-file>] [-v]
[-f <first> -l <last> -i <increment>] [-s <shading file>] [-T <nbTiles> -t <tile>]
<parameter-file> = name of file containing rendering parameters
<geometry-file> = name of cinesite file with geometry data
<edit-parameters> = parameters passed to the PDB editor
OPTIONAL PARAMETERS
<first> = first frame number
<last> = last frame number
<increment> = frame number increment
<nbTiles> = Divide up image plane into this many tiles
<tile> = Render this tile
<shading file> = Filename of surface shader
-v Turns on image statistics after render completes.
--------------------------------------------------------------------
An example of its usage is:
partman -f 3 -l 100 -i 2 -p example2.jpa -g water_test2.cig -e"-i /data/3d/IB/9999/CG/water_test/pdb/particleShape1 -p ibemit2 -V rgbPP -R radiusPP -A "
This example illustrates most of the command line options in partman. There are options for controling the frame range, the camera and scene data, the particle "editing" process, and finally an option for shading the surfaces of the particles. the options related to tiling are not guaranteed to work.
The first command line options are just the standard frame range specifiers -f, -l, and -i. These limit the range of frames rendered between the first (-f), the last (-l), and increment in steps (-i). If the -l option is omitted, only one frame (the -f frame) is rendered.
Although not really supported, the tiling options allow you to break up the image plane into virtical strips, the total number of which is fixed by the -T option. The purpose of dividing up the image plane this way is to allow partman to render only one of these tiles, the one designated in the -t option.
The -p option on the command line designates the .jpa file that partman is to use. The jpa file controls the overall rendering characteristics that are specific to partman, and which are not provided in other input files. Primarily, the jpa contains information on some aspect of rendering surfaces, shadow maps, image size in pixels, image file format, the channels to be rendered, and the channels to be output to a hdf file. More detail is provided in a later section below.
The -g option designates the .cig file that partman is to use. These files are generated from Maya using mayaMonkey, and contained multiple frames of ascii data for cameras, lights, and nurbs surfaces and curves. So when rendering the particles from a maya scene, you should use dynExport to extract the particle data into pdb files, and mayaMonkey to extract the remaining relevant information into cig files. More detail is provided in a later section below.
The -s option allows you to provide a file containing a surface shader which partman will use to render the particles. If the -s option is not used, there is a simple default shading applied. The contents of the shader file are defined in more detail in later section.
The -e option provides a double-quoted string, the contents of which are a command-line-like set of options for the pdbEditor to apply in order to read pdb files and spawn child particles for rendering. The meaning of these input options are discussed in a later section.
The jpa file has a hard and tainted legacy. Consequently, items appear in the file which are necessary for the proper parsing of the file, but which have absolutely no impact or value on partman's function. Please bear with us. There are 5 parts necessary to the minimal form of the jpa file (1) the type includes, (2) the control info, (3) the camera info, (4) the lights info, and (5) the garbage info. An example of a minimal jpa file is as follows:
// Part (1) the INCLUDE info
#include "partmanInputTypes.h"
// ----------------------------------------------
// Part (2) the CONTROL info
ControlInfo control;
control . RenderSurface = 5;
control . UseGeometryLights = 1;
control . UseGeometryCamera = 1;
control . DoMotionBlur3D = 0;
control . DoMatteErosion = 0;
// ----------------------------------------------
// Part (3) the CAMERA info
CameraInfo cameraInfo1;
cameraInfo1 . ApplyPSF = 0; // 1=boxcar, 2= bartlett, 3=gaussian
cameraInfo1 . ImageFile = "water_test3";
cameraInfo1 . NX = 512;
cameraInfo1 . NY = 289;
cameraInfo1 . RenderImage = 1; // rgb channels
cameraInfo1 . RenderABuffer = 1;
cameraInfo1 . RenderZBuffer = 1;
cameraInfo1 . RenderTransparency = 0;
cameraInfo1 . RenderUnknown = 1; // effective alpha channel for particles
cameraInfo1 . FarPlane = 1000;
cameraInfo1 . NearPlane = 0;
cameraInfo1 . OutputImage = 1;
cameraInfo1 . OutputABuffer = 0;
cameraInfo1 . OutputZBuffer = 0;
cameraInfo1 . OutputTransparency = 0;
cameraInfo1 . OutputUnknown = 1;
cameraInfo1 . ZDepthTexture . FileName = "";
cameraInfo1 . ABufferTexture . FileName = "";
// ------------------------------------------------
// Part (4) the LIGHTS info
LightingInfo light1;
light1 . RenderShadow = 0;
light1 . ShadowNx = 1024;
light1 . ShadowNy = 1024;
light1 . ShadowTransparency = .0;
light1 . ZDepthTexture . FileName = "";
light1 . ABufferTexture . FileName = "";
//-----------------------------------------------------------
// Part (5) the garbage info
GodRayInfo godray;
WaterSurfaceInfo water;
PartyKullInfo partykull;
StrandPatchInfo patch1;
patch1 . geometryInfo . FileName = "curlyball.cig";
patch1 . geometryInfo . ObjectNames = "nurb";
A jpa file can have much more infomation and structures, but these are all that is necessary
for partman to work.
As you can see from the structure of the jpa file, commenting is accomplished with C++ style double slash. Also, all non-commented information must end in a semicolon.
We will now discuss each part of the jpa file, as it relates to partman.
(1) The INCLUDE part.
The jpa file contains a set of predefined c-like data structures for the control, camera, lights, and other objects. The definition of these structures is contained in the cinclude file. This file may have any name and be located anywhere you wish, but it has to be the same version that was used to compile partman. This file is maintained in the vob as
/proj/CSapps/volume/renderV2/jahasaInputTypes.h
(2) The CONTROL part.
The ControlInfo data structure contains many more parameters than just the three listed, but they are not relevant to partman rendering. The RenderSurface parameter control whether nurbs surfaces are rendered (primarily as hold-out mattes for the particles), and if they are, the level of detail of their rendering. The value RenderSurface = 0 causes partman to ignore all surface geometry. The higher the value of RenderSurface, the finer the tesselation of the surface during its rendering. This is a relatively dumb tesselation because the surface rendering was designed primarily to only provided abuffer and zbuffer information.
The two parameters UseGeometryLights and UseGeometryCamera simply tell partman to use the data in the cig file to specifiy the camera(s) and the light(s) in the scene. Always set their values to 1, because we are not telling you here how to use the jpa file to set the camera and light details (although it can be done).
The parameter DoMotionBlur3D is a switch to turn on (= 1) or off (= 0) camera-based motion blur. When the faster streak motion blur is used (see below), the parameter DoMatteErosion is a switch for whether to do careful depth-checking along the length of the streak. DoMatteErosion = 1 allows blur streaks that begin outside of geometry to penetrate the geometry in the render, whereas DoMatteErosion = 0 does careful checking and preserves the boundaries of objects that may be in the scene.
(3) The CAMERA part.
Most of the data needed for the camera comes from the cig file, e.g. position, orientation, fov, aspect ratio. The CameraInfo stucture is still needed however to specify the number of pixels in the camera in each direction. The number of pixels does not need to be related to the fov or aspect ratio, i.e. non-square pixels are supported. The ImageFile is the "root" name of the output hdf file containing image data, including the path if desired. The output file name is the content of ImageFile with a four digit frame number and ".img" appended to it. The FarPlane and NearPlane parameters have the usual meaning as clipping planes.
The set of parameters RenderImage, RenderABuffer, RenderZBuffer, RenderTransparency, and RenderUnknown optionally turn on a sequence of different types of channels built into partman. All except the Transparency channel should be on. The ABuffer channel contains the alpha channel information for the combination of the particles and the surface geometry (if rendered), while in all circumstances, the Unknown channel contains the alpha channel of just the visible particles, and is the channel that is used when converting to rgba. The OutputImage, OutputABuffer, OutputZBuffer, OutputTransparency, and OutputUnknown parameters turn on/off which channels are actually written to the hdf data file after rendering.
The two parameters ZDepthTexture.FileName and ABufferTexture.FileName are optional strings for hold-out mattes for the A Buffer and Z Buffer. In practice, if either one is used, then both are needed. The ZDepthTexture file is a "zfile" render from prman, and the ABuffer texture should be a tiff file with the alpha data in the red channel. The actual filenames for the files shoudl be
<ZDepthTexture.FileName>####
<ABufferTexture.FileName>####.tif
so these parameters should have only the root part of the file names, and partman uses the frame number (####) to create the full name. If these files are specified, then their values are loaded into the camera data when the camera is initialized. Afterward, any geometry found in the cig file (assuming the RenderSurface parameter is on) is rendered, followed by particles. If these files are not found, the render continues on without them.
(4) The LIGHTS part.
As with the camera(s), most of the data for any lights in the scene is derived from the cig file. The only information which must be obtained from the jpa file is related to the generation of shadow maps when desired. In the cig file, each light has a parameter that designates when the light is to generate shadows. As a convenience, the jpa file has a parameter RenderShadow that allows you to globally turn off all of the shadow generation. Setting RenderShadow = 1 causes shadows to be generated from nurbs surfaces, and RenderShadow = 2 generates shadows from nurbs surfaces and particles. In all cases, no shadows are ever generated for lights that the cig file designates as shadowless.
For those lights that generate shadows, the shadows are accomplished by building a shadow map independently for each light. The shadowmap is an image of the scene as viewed by the light, but only contains abuffer and zbuffer channels. The jpa file LightingInfo structure fixes the dimensions of the image. These dimensions are global to all lights that have shadows. When sampling the shadow, interpolation is used, so having a low dimension for the shadowmap can produced soft shadows in some situations.
The ShadowTransparency parameter allows a global scaling of the intensity of the color of the shadow light that is set for each light in the cig file.
As with the camera, files containing A and Z buffers can be imported to the shadowmap cameras. The ZDepthTexture.FileName and ABufferTexture.FileName strings specify the root portion of the file names. Because the light info data is used for all of the lights, the actual names of the data files must have a tag identifying which light they belong to. The protocol is for the file names to be
<ZDepthTexture.FileName>_lightXX_####
<ABufferTexture.FileName>_lightXX_####.tif
where XX is a two digit number for the light, with the first light being 00. If these files are not found or properly read, the render continues without them.
(5) The garbage info.
This is the result of some legacy content in the software that reads the jpa file. Just leave it as is, with the possible exception of the FileName string. You must have an actual cig file (with proper path) specified here. The curlyball one listed there is presently in /home/jerryt/partman/test/curlyball.cig.
The cig (cinesite) file is a custom text format. The file is created with the mayaMonkey mel script and dso, usually with the mel command
mayaMonkey -all -geom -light -cam -o <outfilename>
This generates a file with name containing multiple frames of data for all of the cameras, lights, and nurbs curves and surfaces in the maya scene that were flagged visible. The data has been baked into final transformations, so there is no hierachical structure to the data. Once generated, you should generally not expect to modify its contents. The variable labels are relatively standard, should you wish to make changes. The attributes of objects in the cig file are basically the same as set up in maya. For example, to have a light cast shadows in partman, turn on shadowing in maya.
The -e command-line option feeds a string into the pdbEditor portion of partman. The string is a command-line-like set of options for the pdb editor to run at each frame. The appendix at the end of this HOW-TO provides detail documentation on the pdb shading language. This shading language is also relevant to the partman surface shader.
The basic form that partman uses is the following:
-e"-i<pdb file root name> -p<pdb shader name>"
The object is the root name, includeing path, of the set of pdb files that have been generated by dynExport, one frame per file. The root name should not include the period just before the frame number in the pdb file name. The object is the name of the single file containing the shader code for changing particle attributes for the child particles and calling the emit() function (see below).
The third file needed to run partman is the pdbEditor shader file. This file used to spawn any number of child particles from each one of the particles contained in the pdb file. The shader language is flexible enough to allow almost unlimited variety of characteristics to be obtained.
To assist in building complex particle emission shaders, a small library of functions and global variables has been built with predefined techniques such as child-particle counting, random fills, random walks, implicit surfaces, lightening bolts, and pseudo-temporal methods. Check them out.
Partman can render motion blurred particles with two different techniques. The fast method is to render the particle once and streak the rendered image across a blur path on the image plane. The second method is more time consuming but precise: the sphere is rendered multiple times along a 3D motion path. In both cases there is control over the fineness of the sampling. Switching from one method to the other is accomplished in the pdbeditor shader, and so can both methods can be used in a frame, on a particle-by-particle basis.
There are of course, two components to motion blur: blur due to camera motion, and blur due to particle motion. Camera motion blur is turned on/off from the jpa file, as described in the section on the jpa file. Blur based on the particle motion is controled by the velocity attribute of the particle. To render WITHOUT motion blur, the magnitude of the velocity attribute must be zero when emit() is called.
If a particle has a non-zero velocity or acceleration, and none of the special options described later on set, then the motion blur is based on a streak drawn across the image plane. You can create funky effects such as paint-like strokes by varying the direction and magnitude of the velocity and/or acceleration in the pdbEditor shading file. The rendering of the streak is a relatively fast operation, so this kind of motion blur is not a burden on the total render time or memory usage.
When it is desirable to have a true 3D motion blur, this can be accomplished by using a pdb attribute called "do3Dblur". Setting this attribute to a nonzero value (the default value is zero) will cause the motion blur to be generated by rendering a sequence of identical particles along the motion path. This method is more time consuming than the default streak method, but in many cases can be necessary in order to get a good looking blur.
For both methods of bluring, the extend and shape of the blur path is controled via the particle's attributes "velocity" and "acceleration". A zero magnitude acceleration produces a straight-line blur, whereas for non-zero acceleration, the blur-path is curved accordingly.
The attribute "do3Dblur" is a particle attribute, but it is not necessary to have it prebuilt into the pdb file. Using the command-line syntax for the pdb processing, you can add this attribute with the addition of "-R do3Dblur -A" to the pdb command-line. See the appendix on how to do this.
There is an additional attribute which partman recognizes for controlling the content of motion blur. The particle attribute "blursample" allows you to increase or decrease the number of sample points for both the streak and 3D blur methods. As with do3Dblur, you must add this attribute to the attribute list with "-R blursample" on the pdbEditor command line.
One of the benefits of having do3Dblur and blursample as particle attributes is that their values can be set independently for every particle rendered. This allows complete freedom to decide what the best method is for any step of the render.
When the faster blur streak is used, the parameter DoMatteErosion in the CONTROL section of the jpa file acts as a switch for whether to do careful depth-checking along the length of the streak. DoMatteErosion = 1 allows blur streaks that begin outside of geometry to penetrate the geometry in the render, whereas DoMatteErosion = 0 does careful checking and preserves the boundaries of objects that may be in the scene.
The fourth file which can be included on the partman command line is the surface shading file for the -s option. This file is purley optional however. If no -s option is giving, a default shading of just the intrinsic color of the particles is used. Choosing to employ surface shading however, give you complete control over how to apply lighting (if any), procedural texturing (if any), and even whether to render the particle at all. This shader is applied on a point-by-point basis on the surface of the sphere that makes of each particle that is rendered. If the shader does nothing, then the default particle-color-only shading is applied. The description of the shading process for the pdbEditor applied for this surface shader as well, except for the following differences:
The particle attributes accessed via getAtt and setAtt are not available
Instead, the following list of parameters and their data types is available and pre-calculated each time the shader is called:
name = "Ci"
type = vector
description: initialized to color of particle, on return, is the shaded color used.
name = "Oi"
type = vector
description: initialized to 1 in each channel, on return is the optical transparency. For the most part, this is not used.
name = "pos"
type = vector
description: Position on the spherical surface of the particle to be shaded, in world coordinates.
name = "center"
type = vector
description: position of the center of the particle, in world coordinates.
name = "cam_pos"
type = vector
description: position of the camera focal plane, in world coordinates.
name = "cam_dir"
type = vector
description: unit vector giving the camera viewing direction.
name = "point_to_cam"
type = vector
description: unit vector pointing from pos to the cam_pos. Equivalent to normalizing the difference vector cam_pos - pos.
name = "point_to_light"
type = vector
description: Unit vector point from the point pos to the current light.
name = "light_color"
type = vector
description: color of the light, including any impact of shadowing.
name = "dist_to_cam"
type = float
description: distance between the camera and the point on the sphere being shaded.
name = "dist_to_light"
type = float
description: distance between the light and the point on the sphere being shaded.
name = "light_id"
type = int
description: The id of the light. The lights are numbered sequentially 0, 1, ..., N-1 for N lights, in the order they appear in the cig file.
name = "radius"
type = float
description: The radius of the spherical particle.
name = "lamber"
type = float
description: The lambertian cosine factor.
name = "alpha"
type = float
description: The geometric alpha of the particle in the current pixel.
All of the c-like syntax and math routines available in the pdbEditor are also available here.
After rendering, partman writes the channel data to a hdf file. To retrieve an rgb image in tiff format, try
hdftotiff -f 5 -l 10 -i 3 -r<filerootname> -G1 -m0 -M1
This converts files <filerootname>XXXX.img to XXXX.tif for frames XXXX in the range 5 to 10, increment 3. To convert from floating point to integers, the gamma was set to 1 (-G1), and the floating point value corresponding to the black point is 0 (-m0), and the white point floating point value is 1 (-M1). If you chose to output the Unknown channel in the jpa file, then you may also retrieve an alpha channel in the tiff file, via
hdftotiff -f 5 -l 10 -i 3 -r<filerootname> -G1 -m0 -M1 -a0 -A1
which sets the floating point values for the white and black points of the alpha channel, which is always converted with linear gamma.
Below is the man page for the pdbEditor, with a little modification. Since the pdbEditor is more general than just partman, it has some options with do not apply, or only apply to portions of partman. We have therefore edited the man page to reflect these differences.
NAME
PdbEditor - edit particle database files
SYNOPSIS
PdbEditor [ options ] -f first -i input_base -o output_base -p prog_name
DESCRIPTION
PdbEditor reads one or more PDB (Dynamation Particle Database) files
and generates one or more processed PDB files. The primary
applications of this program are debugging and editing of
particle database files.
The central element of the editor is a C-like language that can
manipulate an input particle database one particle at a time. As
each particle is read by the editor, its attributes are loaded.
Additional user-defined particle attributes may also be defined on
the command line; these attributes are likewise created and loaded
with zeros. The editor then executes the user-specified program.
Whenever the "emit" command is encountered, a copy of the current
particle is output. Because there are decision operators in the
language, each input particle may result in zero or more output
particles. Also, command line flags control the stucture of the
output particles. Therefore, the attributes of the generated particles
need not be the same as those in the input particles.
ARGUMENTS
The PDB editor takes the following arguments:
-A
If any of the "-I", "-R", or "-V" options are used, the
"-A" flag causes all of the input attributes to be output.
-a attribute
The normal behavior is for all of the attributes in the input
particle to present in the output particle. If this flag is used
one or more times on the command line, only the named attributes
are output. For instance, "PdbEditor -a id -a position ..." will
cause only the id and position attributes to be emitted.
-I|R|V attribute
If one of these flags is present, new particle attributes are
created. The attribute type is determined by which flag is used
("I", "R" or "V"). Such created attributes are always output.
Using one of these flags has the same effect as the "-a" flag in
supressing the output of the input particle attributes. If you
wish to also have the input particle attributes present in the
output, you may use the "-a" flag to selectively include them or
the "-A" flag to include all of them.
-c
Concatenation mode. This flag causes all of the input PDB files to
be merged into a single PDB database. A really interesting use of
this function is to extract the time behavior of a single particle.
For example, "if (id == 3) emit" will cause only the particle with
the id of 3 to be emitted. If a series of PDB files are presented
as input, the output will consist of all of the states of particle
number 3 in that series of PDB files. This is useful either for
debugging or converting the temporal behavior of a particle into a
spatial distribution of particles.
-r frame_num
Sets the internal variable for frame to desired value. This
variable is incremented each time a PDB file is processed.
-p program_name
Name of the program to be run on the PDB files.
-f first
Frame number of first PDB file to be processed.
-l last
Frame number of last PDB file to be processed. The default value is the
first frame number.
-i input_base
Base name of input PDB files. Therefore, if input_base is "in" and
the -f argument equals 5, it is assumed that the first input PDB file is
named "in.5.pdb".
-o output_base
Base name of output PDB files. Therefore, if output_base is "out" and
the -r argument equals 7, it is assumed that the first output PDB file is
named "out.7.pdb".
Of these arguments, the -f, -i, -o, and -p are required.
EXAMPLES
As stated earlier, the editor language is very similar to C. Let us look
at a couple of simple examples to see how it works.
Consider first an example in which 1000 is added to each particle "id"
attribute:
int
main()
{
int foo;
getAtt("id", &foo);
setAtt("id", foo + 1000);
emit();
return 0;
}
The first statement declares an integer variable called "foo". The second
statement sets the value of the "foo" variable to the value of the
attribute named "id". The third statement sets the value of the
attribute named "id" to foo + 1000. Lastly, the emit command copies
the modified particle to the output. It should be noted that namespace
collisions between variable names and attribute names are not possible.
Therefore, "foo" could be renamed to "id" in the above example.
To understand how decision operators can be used to emit or not emit a
particle consider the following example. Here, only particles with an
"id" greater than or equal to 10 are emited:
int
main()
{
int id;
getAtt("id", &id);
if (id >= 10)
emit();
return 0;
}
Also, looping constructs can be used to "beef up" particles with
multiple copies. If we want to randomly duplicate a particle along its
velocity vector, we could use the following code:
int
main()
{
/* Emit current particle. */
emit();
/* Get initial value for position. */
vector initPos;
getAtt("position", &initPos);
/* Seed the random number generator. */
int id;
getAtt("id", &id);
srand48(id);
/* Get the velocity attribute value. */
vector velocity;
getAtt("velocity", &velocity);
/* Output three particles along the velocity vector. */
int cnt;
cnt = 0;
vector curPos;
while (cnt < 3) {
curPos = initPos + (velocity * drand48());
setAtt("position", curPos);
emit();
cnt = cnt + 1;
}
return 0;
}
Seeding the random number generator with "id" ensures that the
randomized calculations will always be the same for this particle
when it is part of a series of PDB files.
EDITOR LANGUAGE
The above examples of the particle processing language describe
its basic capabilities. A tabluation of all the currently exisiting
operators and functions follows:
getAtt(string, pointer)
Get the particle attribute named "string" and assign it to the
variable associated with "pointer". Run time type checking is
performed on the variable and the attribute to ensure that a vector
isn't assigned to a scalar variable (or vice versa).
setAtt(string, expression)
Set the particle attribute named "string" to the value of
"expression". Run time type checking is performed on the expression
and the attribute to ensure that a vector isn't assigned to scalar
attribute (or vice versa).
$variable
A reference to an internal variable. Current internal variables are:
$frame frame number
$np_out number of particles emitted on current frame (running
count).
$tot_np_out total number of particles emitted for all frames thus
far.
(expression0, expression1, expression2)
Form a vector whose elements are "expression0", "expression1", and
"expression2".
variable[integer]
Reference a given item of a vector.
Binary Operators
'+', '-', '*', '/', '%', '|', '&', '^', '||', '&&', '==', '!=', '>', '<',
'>=', and '<=' all have their usual C language meanings. In addition,
'.' denotes the dot product, and '^' denotes the cross product on vectors.
Math Functions / Procedures
sin, asin, cos, acos, tan, atan, log, log10, drand48, srand48, sqrt, pow
and fabsf make calls to the C language functions/procedures. Also, Perlin
noise functions having the following prototypes have been implemented:
float noise1(double x);
float noise2(double x, double y);
float noise3(double x, double y, double z);
printf(format, expression0, expression1, ... , expressionN)
C-style print statement with the exception "%v" prints a vector.
Flow Control
"if" and "while" are currently the only flow control statements.