20 Picking and Using Selection

20.010 How can I know which primitive a user has selected with the mouse?

OpenGL provides the GL_SELECTION render mode for this purpose. However, you can use other methods.

You might render each primitive in a unique color, then use glReadPixels() to read the single pixel under the current mouse location. Examining the color determines the primitive that the user selected. Here's information on rendering each primitive in a unique color and information on using glDrawPixels().

Yet another method involves shooting a pick ray through the mouse location and testing for intersections with the currently displayed objects. OpenGL doesn't test for ray intersections (for how to do, see the BSP FAQ), but you'll need to interact with OpenGL to generate the pick ray.

One way to generate a pick ray is to call gluUnProject() twice for the mouse location, first with winz of 0.0 (at the near plane), then with winz of 1.0 (at the far plane). Subtract the near plane call's results from the far plane call's results to obtain the XYZ direction vector of your ray. The ray origin is the view location, of course.

Another method is to generate the ray in eye coordinates, and transform it by the inverse of the ModelView matrix. In eye coordinates, the pick ray origin is simply (0, 0, 0). You can build the pick ray vector from the perspective projection parameters, for example, by setting up your perspective projection this way

aspect = double(window_width)/double(window_height);
glMatrixMode( GL_PROJECTION );
glLoadIdentity(); 
glFrustum(-near_height * aspect, 
   near_height * aspect, 
   -near_height,
   near_height, zNear, zFar );

you can build your pick ray vector like this:

int window_y = (window_height - mouse_y) - window_height/2;
double norm_y = double(window_y)/double(window_height/2);
int window_x = mouse_x - window_width/2;
double norm_x = double(window_x)/double(window_width/2);

(Note that most window systems place the mouse coordinate origin in the upper left of the window instead of the lower left. That's why window_y is calculated the way it is in the above code. When using a glViewport() that doesn't match the window height, the viewport height and viewport Y are used to determine the values for window_y and norm_y.)

The variables norm_x and norm_y are scaled between -1.0 and 1.0. Use them to find the mouse location on your zNear clipping plane like so:

float y = near_height * norm_y;
float x = near_height * aspect * norm_x;

Now your pick ray vector is (x, y, -zNear).

To transform this eye coordinate pick ray into object coordinates, multiply it by the inverse of the ModelView matrix in use when the scene was rendered. When performing this multiplication, remember that the pick ray is made up of a vector and a point, and that vectors and points transform differently. You can translate and rotate points, but vectors only rotate. The way to guarantee that this is working correctly is to define your point and vector as four-element arrays, as the following pseudo-code shows:

float ray_pnt[4] = {0.f, 0.f, 0.f, 1.f};
float ray_vec[4] = {x, y, -near_distance, 0.f};

The one and zero in the last element determines whether an array transforms as a point or a vector when multiplied by the inverse of the ModelView matrix.

20.020 What do I need to know to use selection?

Specify a selection buffer:

GLuint buffer[BUF_SIZE];
glSelectBuffer (BUF_SIZE, buffer);

Enter selection mode, render as usual, then exit selection mode:

GLint hits;

glRenderMode(GL_SELECT);
// ...render as usual...
hits = glRenderMode(GL_RENDER);

The call to glRenderMode(GL_RENDER) exits selection mode and returns the number of hit records stored in the selection buffer. Each hit record contains information on the primitives that were inside the view volume (controlled with the ModelView and Projection matrices).

That's the basic concept. In practice, you may want to restrict the view volume. The gluPickMatrix() function is a handy method for restricting the view volume size to within a set number of pixels away from a given (X,Y) position, such as the current mouse or cursor location.

You'll also want to use the name stack to specify unique names for primitives of interest. After the stack is pushed once, any number of different names may be loaded onto the stack. Typically, load a name, then render a primitive or group of primitives. The name stack allows for selection to occur on heirarchical databases.

After returning to GL_RENDER render mode, you'll need to parse the selection buffer. It will contain zero or more hit records. The number of hit records is returned by the call to glRenderMode(GL_RENDER). Each hit record contains the following information stored as unsigned ints:

You can use the minimum and maximum Z values with the device coordinate X and Y if known (perhaps from a mouse click) to determine an object coordinate location of the picked primitive. You can scale the Z values to the range 0.0 to 1.0, for example, and use them in a call to gluUnProject().

20.030 Why doesn't selection work?

This is usually caused by one of two things.

Did you account for the inverted Y coordinate? Most window systems (Microsoft Windows, X Windows, others?) usually return mouse coordinates to your program with Y=0 at the top of the window, while OpenGL assumes Y=0 is at the bottom of the window. Assuming you're using a default viewport, transform the Y value from window system coordinates to OpenGL coordinates as (windowHeight-y).

Did you set up the transformations correctly? Assuming you're using gluPickMatrix(), it should be loaded onto the Projection matrix immediately after a call to glLoadIdentity() and before you multiply your projection transform (using glFrustum(), gluPerspective(), glOrtho(), etc.). Your ModelView transformation should be the same as if you were rendering normally.

20.040 How can I debug my picking code?

A good technique for debugging picking or selection code is not to call glRenderMode(GL_SELECT). Simply comment out this function call in your code. The result is instead of performing a selection, your code will render the contents of the pick box to your window. This allows you to see visually what is inside your pick box.

Along with this method, it's generally a good idea to enlarge your pick box, so you can see more in your window.

20.050 How can I perform pick highlighting the way PHIGS and PEX provided?

There's no elegant way to do this, and that's why many former PHIGS and PEX implementers are now happy as OpenGL implementers. OpenGL leaves this up to the application.

After you've identified the primitive you need to highlight with selection, how you highlight it is up to your application. You might render the primitive into the displayed image in the front buffer with a different color set. You may need to use polygon offset to make this work, or at least set glDepthFunc(GL_EQUAL). You might only render the outline or render the primitive consecutive times in different colors to create a flashing effect.