Adding new tests to glean
On this page, we'll consider some general testing principles and
suggest areas in which new tests are needed.
When you're ready to implement a new test, check out the
overview of glean internals.
Guidelines for writing glean tests
To define a new test, first create a new test class derived from
Test. Most of the time you'll want to write tests
that run on all drawing surface configurations and are portable
(in the sense that they don't use any OS- or window-system-dependent
data structures exposed in the Environment, WindowSystem,
and DrawingSurface classes). Starting from BasicTest may
be easiest in those cases.
Once you've created a new test class, create one or more global
objects of that class. The constructor for Test ensures that
all such objects are automatically linked onto the master list of
tests. (Creating test objects dynamically isn't recommended, but if
you can make a good argument for it we can figure out the rules needed to
make it work.) Every test must have a good description; it's very much
worthwhile to add text that will help the user understand the results
or debug the problem if something goes wrong.
The first workhorse method in any test class is run().
Here are some things to remember about it:
Every run must produce at least one PASS, FAIL, or NOTE message in the
In the case of failure, never just report failure and assume the user
will go add debugging code to the source to figure out exactly what failure
occurred. Ideally, provide enough information so that a driver
developer could start debugging his code without having to look at
the glean source first.
Each test is producing two kinds of result: The result that goes into
the results database, and the result that remains in memory, accessible
through the test object. Eventually there will be tests that refer
back to the results of previous tests! (This is the reason for the
hasRun mechanism in the Test class; one test may be
forced to run another as a prerequisite, but we never need to run a
Try to keep most test runs short. Eventually there will be a lot of
tests, and some machines will have a lot of drawing surface
configurations, and runtime will add up.
Whenever possible, avoid tests that just generate images for the
user to check, rather than determining for themselves whether they
pass or fail. Even experts find it hard to tell which of two
complicated images is correct, and when some subtle part of the
OpenGL driver changes (like the precision with which colors are
computed) the user might be faced with comparing hundreds or
thousands of images for all existing tests.
Prefer statistical summaries of error to simple pass/fail results.
They provide a lot more useful information, and in some cases they
make writing test comparisons much easier and more effective.
The second major method is compare(). Some things to keep in mind:
When two test results are the same (whatever that means for the
particular test in question), don't report that they are the same
unless the user has requested verbose output. Ideally, if two
test runs are identical, glean should report nothing.
Subject to the constraint in the previous item, every comparison
must yield at least one SAME or DIFF message in the log.
As with run(), statistical summaries are preferable to simple
It's important to make sure that all OpenGL state variables are set
in the way you expect; otherwise a drawing operation can have surprising
results. This OpenGL Rendering and State
Cheat-Sheet can help you determine what state variables you need
to set (and might suggest a few things that need testing, too).
Here are some suggestions for categories of tests that could be added to
(These are not listed in any particular order; any of them would be
In addition, please see the JOBS file in the top level of
the glean source tree for an informal list of specific
We could use a set of tests that check everything John Carmack mentioned
Default state check.
The OpenGL spec defines initial values for all state variables, and
many programs (including glean!) depend on those initial
values being correct. It would be great to have a test that actually
Enumerant coverage check.
It would be good to verify that each OpenGL command actually accepts
all the argument values that it's supposed to accept, without generating
an error. Conversely, it would be nice to know that errors are
generated appropriately when illegal arguments are used.
(This is especially relevant to everyone working on Mesa.)
Regression tests simply check that a previously-discovered bug or
performance problem doesn't reappear after later changes to an OpenGL
driver. Regression tests are really valuable, because bugs tend to
appear in the parts of the driver that are the most difficult to
implement correctly. Therefore, writing tests in response to each bug
report tends to cluster tests in precisely the area where they're most
needed. As a general principle, whenever you report a bug or fix a bug,
you could write a glean test to make sure that bug will never
Stress tests assess the robustness of an OpenGL implementation. Some
possibilities are: create and download many texture objects, then
switch rapidly from one to another; perform many buffer swaps in
rapid succession; execute ten thousand glClear() operations and see
whether the hardware FIFO times out; and create a large number of windows
and rendering contexts and randomly bind arbitrary pairs of them.
Every OpenGL implementation has limits (sometimes enforced by the graphics
hardware) and these often determine how applications must be written.
For example, it would be good to know the maximum acceptable texture
size for a variety of texture formats; the largest image that can
be passed to glDrawPixels(); the maximum number of vertices that can
be placed in a triangle strip or vertex array; and the maximum number of
windows or rendering contexts that can be created.
Window-system binding tests.
Most tests concentrate on OpenGL itself, but an area that must not be
overlooked is the window system binding (GLX or WGL). It would be useful
to test the window system binding's extension string and all the basic
operations like context creation, copy, and deletion. Sharing texture
objects and display lists is important to test.
Core OpenGL correctness tests.
Most OpenGL implementations have been checked with the OpenGL
conformance tests. However, the conformance tests are lax in some
areas due to historical reasons. One way to address this would be
to add some correctness tests to glean. Check the contents
page of the OpenGL spec; each heading is a good suggestion for a