Adding new tests to glean

On this page, we'll consider some general testing principles and suggest areas in which new tests are needed. When you're ready to implement a new test, check out the overview of glean internals.

Guidelines for writing glean tests

To define a new test, first create a new test class derived from Test. Most of the time you'll want to write tests that run on all drawing surface configurations and are portable (in the sense that they don't use any OS- or window-system-dependent data structures exposed in the Environment, WindowSystem, and DrawingSurface classes). Starting from BasicTest may be easiest in those cases.

Once you've created a new test class, create one or more global objects of that class. The constructor for Test ensures that all such objects are automatically linked onto the master list of tests. (Creating test objects dynamically isn't recommended, but if you can make a good argument for it we can figure out the rules needed to make it work.) Every test must have a good description; it's very much worthwhile to add text that will help the user understand the results or debug the problem if something goes wrong.

The first workhorse method in any test class is run(). Here are some things to remember about it:

• Every run must produce at least one PASS, FAIL, or NOTE message in the log.
• In the case of failure, never just report failure and assume the user will go add debugging code to the source to figure out exactly what failure occurred. Ideally, provide enough information so that a driver developer could start debugging his code without having to look at the glean source first.
• Each test is producing two kinds of result: The result that goes into the results database, and the result that remains in memory, accessible through the test object. Eventually there will be tests that refer back to the results of previous tests! (This is the reason for the hasRun mechanism in the Test class; one test may be forced to run another as a prerequisite, but we never need to run a test twice.)
• Try to keep most test runs short. Eventually there will be a lot of tests, and some machines will have a lot of drawing surface configurations, and runtime will add up.
• Whenever possible, avoid tests that just generate images for the user to check, rather than determining for themselves whether they pass or fail. Even experts find it hard to tell which of two complicated images is correct, and when some subtle part of the OpenGL driver changes (like the precision with which colors are computed) the user might be faced with comparing hundreds or thousands of images for all existing tests.
• Prefer statistical summaries of error to simple pass/fail results. They provide a lot more useful information, and in some cases they make writing test comparisons much easier and more effective.

The second major method is compare(). Some things to keep in mind:

• When two test results are the same (whatever that means for the particular test in question), don't report that they are the same unless the user has requested verbose output. Ideally, if two test runs are identical, glean should report nothing.
• Subject to the constraint in the previous item, every comparison must yield at least one SAME or DIFF message in the log.
• As with run(), statistical summaries are preferable to simple pass/fail assessments.

It's important to make sure that all OpenGL state variables are set in the way you expect; otherwise a drawing operation can have surprising results. This OpenGL Rendering and State Cheat-Sheet can help you determine what state variables you need to set (and might suggest a few things that need testing, too).

New tests

Here are some suggestions for categories of tests that could be added to glean. (These are not listed in any particular order; any of them would be worthwhile.)

In addition, please see the JOBS file in the top level of the glean source tree for an informal list of specific requests.

• Quake3 Arena.
  We could use a set of tests that check everything John Carmack mentioned in http://www.quake3arena.com/news/glopt.html.
• Default state check.
  The OpenGL spec defines initial values for all state variables, and many programs (including glean!) depend on those initial values being correct. It would be great to have a test that actually verified them.
• Enumerant coverage check.
  It would be good to verify that each OpenGL command actually accepts all the argument values that it's supposed to accept, without generating an error. Conversely, it would be nice to know that errors are generated appropriately when illegal arguments are used.
• Regression tests.
  (This is especially relevant to everyone working on Mesa.) Regression tests simply check that a previously-discovered bug or performance problem doesn't reappear after later changes to an OpenGL driver. Regression tests are really valuable, because bugs tend to appear in the parts of the driver that are the most difficult to implement correctly. Therefore, writing tests in response to each bug report tends to cluster tests in precisely the area where they're most needed. As a general principle, whenever you report a bug or fix a bug, you could write a glean test to make sure that bug will never reappear.
• Stress tests.
  Stress tests assess the robustness of an OpenGL implementation. Some possibilities are: create and download many texture objects, then switch rapidly from one to another; perform many buffer swaps in rapid succession; execute ten thousand glClear() operations and see whether the hardware FIFO times out; and create a large number of windows and rendering contexts and randomly bind arbitrary pairs of them.
• Limits tests.
  Every OpenGL implementation has limits (sometimes enforced by the graphics hardware) and these often determine how applications must be written. For example, it would be good to know the maximum acceptable texture size for a variety of texture formats; the largest image that can be passed to glDrawPixels(); the maximum number of vertices that can be placed in a triangle strip or vertex array; and the maximum number of windows or rendering contexts that can be created.
• Window-system binding tests.
  Most tests concentrate on OpenGL itself, but an area that must not be overlooked is the window system binding (GLX or WGL). It would be useful to test the window system binding's extension string and all the basic operations like context creation, copy, and deletion. Sharing texture objects and display lists is important to test.
• Core OpenGL correctness tests.
  Most OpenGL implementations have been checked with the OpenGL conformance tests. However, the conformance tests are lax in some areas due to historical reasons. One way to address this would be to add some correctness tests to glean. Check the contents page of the OpenGL spec; each heading is a good suggestion for a test.