OpenXR™ Conformance Test Suite Usage Instructions and Developer Guide

The same basic instance configuration is used across the test suite for a single execution, with additional versions and extensions added as required. By default, all tests are run as OpenXR 1.0 except those that require a higher version. Tests whose behavior is expected to change between versions are invoked from two separate test cases: one run as each version. In this way, a default run will test all API versions reported to be supported by the runtime. In the rare case that you are testing a runtime that is 1.1 conformant but not aiming to be 1.0 conformant (not recommended!), or if you want to verify that even basic functionality works when running at the OpenXR 1.1 level, you can pass:

Specify with the --graphicsPlugin <plugin> or -G <plugin> option.

If a testing step does not require you to run every supported graphics plugin, you may select an arbitrary one to use for that step. However, your runtime must: be able to pass all tests in that step with any supported graphics plugin. The Working Group has simply designated the tests in that step as not heavily depending on the graphics API.

The only case in which a graphics plugin is not specified, is if a "headless" or no-display extension is in use (via the -E option) and is being tested.

Some tests use a user-specified interaction profile.

Pass -I or --interactionProfiles followed by an interaction profile to test, without the leading /interaction_profile/ (omitted due to technical reasons).

Any extension required to permit the use of a given interaction profile specified this way (but not any additional binding paths in it) is automatically enabled, but only for the tests that use interaction profile specified in this way.

To verify that the runtime under test checks handle values for validity and returns XR_ERROR_INVALID_HANDLE if a handle provided is invalid, add --invalidHandleValidation (or -H, its shorthand version) to your conformance_cli command line.

To verify that the runtime under test checks the type member of structures passed and returns XR_ERROR_VALIDATION_FAILURE if the given enumerant is invalid, add --invalidTypeValidation (or -T, its shorthand version) to your conformance_cli command line.

One run and result file for the each graphics API supported by your runtime.

When using the following examples, omit any graphics API binding extensions your runtime does not support.

These are separate as a tester must evaluate the results and determine if each test is passed or failed.

These tests use the khr/simple_controller interaction profile to review instructions and sample output, as well as to indicate pass or fail.

One run and result file for the each graphics API supported by your runtime.

As previously, when using these examples, omit any graphics API binding extensions your runtime does not support.

These tests focus on interactive behavior of the runtime and are generally undemanding on the graphics API. While they should be possible to pass on all graphics APIs, a conformance submission only requires a report showing an overall pass on a single API.

These tests use the khr/simple_controller interaction profile to perform actions and indicate pass or fail.

These tests are selected with [actions][interactive]. Pass the interaction profile to test, without the leading /interaction_profile/, with the option -I

Note that if any extension or extensions are required for a specified interaction profile to be available, the CTS will automatically enable them for the appropriate tests.

One of these tests requires being able to activate all inputs specified (and not marked as "system") for an interaction profile.

For any supported interaction profiles that are valid for /user/gamepad rather than /user/hand/left and /user/hand/right, further filter the tests by specifying the tag [gamepad].

To clarify, if you support providing input for all components of one of the following interaction profiles, specify them as well.

One or more automated test result XML files, 1 per graphics API supported, therefore one or more of the following generated output files:

The output XML file(s) from running the interactive tests, 1 per supported graphics API, therefore one or more of the following generated output files:

At least one output file from running the interactive scenario tests on a single graphics API (more is better):

The output XML file(s) from running the interactive action tests, 1 per supported interaction profile, therefore one or more of the following generated output files. This list below are example files, each platform may have their own controllers though simple_controller is expected to be supported at a minimum.

Each test suite run starts by printing test configuration data, and ends by printing a "Report" showing details of the runtime and environment (extensions, etc) used in that run. A few tests produce console output in-between that does not show up in the result XML. It is important to have this data for the interpretation of the results.

Files containing the result of the commands git status and git log from the CTS directory:

If there were changes required to pass the conformance test suite, a diff of the changes from a tagged approved release of the suite must be included as well:

Alternately, the diff may be supplied in git format-patch format.

Note that only the tagged releases on the OpenXR-CTS repo are accepted without a diff: the latest such release will always be on the approved branch. The default devel branch is useful during development, but has not yet been voted on by the working group and is thus ineligible for submissions without a full diff. If the devel branch works better for you, you may consider encouraging the working group to tag a new release of conformance.

A file containing information regarding the submission called statement-<adopter>.txt

The actual submission package consists of the above set of files which must be bundled into a gzipped tar file named XR<API major><API minor>_<adopter><_info>.tgz. <API major> is the major version of the OpenXR API specification, <API minor> is the minor version of the OpenXR API specification. <adopter> is the name of the Adopting member company, or some recognizable abbreviation. The <_info> field is optional. It may be used to uniquely identify a submission by OS, platform, date, or other criteria when making multiple submissions. For example, a company XYZ may make a submission for an OpenXR 1.1 implementation named XR11_XYZ_PRODUCTA_Windows10.tgz.

Any test failures due to presumed bugs in the conformance tests not matching specification behavior should be submitted as issues with potential fixes against the conformance suite. Waivers are requested for test failures where the underlying platform fails to meet the expected specification behavior. These are requested in the statement file as described above. Enough detail should be provided such that submission reviewers can judge the potential impact and risk to the ecosystem of approving the submission.

Command line argument --pollGetSystem is available to retry xrGetSystem at startup of the CTS. Subsequent XrInstance creation is assumed to immediately have access to a system after this initial poll.

Some tests require that a begun session progresses through the lifecycle to XR_SESSION_STATE_FOCUSED without user interaction. These tests will synchronize their frame loop and start submitting frames and expect to proceed to FOCUSED. This may require donning a device or otherwise triggering a user presence sensor, for these tests to run fully automatedly.

Some tests require valid view pose (XR_VIEW_STATE_ORIENTATION_VALID_BIT & XR_VIEW_STATE_POSITION_VALID_BIT), though not necessarily tracked view position, relative to LOCAL reference space (XR_REFERENCE_SPACE_TYPE_LOCAL).

Unless otherwise specified, the behavior of the runtime must match the "when following suggested bindings" case. Any manual rebindings necessary to pass the tests must be discussed with the Working Group.

Assumes both a left and right hand input device (controller, hand interaction) is available unless arguments are specified to switch to a single hand mode.

Tests without the [interactive] flag do not depend on the presence of a controller.

The runtime must support khr/simple_controller to manually pass or fail each test through the …​/input/menu/click and …​/input/select/click input paths. The tester must evaluate the composed output and pass or fail the tests by comparing it to the provided expected result image. While /interaction_profile/khr/simple_controller is not formally required for conformance as of OpenXR 1.1, it is strongly encouraged. If it cannot be included for some reason, use of another interaction profile may be performed through the porting process.

Some interactive tests require tracked pose (XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT & XR_SPACE_LOCATION_POSITION_TRACKED_BIT) of VIEW reference space relative to LOCAL reference space (XR_REFERENCE_SPACE_TYPE_LOCAL). They must remain tracked throughout the test: if tracking is lost during the test, a spurious failure may be recorded, requiring a re-run of the tests.

Interactive tests typically show their instructions and sample output only while a boolean action, with suggested bindings for left and right …​/input/menu/click, is "true". Additionally, a boolean action with suggested bindings for left and right …​/input/select/click is typically used to report pass or fail, with both of those actions being "true" indicating failure. If pressing or holding the "menu" button brings up system UI, reviewing the instructions and expected behavior for interactive tests may be difficult. (It would be possible to remove this assumption through the porting process or other CTS development.) If pressing "select" while holding "menu" is not possible, it will be impossible to record an interactive test failure.

(Tests using InteractiveLayerManager have this behavior.)

A command line flag is available, --nonDisconnectableDevices, although it is primarily for debugging. Submissions that require it must explain their use of this flag in the conformance statement.

For some interactive tests, a person operating the test suite must use the in-test instructions to evaluate whether the behavior of the runtime matches the expected behavior according to the specification. If there is any doubt about whether the behavior in a test is conformant, consult with the Working Group.