🏡 index : cold-air-networks/mirror-webgpu-cts.git

# Terminology

Each test suite is organized as a tree, both in the filesystem and further within each file.

- _Suites_, e.g. `src/webgpu/`.
  - _READMEs_, e.g. `src/webgpu/README.txt`.
  - _Test Spec Files_, e.g. `src/webgpu/examples.spec.ts`.
    Identified by their file path.
    Each test spec file provides a description and a _Test Group_.
    A _Test Group_ defines a test fixture, and contains multiple:
    - _Tests_.
      Identified by a comma-separated list of parts (e.g. `basic,async`)
      which define a path through a filesystem-like tree (analogy: `basic/async.txt`).
      Defines a _test function_ and contains multiple:
      - _Test Cases_.
        Identified by a list of _Public Parameters_ (e.g. `x` = `1`, `y` = `2`).
        Each Test Case has the same test function but different Public Parameters.

## Test Tree

A _Test Tree_ is a tree whose leaves are individual Test Cases.

A Test Tree can be thought of as follows:

- Suite, which is the root of a tree with "leaves" which are:
  - Test Spec Files, each of which is a tree with "leaves" which are:
    - Tests, each of which is a tree with leaves which are:
      - Test Cases.

(In the implementation, this conceptual tree of trees is decomposed into one big tree
whose leaves are Test Cases.)

**Type:** `TestTree`

## Suite

A suite of tests.
A single suite has a directory structure, and many _test spec files_
(`.spec.ts` files containing tests) and _READMEs_.
Each member of a suite is identified by its path within the suite.

**Example:** `src/webgpu/`

### README

**Example:** `src/webgpu/README.txt`

Describes (in prose) the contents of a subdirectory in a suite.

READMEs are only processed at build time, when generating the _Listing_ for a suite.

**Type:** `TestSuiteListingEntryReadme`

## Queries

A _Query_ is a structured object which specifies a subset of cases in exactly one Suite.
A Query can be represented uniquely as a string.
Queries are used to:

- Identify a subtree of a suite (by identifying the root node of that subtree).
- Identify individual cases.
- Represent the list of tests that a test runner (standalone, wpt, or cmdline) should run.
- Identify subtrees which should not be "collapsed" during WPT `cts.https.html` generation,
  so that that cts.https.html "variants" can have individual test expectations
  (i.e. marked as "expected to fail", "skip", etc.).

There are four types of `TestQuery`:

- `TestQueryMultiFile` represents any subtree of the file hierarchy:
  - `suite:*`
  - `suite:path,to,*`
  - `suite:path,to,file,*`
- `TestQueryMultiTest` represents any subtree of the test hierarchy:
  - `suite:path,to,file:*`
  - `suite:path,to,file:path,to,*`
  - `suite:path,to,file:path,to,test,*`
- `TestQueryMultiCase` represents any subtree of the case hierarchy:
  - `suite:path,to,file:path,to,test:*`
  - `suite:path,to,file:path,to,test:my=0;*`
  - `suite:path,to,file:path,to,test:my=0;params="here";*`
- `TestQuerySingleCase` represents as single case:
  - `suite:path,to,file:path,to,test:my=0;params="here"`

Test Queries are a **weakly ordered set**: any query is
_Unordered_, _Equal_, _StrictSuperset_, or _StrictSubset_ relative to any other.
This property is used to construct the complete tree of test cases.
In the examples above, every example query is a StrictSubset of the previous one
(note: even `:*` is a subset of `,*`).

In the WPT and standalone harnesses, the query is stored in the URL, e.g.
`index.html?q=q:u,e:r,y:*`.

Queries are selectively URL-encoded for readability and compatibility with browsers
(see `encodeURIComponentSelectively`).

**Type:** `TestQuery`

## Listing

A listing of the **test spec files** in a suite.

This can be generated only in Node, which has filesystem access (see `src/tools/crawl.ts`).
As part of the build step, a _listing file_ is generated (see `src/tools/gen.ts`) so that the
Test Spec Files can be discovered by the web runner (since it does not have filesystem access).

**Type:** `TestSuiteListing`

### Listing File

Each Suite has one Listing File (`suite/listing.[tj]s`), containing a list of the files
in the suite.

In `src/suite/listing.ts`, this is computed dynamically.
In `out/suite/listing.js`, the listing has been pre-baked (by `tools/gen_listings_and_webworkers`).

**Type:** Once `import`ed, `ListingFile`

**Example:** `out/webgpu/listing.js`

## Test Spec File

A Test Spec File has a `description` and a Test Group (under which tests and cases are defined).

**Type:** Once `import`ed, `SpecFile`

**Example:** `src/webgpu/**/*.spec.ts`

## Test Group

A subtree of tests. There is one Test Group per Test Spec File.

The Test Fixture used for tests is defined at TestGroup creation.

**Type:** `TestGroup`

## Test

One test. It has a single _test function_.

It may represent multiple _test cases_, each of which runs the same Test Function with different
Parameters.

A test is named using `TestGroup.test()`, which returns a `TestBuilder`.
`TestBuilder.params()`/`.paramsSimple()`/`.paramsSubcasesOnly()`
can optionally be used to parametrically generate instances (cases and subcases) of the test.
Finally, `TestBuilder.fn()` provides the Test Function
(or, a test can be marked unimplemented with `TestBuilder.unimplemented()`).

### Test Function

When a test subcase is run, the Test Function receives an instance of the
Test Fixture provided to the Test Group, producing test results.

**Type:** `TestFn`

## Test Case / Case

A single case of a test. It is identified by a `TestCaseID`: a test name, and its parameters.

Each case appears as an individual item (tree leaf) in `/standalone/`,
and as an individual "step" in WPT.

If `TestBuilder.params()`/`.paramsSimple()`/`.paramsSubcasesOnly()` are not used,
there is exactly one case with one subcase, with parameters `{}`.

**Type:** During test run time, a case is encapsulated as a `RunCase`.

## Test Subcase / Subcase

A single "subcase" of a test. It can also be identified by a `TestCaseID`, though
not all contexts allow subdividing cases into subcases.

All of the subcases of a case will run _inside_ the case, essentially as a for-loop wrapping the
test function. They do _not_ appear individually in `/standalone/` or WPT.

If `CaseParamsBuilder.beginSubcases()` is not used, there is exactly one subcase per case.

## Test Parameters / Params

Each Test Subcase has a (possibly empty) set of Test Parameters,
The parameters are passed to the Test Function `f(t)` via `t.params`.

A set of Public Parameters identifies a Test Case or Test Subcase within a Test.

There are also Private Parameters: any parameter name beginning with an underscore (`_`).
These parameters are not part of the Test Case identification, but are still passed into
the Test Function. They can be used, e.g., to manually specify expected results.

**Type:** `TestParams`

## Test Fixture / Fixture

_Test Fixtures_ provide helpers for tests to use.
A new instance of the fixture is created for every run of every test case.

There is always one fixture class for a whole test group (though this may change).

The fixture is also how a test gets access to the _case recorder_,
which allows it to produce test results.

They are also how tests produce results: `.skip()`, `.fail()`, etc.

**Type:** `Fixture`

### `UnitTest` Fixture

Provides basic fixture utilities most useful in the `unittests` suite.

### `GPUTest` Fixture

Provides utilities useful in WebGPU CTS tests.

# Test Results

## Logger

A logger logs the results of a whole test run.

It saves an empty `LiveTestSpecResult` into its results map, then creates a
_test spec recorder_, which records the results for a group into the `LiveTestSpecResult`.

**Type:** `Logger`

### Test Case Recorder

Refers to a `LiveTestCaseResult` created by the logger.
Records the results of running a test case (its pass-status, run time, and logs) into it.

**Types:** `TestCaseRecorder`, `LiveTestCaseResult`

#### Test Case Status

The `status` of a `LiveTestCaseResult` can be one of:

- `'running'` (only while still running)
- `'pass'`
- `'skip'`
- `'warn'`
- `'fail'`

The "worst" result from running a case is always reported (fail > warn > skip > pass).
Note this means a test can still fail if it's "skipped", if it failed before
`.skip()` was called.

**Type:** `Status`

## Results Format

The results are returned in JSON format.

They are designed to be easily merged in JavaScript:
the `"results"` can be passed into the constructor of `Map` and merged from there.

(TODO: Write a merge tool, if needed.)

```js
{
  "version": "bf472c5698138cdf801006cd400f587e9b1910a5-dirty",
  "results": [
    [
      "unittests:async_mutex:basic:",
      { "status": "pass", "timems": 0.286, "logs": [] }
    ],
    [
      "unittests:async_mutex:serial:",
      { "status": "pass", "timems": 0.415, "logs": [] }
    ]
  ]
}
```