5.6 Grading Test Results

Although a single test may examine multiple language issues, ACATS test results are graded "passed", "failed", or "not applicable" as a whole.

All customized, applicable tests must be processed by an implementation. Results must be evaluated against the expected results for each class of test. Results that do not conform to expectations constitute failures. The only exceptions allowed are discussed in 5.2.5; in such cases, processing the approved modified test(s) must produce the expected behavior. Any differences from the general discussion of expected results below for executable or non-executable tests are included as explicit test conditions in test prologues.

Warning or other informational messages do not affect the pass/fail status of tests.

Expected results for executable and non-executable tests are discussed in Sections 5.6.1, 5.6.2, and 5.6.3. Tests that are non-applicable for an implementation are discussed in 5.6.4. Withdrawn tests are discussed in 5.6.5.