ACATS 3.0 User's Guide
5.6.1 Expected Results for Executable Tests
Executable tests (classes A, C, D, E) must be processed
by the compiler and any post-compilation steps (e.g., binder, partitioner)
without any errors. They must be loaded into an execution target and
run. Normal execution of tests results in an introductory message that
summarizes the test objective, possibly some informative comments about
the test progress, a final message giving pass / fail status, and graceful,
silent termination. They may report "PASSED", "TENTATIVELY
PASSED", "FAILED", OR "NOT APPLICABLE".
A test that fails to compile and bind, including
compiling and binding any foundation code on which it depends is graded
as "failed", unless the test includes features that need not
be supported by all implementations. For example, an implementation may
reject the declaration of a numeric type that it does not support. Allowable
cases are clearly stated in the Applicability Criteria of tests. Annex
M of the Ada Standard requires implementations to document such implementation-defined
A test that reports "FAILED" is graded
as "failed" unless the ACAL, and possibly the ACAA, determine
that the test is not applicable for the implementation.
A test that reports "PASSED" is graded
as "passed" unless the test produces the pass message but fails
to terminate gracefully (e.g., crashes, hangs, raises an unexpected exception,
produces an earlier or later "FAILED" message). This kind of
aberrant behavior may occur, for example, in certain tasking tests, where
there are multiple threads of control. A pass status message may be produced
by one thread, but another thread may asynchronously crash or fail to
A test that reports "NOT APPLICABLE" must
be run by the implementation and is graded as "not applicable"
unless it produces the not-applicable message and then fails to terminate
A test that reports "TENTATIVELY PASSED"
is graded as "passed" if the test results satisfy the pass/fail
criteria in the test. Normally, verification requires manual inspection
of the test output.
A test that fails to report, or produces only a partial
report, will be graded as "failed" unless the ACAL, and possibly
the ACAA, determine that the test is not applicable for the implementation.