ACATS 3.0 User's Guide
5.6.2 Expected Results for Class B Tests
Class B tests are expected to be compiled but are
not subject to further processing and are not intended to be executable.
An implementation must correctly report each clearly marked error (the
notation -- ERROR: occurs at the right hand side of the source).
A multiple unit B test file generally will have errors only in one compilation
unit per source file. Error messages must provide some means of specifying
the location of an error, but they are not required to be in direct proximity
with the -- ERROR: marking of the errors.
Some B-tests also include the notation -- OK
to indicate constructs that must not
be identified as errors.
This is especially important since some constructs that were errors in
are now legal in later versions
Some B-tests exercise constructs whose correctness
depends on source code that is textually separated (for example, a deferred
constant and its full declaration). In these cases, it may be reasonable
to report an error at both locations. Such cases are marked with --
OPTIONAL ERROR. These lines may be flagged as errors by some, but
not all, implementations. Unless an optional error is marked as an error
for the wrong reason, an error report (or lack of it) does not affect
the pass/fail status of the test.
Some B-tests contain constructs where it would be
reasonable for a compiler to report an error at one of several source
locations. When it would not be appropriate for the ACATS to insist on
a particular source location, all such source locations are marked with
-- POSSIBLE ERROR: and an indication of which error set (if
the test contains several) the location belongs to. In such cases, an
implementation is considered to have properly reported the error if it
reports an error at any of the places marked -- POSSIBLE ERROR:
for a particular set. The implementation may flag more than one such
place; this does not affect the pass/fail status of the test. However,
the test is graded "failed" if no error is reported at any
of the places marked -- POSSIBLE ERROR: for an error set.
A test is graded as "passed" if it reports
each error in the test. The content of error messages is considered only
to determine that they are indeed indications of errors (as opposed to
warnings) and that they refer to the expected errors. The Reference Manual
does not specify the form or content of error messages. In particular,
a test with just one expected error is graded as "passed" if
the test is rejected at compile time for any reason.
A test is graded as "failed" if it fails
to report on each error in the test or if it marks legal code as incorrect.