5.6.2 Expected Results for Class B Tests

Class B tests are expected to be compiled but are not subject to further processing and are not intended to be executable. An implementation must correctly report each clearly marked error (the notation -- ERROR: occurs at the right hand side of the source). A multiple unit B test file generally will have errors only in one compilation unit per source file. Error messages must provide some means of specifying the location of an error, but they are not required to be in direct proximity with the -- ERROR: marking of the errors.

Some B-tests also include the notation -- OK to indicate constructs that must not be identified as errors. This is especially important since some constructs that were errors in [Ada83] are now legal in later versions of Ada.

Some B-tests exercise constructs whose correctness depends on source code that is textually separated (for example, a deferred constant and its full declaration). In these cases, it may be reasonable to report an error at both locations. Such cases are marked with -- OPTIONAL ERROR. These lines may be flagged as errors by some, but not all, implementations. Unless an optional error is marked as an error for the wrong reason, an error report (or lack of it) does not affect the pass/fail status of the test.

Some B-tests contain constructs where it would be reasonable for a compiler to report an error at one of several source locations. When it would not be appropriate for the ACATS to insist on a particular source location, all such source locations are marked with -- POSSIBLE ERROR: and an indication of which error set (if the test contains several) the location belongs to. In such cases, an implementation is considered to have properly reported the error if it reports an error at any of the places marked -- POSSIBLE ERROR: for a particular set. The implementation may flag more than one such place; this does not affect the pass/fail status of the test. However, the test is graded "failed" if no error is reported at any of the places marked -- POSSIBLE ERROR: for an error set.

A test is graded as "passed" if it reports each error in the test. The content of error messages is considered only to determine that they are indeed indications of errors (as opposed to warnings) and that they refer to the expected errors. The Reference Manual does not specify the form or content of error messages. In particular, a test with just one expected error is graded as "passed" if the test is rejected at compile time for any reason.

A test is graded as "failed" if it fails to report on each error in the test or if it marks legal code as incorrect.