3.2.3 Coverage Guidelines for Specific Rule Categories

This is generally introductory text, and does not require any testing. This text sometimes includes definitions; these are handled as described in General.

Testing that legal syntax is implemented is accomplished by all of the other kinds of tests, especially by the user-oriented tests. Generally, tests for illegal syntax have a very low value. Most compilers use a syntax that is similar to that of the standard, and many use parsers generated by tools, and thus the likelihood of errors is low. Moreover, the number of possible bugs is essentially unlimited. There are two cases where testing is warranted: First, rules that are given in text form rather than BNF are treated as legality rules (see below). Second, the Ada syntax grammar is not directly usable by either of the common technologies for parsers (LL/recursive descent, and LR/LALR). In cases where the grammar used by a compiler must necessarily be more general than that given in the standard, the additional requirements are treated as legality rules.

Name resolution rules should usually have a B-Test that checks that too much information is not used for resolution. (For instance, whether a type is limited or non-limited should generally not be used. Another example is whether an access type is pool-specific or general access usually should not be used.) It also may have a C-Test to check that legal cases are allowed; but this objective is often handled by tests for other test objectives (if that's the case, a note should point out where they are tested).

Legality rules should almost always have an associated B-Test that checks that illegal cases are detected. They may also have a C-Test to check that legal cases are allowed; again, this is often handled by tests for other test objectives. (They also can be covered in the B-Test with "OK" cases.)

Many Static Semantic rules are definitions, and as such can be handled as described in 3.2.2, “General Coverage Guidelines”.

Many other static semantic rules are miscategorized by the standard. For instance, most of the library packages are defined as static semantics, even though most of the rules are really dynamic. These should be tested as appropriate for the correct category.

A post-compilation rule should always have L-Tests to check that the check is made in illegal cases. These checks are especially likely to be omitted (because they're often complex to implement), so these tests should have a high priority. In some cases, a C-Test may be needed to ensure that legal cases work and the check does not go too far. Often that will be covered by other tests, so a separate test is not needed (a note should be provided to show where it is covered).

Dynamic semantics rules can be divided into two sub-categories.

First, these rules can define run-time checks. These checks should have C-Tests that verify that the check is made and the appropriate exception is raised. Careful: tests like this have to be carefully constructed to avoid running afoul of 11.6 permissions.

Otherwise, these rules can define the normal operation of a construct. These should have user-oriented C-Tests. By making these tests user-oriented, the ACATS tests the normal usage of these features, not unusual corner-cases.

One special case: rules that say the execution of something "has no effect" can't be usefully tested. Testing that nothing happens is not interesting. And testing for the absence of an effect requires guessing incorrect effects that might have occurred (which is unlikely to actually detect anything), and thus any such test would have a very low value.

These usually require C-Tests, but not always. If the bound includes "work normally", then the bounded error is not usefully testable, since the ACATS does not test implementation choices. Whether something works or raises Program_Error is not interesting. In other cases, however, since there is a bound on the behavior, it can be useful to ensure that one of the prescribed results happens rather than havoc.

This is not testable. Since the language allows anything to happen, there is no useful information to be gained from including this in a test. Indeed, it's important to avoid any such case in an executable test.

These usually are dynamic requirements that should be tested like the appropriate dynamic semantics rules. Otherwise, they should be tested like a legality rule.

Usually permissions are not testable. Since the ACATS is not testing the quality of implementations, the choices made by an implementation are not appropriate things to test. So a test should be created only if there is value in testing for implementations exceeding the permission.

Advice is usually not testable. Again, since the ACATS is not testing quality of implementations, the choices made by an implementation are not appropriate things to test. Moreover, advice need not be followed, and other implementation choices can be made. Usually, if there are bounds to what is acceptable, they are covered by other rules, and thus the tests would be there.

No tests are required for documentation requirements. Documentation existence could be part of the conformity assessment process, but in any case it is outside of the scope of the ACATS (which is testing the implementation, not the documentation).

These are non-normative text, and do not require any testing.