ACATS 3.0 User's Guide
3.2.3 Coverage Guidelines for Specific Rule Categories
The Ada standard is
divided into various categories of rules. These rule categories have
different requirements and thus differ in what is the appropriate testing
philosophy. This translates into different types of test objectives for
each rule category.
This is generally introductory text, and does not require any testing.
This text sometimes includes definitions; these are handled as described
Testing that legal syntax is implemented is accomplished by all of the
other kinds of tests, especially by the user-oriented tests. Generally,
tests for illegal syntax have a very low value. Most compilers use a
syntax that is similar to that of the standard, and many use parsers
generated by tools, and thus the likelihood of errors is low. Moreover,
the number of possible bugs is essentially unlimited. There are two cases
where testing is warranted: First, rules that are given in text form
rather than BNF are treated as legality rules (see below). Second, the
Ada syntax grammar is not directly usable by either of the common technologies
for parsers (LL/recursive descent, and LR/LALR). In cases where the grammar
used by a compiler must necessarily be more general than that given in
the standard, the additional requirements are treated as legality rules.
Name Resolution Rules:
Name resolution rules should usually have a B-Test that checks that too
much information is not used for resolution. (For instance, whether a
type is limited or non-limited should generally not be used. Another
example is whether an access type is pool-specific or general access
usually should not be used.) It also may have a C-Test to check that
legal cases are allowed; but this objective is often handled by tests
for other test objectives (if that's the case, a note should point out
where they are tested).
Legality rules should almost always have an associated B-Test that checks
that illegal cases are detected. They may also have a C-Test to check
that legal cases are allowed; again, this is often handled by tests for
other test objectives. (They also can be covered in the B-Test with "OK"
Many Static Semantic rules are definitions, and as such can be handled
as described in 3.2.2
Many other static semantic rules are miscategorized by the standard.
For instance, most of the library packages are defined as static semantics,
even though most of the rules are really dynamic. These should be tested
as appropriate for the correct category.
A post-compilation rule should always have L-Tests to check that the
check is made in illegal cases. These checks are especially likely to
be omitted (because they're often complex to implement), so these tests
should have a high priority. In some cases, a C-Test may be needed to
ensure that legal cases work and the check does not go too far. Often
that will be covered by other tests, so a separate test is not needed
(a note should be provided to show where it is covered).
Dynamic semantics rules can be divided into two sub-categories.
First, these rules can define run-time checks. These checks should have
C-Tests that verify that the check is made and the appropriate exception
is raised. Careful: tests like this have to be carefully constructed
to avoid running afoul of 11.6 permissions.
Otherwise, these rules can define the normal operation of a construct.
These should have user-oriented C-Tests. By making these tests user-oriented,
the ACATS tests the normal usage of these features, not unusual corner-cases.
One special case: rules that say the execution of something "has
no effect" can't be usefully tested. Testing that nothing happens
is not interesting. And testing for the absence of an effect requires
guessing incorrect effects that might have occurred (which is unlikely
to actually detect anything), and thus any such test would have a very
Bounded (Run-Time) Errors:
These usually require C-Tests, but not always. If the bound includes
"work normally", then the bounded error is not usefully testable,
since the ACATS does not test implementation choices. Whether something
works or raises Program_Error is not interesting. In other cases, however,
since there is a bound on the behavior, it can be useful to ensure that
one of the prescribed results happens rather than havoc.
This is not testable. Since the language allows anything to happen, there
is no useful information to be gained from including this in a test.
Indeed, it's important to avoid any such case in an executable test.
These usually are dynamic requirements that should be tested like the
appropriate dynamic semantics rules. Otherwise, they should be tested
like a legality rule.
Usually permissions are not testable. Since the ACATS is not testing
the quality of implementations, the choices made by an implementation
are not appropriate things to test. So a test should be created only
if there is value in testing for implementations exceeding the permission.
Advice is usually not testable. Again, since the ACATS is not testing
quality of implementations, the choices made by an implementation are
not appropriate things to test. Moreover, advice need not be followed,
and other implementation choices can be made. Usually, if there are bounds
to what is acceptable, they are covered by other rules, and thus the
tests would be there.
Documentation Requirements and Metrics:
No tests are required for documentation requirements. Documentation existence
could be part of the conformity assessment process, but in any case it
is outside of the scope of the ACATS (which is testing the implementation,
not the documentation).
Notes and Examples:
These are non-normative text, and do not require any testing.