1.2 ACATS Coverage of Ada

The ACATS needs to test as many rules as possible in order to meet the goal of enhancing Ada software portability. After all, a rule that is not tested is far more likely to be incorrectly implemented than one that is tested.

Therefore the ACATS strives for complete coverage of the standard. Complete coverage means that every rule in the Ada standard has one or more associated tests that ensure that the rule is implemented properly.

Complete coverage is especially important for legality rules and runtime checks. It is easy for implementers to miss these rules, as their compiler may do something useful in the absence of the checks. But allowing such incorrect code can be a major portability problem when a program is moved to a compiler (including a later version of the same compiler) that properly implements the checks.

Of course, complete coverage does not mean that every sentence in the Ada standard has an associated test. There are many lines in the standard that are not rules at all, such as notes and examples. There are also many lines in the standard that are not testable; documentation requirements are but one example. Finally, there are rules in the standard that could be tested, but such tests are unlikely to find errors; for instance, most tests for illegal syntax are in this category as the test would need to guess what error might be made.

Complete coverage is a goal for the ACATS; it is not expected to be achieved in near future. Thus the test coverage analysis of the ACATS is a set of living documents, which will be updated with new information and tests with each new ACATS version.

Coverage testing will generally not test combinations of features, so problems that only manifest themselves in such combinations will not be detected. Tests designed primarily to cover language rules are most useful to prevent gross errors in implementations (such as forgetting to implement checks or features). As such, the ACATS also supplements those tests with tests written to emulate the patterns of use of Ada features. Such tests provide tests of common combinations of features, and ensure that common idioms are implemented properly.

A detailed description of how coverage is determined for the Ada standard can be found in Section 3.2.