ACATS 3.0 User's Guide
Section 1: Introduction
The Ada Conformity Assessment Test Suite (ACATS)
provides the official tests used to check conformity of an Ada implementation
with the Ada programming language standard (
ANSI/ISO/IEC
8652:1995 and later corrigenda and amendments). The ACATS User's
Guide is part of the ACATS and is distributed with the test programs
and testing support packages. It explains the contents and use of the
test suite.
The ACATS is an important part of the conformity
assessment process described in ISO/IEC-18009, Ada: Conformity of a Language
Processor
[ISO99]. This standard provides
a framework for testing language processors, providing a stable and reproducible
basis for testing. The Ada Resource Association (ARA) has sponsored an
instantiation of that process since October 1998. The process is managed
by the Ada Conformity Assessment Authority (ACAA).
Prior to the ISO standard, the U.S. Department of
Defense sponsored a similar conformity assessment process under the Ada
Joint Program Office (AJPO). The test suite for that process was known
as the Ada Compiler Validation Capability (ACVC).
The AJPO developed ACVC versions based on
ANSI/MIL-STD-1815A-1983,
ISO/8652:1987 (Ada 83), which were numbered 1.x where x ranged from
1 to 11. It later developed ACVC versions based on ANSI/ISO/IEC 8652:1995
(
[Ada95]), numbered 2.0, 2.0.1, 2.1, and
2.2.
When the ACAA took over Ada conformity assessment,
it adopted the ACVC as the basis for its test suite. The ACAA determined
to continue to use the same version numbering for the test suite in order
to avoid confusion. The version of the ACVC current at the time (2.1)
was initially used as ACATS 2.1. Later, the already developed but unreleased
ACVC 2.2 was released and used as ACATS 2.2. The ACAA later released
ACATS 2.3, ACATS 2.4, ACATS 2.5, and then ACATS 2.6 to include maintenance
changes and a few new tests.
This version of the ACATS is version 3.0. As with
ACATS 2.3 and later, this version was completely developed under the
auspices of the ACAA. ACATS 3.0 contains test programs to check for conformity
to new language features defined in ISO/IEC 8652:1995/AMD 1:2007 (
[Amend1]),
as well as test programs to check for conformity to language features
defined in earlier versions of Ada, including
[Ada95]
and
[Ada83]. Subsequent maintenance or
enhancement versions of the suite, if they are required, will be numbered
3.1, etc.
The ACATS User's Guide describes the set of ACATS
tests and how they are to be used in preparation for conformity assessment.
The formal procedures for conformity assessment are described in
[Pro01],
and the rules in that document govern all conformity assessments, notwithstanding
anything in this document that may be interpreted differently. Moreover,
this guide does not discuss specific requirements on processing of the
ACATS test suite, or submission and grading of results that an Ada Conformity
Assessment Laboratory (ACAL) may impose.
The User's Guide is intended to be used by compiler
implementers, software developers who maintain a version of the ACATS
as a quality control or software acceptance tool, and third-party testers
(e.g., Ada Conformity Assessment Laboratories).
Section
2 of the User's Guide
for ACATS 3.0 summarizes the changes between ACATS 2.6 and ACATS 3.0.
Section
3 describes test objectives and their
relationship to ACATS tests and to the rules of the Ada Standards documents.
Section
4 describes the configuration of the ACATS,
including a description of the ACATS software and delivery files. Section
5 provides step-by-step instructions for installing
and using the test programs and test support packages, and for grading
test results. The appendices include other information that characterizes
the ACATS 3.0 release, along with information on test construction.
Refer to
Annex F and Section
5.6 for the definition of an acceptable result
and the rules for grading ACATS 3.0 test program results. Section
5.7.2
provides instructions for submitting a petition against a test program
if a user believes that a deviation from the acceptable results for a
given test program is in fact conforming behavior.
