ACATS 3.0 User's Guide
5.7.2 Deviation from Expected Results - Petition & Review
Each test indicates in its prologue what it expects
from a conforming implementation. The result of processing a test is
acceptable if and only if the result is explicitly allowed by the grading
criteria for the test.
A user may challenge an ACATS test on the grounds
of applicability or correctness. A challenger should submit a petition
against the test program to an ACAL or to the ACAA, following the procedure
and the format presented in [Pro01]
petition must clearly state whether it is a claim that the test does
not apply to the implementation or that the test is incorrect. The petition
must indicate the specific section of code that is disputed and provide
a full explanation of the reason for the dispute.
ACALs will forward
petitions from their customers to the ACAA for decisions. The ACAA will
evaluate the petitioner's claims and decide whether:
the test is applicable to the implementation (i.e.,
deviation is not allowed);
the test is not applicable to the implementation
(i.e., deviation is allowed);
the test should be repaired (deviation is allowed,
and the modified test should be used for determining conformity assessment
the test should be withdrawn (deviation is allowed
and the test is not considered in determining conformity assessment results).
A deviation is considered to be a test failure unless
a petition to allow the deviation has been accepted by the ACAA.