WPCI 2%.B pz W"S^11>bbu"::Dg1:11bbbbbbbbbb11gggbuuuk1Xubuukuuuk111Rb:bbXbb1bb''X'bbbb:X1bXXXX;.;g:=::m:::mmmmm::::::mm:k1mubububububXubububub11111111bbbbbbbbbuXubbkbuXmmmmumububXXXXbububububbmbbbbbb:k:k::=kmmX:uXb'b:b:b:b'bmbbbb:::uXuXuXuXk:k:k:mbbbmbuXkXkXKQmmmm^b:kbbbbmbA@mmbmmbmmmmmmm:b:mmmbbmmmmmmmmmmmmXXmmmmmmmmmmmmmmmmmmcm`m`mm`m:mmmmmm}}}mjjmmmmmmmmmmmmmmm0mm}mmmmmmmmmmmmmmmmmmmmmmm}Mmmmmmmmmmmmmjmmmtmmmmmmmmm`'mmm`mmjmlWmmmmmmmmmmmmmmmmmmmW`mmmmjmM-lvetica#|x`H1`D4PkCQMS PS Jet Plus /800 II QPJPII.PRSPl`D4PkCg2W_qr|xHelveticaCourier@ ,`H1`D4PkCmQrrr r  @C2 KLG@ ,`H1`D4PkCmQrrr r  @C;,>>> >  @C ` X` hp x (#%'H   x|@  3'3'Standard6'6'StandardC6QMS $=R-   PART 2: ABSTRACT TEST SUITE SPECIFICATION Introduction  ` ( HThis part of the Recommendation provides a common approach to the specification of "OSI or related CCITT XSeries or TSeries" (hereafter abbreviated to "OSI*") conformance test suites at a level which is  ( independent of the means of executing those test suites (hereafter called "abstract test suites"). This level of abstraction is suitable for standardization and facilitates the comparison of results produced by different organizations which run the corresponding executable test suites. HSection 1 recalls that there are requirements put on the OSI* protocol specifiers which have to be fulfilled before there can be an objective basis for the process of developing an abstract test suite. The need for consistent conformance sections and for PICS and PIXIT proformas in OSI* protocol Recommendations* is expressed. HSection 2 describes the process of developing an abstract test suite, including the design criteria to be used and guidance on its structure and coverage. The possible abstract test methods are defined and guidance is given to help the test suite specifier to decide which method(s) to use in the production of a particular test suite. The relationship between abstract test suites for different methods is provided by a generic test suite which is independent of the test method. A test notation is defined and requirements and guidance are given on its use for specifying both generic and abstract test cases. These include the subdivision of test cases into test steps and the assignment of verdicts to outcomes. HThe test suite specifier is also required to provide information to the test realizers, particularly on the constraints governing test case selection and ordering. HFinally, guidance and requirements are given on test suite maintenance. 1.HScope and field of application HThis part of the Recommendation specifies the requirements and provides guidance for the production of systemindependent conformance test suites for one or more OSI* Recommendations*. In particular, it applies to the production of all conformance test suite Recommendations* for OSI* protocols. HIt applies to the production of conformance test cases which check the adherence of an implementation to the relevant static and/or dynamic conformance requirements by controlling and observing protocol behaviour. The abstract test methods included in this Recommendation are in fact capable of being used to specify any test case which can be expressed abstractly in terms of control and observation of protocol data units, abstract service primitives and abstract local primitives. Nevertheless, for some protocols, test cases may be needed which cannot be expressed in these terms. The specification of such test cases is outside the scope of this Recommendation, although those test cases may need to be included in a Recommendation* for a conformance test suite. Note For example, some static conformance requirements related to an application service may require testing techniques which are specific to that particular application. HThe production of conformance test suites for multipeer or physical layer protocols is outside the scope of this Recommendation. HThe relation between abstract test specification and formal description techniques is outside the scope of this Recommendation. F-Ԍ 2.HReferences  h RecommendationX.200 Reference model of open systems interconnection for CCITT applications. (See also ISO7498.) RecommendationX.214 Transport service definition for open systems interconnection for CCITT applications. (See also ISO8072.) RecommendationX.224 Transport protocol specification for open systems interconnection for CCITT applications. (See also ISO8073.) Recommendation X.210 Open systems interconnection layer service definition conventions. (See also ISOTR8509.) RecommendationX.208 Specification of Abstract Syntax Notation One (ASN.1). (See also ISO8824.) RecommendationX.209 Specification of basic encoding rules for Abstract Syntax Notation One (ASN.1). (See also ISO8825.) RecommendationX.290/1 OSI conformance testing methodology and framework for protocol Recommendations for CCITT applications Part1: General concepts. ISO85714 Information processing systems Open Systems Interconnection File protocol specification. 3.HDefinitions HFor the purposes of this part of the Recommendation, all the definitions in Part1 of the Recommendation apply. 4.HAbbreviations HFor the purposes of this Recommendation the abbreviations given in section4 of Part1 of this Recommendation apply. The following abbreviations also apply to this Recommendation:  h HHASP : the abstract service primitive on the side of the service provider remote from the IUT!H  `  HCM : coordinated multilayer (test method) HCS : coordinated singlelayer (test method) HCSE : coordinated singlelayer embedded (test method) HDM : distributed multilayer (test method) HDS : distributed singlelayer (test method) HDSE : distributed singlelayer embedded (test method) HFDT : formal description technique HLM : local multilayer (test method) HLS : local singlelayer (test method) HLSE : local singlelayer embedded (test method) HRM : remote multilayer (test method) F-Ԍ HRS : remote singlelayer (test method) HRSE : remote singlelayer embedded (test method) HTTCN: tree and tabular combined notation HYL : loopback (test method) HYT : transverse (test method). 5.HCompliance  `  5.1HA protocol Recommendation* which complies with this part of this Recommendation shall satisfy all the requirements stated in section 1.   Note Such compliance is a precondition for the protocol Recommendation* to be an effective basis for conformance testing of implementations. 5.2HAn abstract test suite specification which complies with this part of this Recommendation HHa)X  shall be a conformance test suite;%  `   ` ( HHb)X  shall be specified in a test notation standardized by ISO or CCITT;ư"  (`   `  HHc)X  shall satisfy all the requirements stated in section2.  `   ` H 5.3HIt is recommended that the test notation used be the Tree and Tabular Combined Notation (TTCN). If TTCN is used, the abstract test suite shall comply with all the requirements in AnnexD. Section 1 Requirements on protocol specifiers 6.HConformance requirements in OSI* Recommendations* 6.1HIntroduction HThe meaning of conformance in OSI* is discussed in Part1 of this Recommendation. It is necessary that there be an unambiguous and objective understanding of the conformance requirements of an OSI* protocol or transfer syntax Recommendation*, as a prerequisite to the production of an abstract test suite for that Recommendation*. This section states the requirements on protocol specifiers to ensure that there is such an understanding of the conformance requirements. 6.2HGeneral requirements 6.2.1HA clear distinction shall be made between static and dynamic conformance requirements. To avoid ambiguity, they should be stated separately from one another. 6.2.2HIt shall be clear what conformance to the Recommendation* means, in the sense of what shall be done, what is permitted but not mandatory, and what shall not be implemented, to conform to the Recommendation*. 6.2.3HIt shall always be decidable whether an instance of communication conforms dynamically or not. HFor example, one should be able to look at a record of PDU activity and decide whether it is valid.  F-Ԍ6.2.4HRequirements on the need to produce and content of a PICS shall be stated separately from the requirements on the protocol implementation itself. 6.3HConformance sections 6.3.1HEach OSI* protocol and transfer syntax Recommendation* shall include a conformance section, which shall be expressed clearly and unambiguously. 6.3.2HConformance sections shall distinguish between the following categories of information that they may contain:  H HHa)X  references to sections which state dynamic conformance requirements;  `   `  HHb)X  static conformance requirements concerning the protocol implementation;X   `   `  HHc)X  static conformance requirements concerning multilayer dependencies;  `   `  HHd)X  what has to be stated in the PICS concerning (b) above;  `   `  HHe)X  what has to be stated in the PICS concerning (c) above;  `   ` X HHf)X  what other information shall be provided (e.g. to assist testing) and whether this should be in the PICS or elsewhere.ƀ%  X`   ` ( 6.3.3HIn connectionoriented protocol Recommendations*, the conformance section should also include:  (8 HHa)X  the option to support either the initiation of a connection or the acceptance of a connection, or both;Ơ#  8`   `  HHb)X  the requirement to be able to accept all correct sequences of PDUs received from peers, and respond with correct PDU sequences appropriate to the defined state of the connection;(#  `   ` H HHc)X  the requirement to be able to respond correctly to all incorrect sequences of PDUs received, the response being appropriate to the defined state of the connection.Ɛ$  H`  6.4HAdditional guidance for new protocol Recommendations*  ` H HIt is recognized that although existing protocol Recommendations* can be improved by the progression of an addendum to add a PICS proforma and make the conformance section align with the requirements stated above, it is unrealistic to expect any greater improvement. However, new protocol Recommendations* should be developed using the additional guidance given in AnnexB to make the task of understanding the conformance requirements easier and less prone to ambiguity. 7.HPICS proformas 7.1HRequirements on PICS proformas 7.1.1HThe specific requirements to be met by suppliers in respect of each PICS they provide shall normally be stated in the relevant protocol Recommendation*. The specification of these requirements shall include a PICS proforma. In exceptional circumstances, the PICS proforma may be found in the abstract test suite Recommendation* rather than in protocol Recommendation*; in particular, this applies when the PICS proforma has to cover different versions of the same protocol coming from both ISO and CCITT. In normal circumstances, the PICS proforma should be found in an annex to the protocol Recommendation* and F- referenced in the conformance section, and, if necessary, progressed as an addendum rather than as part of the original Recommendation*.  Hh Note A PICS for a specific protocol implementation will need to be accompanied by administrative and documentary information relating to the supplier, the system and its intended environment. 7.1.2HThe PICS proforma shall be in the form of a questionnaire or checklist to be completed by the supplier or implementor of an implementation of the relevant OSI* protocol. 7.1.3HThe PICS proforma shall cover all functions, elements of procedure, parameters, options, PDUs, timers, multilayer dependencies and other capabilities identified in the protocol Recommendation*, including optional and conditional capabilities. It is desirable, where practicable, to include all mandatory features. There shall be a welldefined mapping between static conformance requirements and the PICS proforma and there shall be consistency between them. 7.1.4HThe PICS proforma shall be preceded by a section that states:  h HH"The supplier of a protocol implementation which is claimed to conform to this Recommendation shall complete the following Protocol Implementation Conformance Statements (PICS) proforma and shall provide the information necessary to identify uniquely both the supplier and the implementation."%H  `   ` 8 7.1.5HThe name, version and date of the relevant protocol Recommendation* shall be stated on each page of the PICS proforma. 7.1.6HThe PICS proforma for a specific protocol shall contain:  88 HHa)X  explanations of special symbols, abbreviations, special terms, together with appropriate references;Ơ#  8`   ` 8 HHb)X  explicit instructions for completing and interpreting the PICS;Ơ#  8`   ` H HHc)X  provision for mentioning any mandatory feature that has not been implemented, with the appropriate rationale;Ɛ$  H`   ` 8 HHd)X  one or more tables (or other kinds of form as necessary) to be completed to state the capabilities of the implementation in detail, including:Ơ#  8`   `  HHX 1)Xname of the feature, PDU type, timer, parameter, and other capabilities;$  `   `  HHX 2)Xa column stating whether each is mandatory, optional, negotiable, or conditional;!  `   `  HHX 3)Xwherever feasible, a column giving references to the relevant sections in the Recommendation;H!  `   `  HHX 4)Xa column giving the permitted range or values, if appropriate;  `   `  HHX 5)Xa column to be filled in to give the supported values or range of values, if appropriate;(#  `   `  HHX 6)Xa column to be filled in to state if each capability has been implemented;(#  `   ` H HHe)X  the proforma shall give a clear indication of the preferred data types (e.g. number bases, string types, octets, bits, seconds, F- minutes, etc.) for responses.Ɛ$  H`   ` X 7.1.7HThe PICS proforma shall use the following abbreviations as appropriate, unless they conflict with the abbreviations used in the specific protocol Recommendation*: Hm: mandatory Ho: optional Hc: conditional Hn: negotiable (using the protocol) Hx: exclusion of capability H: not applicable Hs: ability to send Hr: ability to receive 7.2HGuidance on PICS proformas HAppendixIII provides some general purpose examples to give guidance on the construction of PICS proformas. Section 2 Requirements on abstract test suite specifiers 8.HTest suite production process HIn order to present the requirements and general guidance for abstract test suite specification, it is useful to assume a normal form of the process of test suite production. This section describes the process in just such a normal form. Abstract test suite specifiers are not required to follow this normal form exactly, however, they are recommended to use a similar process involving the same steps, possibly in a different order. HFor the purposes of this Recommendation, the abstract test suite production process is assumed to be as follows:  X HHa)X  study the relevant Recommendation(s)* to determine what the conformance requirements (including options) are which need to be tested, and what needs to be stated in the PICS (see section9);8"  `   `  HHb)X  decide on the test groups which will be needed to achieve the appropriate coverage of the conformance requirements and refine the test groups into sets of test purposes (see section10);(#  `   ` ( HHc)X  specify generic test cases for each test purpose, using some appropriate test notation (see section11);ư"  (`   ` 8 HHd)X  choose the test method(s) for which the complete abstract test cases need to be specified, and decide what restrictions need to be placed on the capabilities of the lower tester and (if appropriate to the chosen test method(s)) the upper tester and test coordination procedures (see section12);Ơ#  8`   `  HHe)X  choose the test notation for specifying the complete abstract test cases, and specify the complete abstract test cases, including the test step structure to be used (see section13);(# F-Ԍ `   ` 8 HHf)X  specify the interrelationships among the test cases and those between the test cases and the PICS and, as far as possible, the PIXIT, in order to determine the restrictions on the selection and parameterization of test cases for execution, and on the order in which they can be executed (see section14);Ơ#  8`   `  HHg)X  consider the procedures for maintaining the abstract test suite (see section15).$  `   `  HThe remainder of this section provides requirements and guidance which relate to each step in the above process. 9.HDetermining the conformance requirements and PICS 9.1HIntroduction HBefore an abstract test suite can be specified, the test suite specifier shall first determine what the conformance requirements are for the relevant Recommendation(s)* and what is stated in the PICS proforma concerning the implementation of those Recommendation(s)*. 9.2HHConformance requirements%H HSection1 of this part of the Recommendation specifies the requirements to be met by protocol specifiers as a prerequisite to the production of an abstract test suite for a particular protocol. HIn practice, early OSI* Recommendations* might not contain a clear specification of all the relevant conformance requirements. In particular, the static conformance requirements might be badly specified or even omitted. In such cases, the test suite specifier shall contribute to the production of an amendment or addendum to the relevant Recommendation(s)* to clarify the conformance requirements. If, however, an abstract test suite has to be produced in advance of the conformance requirements being clarified in the relevant Recommendation(s)*, then the test suite specifier shall adopt the shortterm solution given in AnnexC and state clearly in the test suite Recommendation* what the implications of this are (i.e. what is assumed to be mandatory, what conditional and what the conditions are, and what is assumed to be optional). 9.3HPICS proforma HIf no PICS proforma is included in a relevant protocol Recommendation*, the test suite specifier shall provide a PICS proforma to be processed as an addendum to that Recommendation*, or in exceptional circumstances (as discussed in 7.1.1) as an annex to the abstract test suite Recommendation*. Note Progressing an addendum to an existing protocol Recommendation* may well be faster than progressing the abstract test suite Recommendation* because the PICS proforma is likely to be less controversial than the test suite and therefore is likely to require fewer revisions before a final text can be agreed. 10.HTest suite structure 10.1HBasic requirements HAn abstract test suite shall comprise a number of test cases. The test cases shall be grouped into test groups, if necessary nested. The structure shall be hierarchical in the sense that an item at a lower level shall be completely contained within a higher level item. The structure need not, however, be strictly hierarchical in the sense that any one test case may occur in more than one test suite or test group. Similar test groups may occur in more than one higher level F- test group or test suite. HThe abstract test suite specifier shall ensure that a subset of the test purposes of each abstract test suite is concerned with capability testing, and another subset is concerned with behaviour testing. This need not lead to distinct test cases for behaviour and capability testing because it may be possible to use the same test steps for both a behaviour test purpose and for a capability test purpose. The test suite specifier shall provide an explanation of how the test purposes are derived from or relate to the protocol Recommendation*. The test suite specifier shall also provide a summary of the coverage achieved by the test suite. 10.2HThe test group structure HIn order to ensure that the resulting abstract test suite provides adequate coverage of the relevant conformance requirements, the test suite specifier is advised to design the test suite structure in terms of nested test groups in a top down manner (see Figure1). HThere are many ways of structuring the same test suite into test groups; no one way is necessarily right and the best approach for one test suite may not be appropriate for another test suite. Nevertheless, the test suite specifier shall ensure that the test suite includes test cases for whichever of the following categories is relevant:   HHa)X  capability tests (for static conformance requirements);  `  HHb)X  behaviour tests of valid behaviour;x HHc)X  behaviour tests of syntactically invalid behaviour;x HHd)X  behaviour tests of inopportune behaviour;x HHe)X  tests focusing on PDUs sent to the IUT;x HHf)X  tests focusing on PDUs received from the IUT;x  `  HHg)X  tests focusing on interactions between what is sent and received;X   `  HHh)X  tests related to each protocol mandatory feature;x  `  HHi)X  tests related to each optional feature which is implemented;8"  `  HHj)X  tests related to each protocol phase;x  ` ( HHk)X  variations in the test event occurring in a particular state;ư"  (`  HHl)X  timing and timer variations;x HHm)X  PDU encoding variations;x HHn)X  variations in values of individual parameters;x HHo)X  variations in combinations of parameter values.x  ` H HThis list is not exhaustive; additional categories might be needed to ensure adequate coverage of the relevant conformance requirements for a specific test suite. Furthermore, these categories overlap one another and it is the task of the test suite specifier to put them into an appropriate hierarchical structure. HThe following structure is an example of a singlelayer test suite, F-  provided for guidance: HHA.X  Capability testsƐ$  H`  HHX A.1XMandatory featuresx  `  HHX X  H  A.2Optional features said by the PICS to be supported!  `  HHB.X  Behaviour tests: response to valid behaviour by peer implementationx H B.1Connection establishment phase (if relevant) HHX B.1.1 Focus on what is sent to the IUTx  ` p HH HHX X  B.1.1.1 Test event variation in each stateh HHX X  B.1.1.2 Timing/timer variationh HHX X  B.1.1.3 Encoding variationh  p H   B.1.1.4 Individual parameter value variationX H  B.1.1.5 Combination of parameter values HHX B.1.2 Focus on what is received from the IUTX   `  H   © substructured as B.1.1x H B.1.3 Focus on interactions H   © substructured as B.1.1x H B.2Data transfer phase H  substructured as B.1 H B.3Connection release phase (if relevant) H  substructured as B.1  `  HC.  Behaviour tests: response to syntactically invalid behaviour by HHX peer implementation $  `  H C.1Connection establishment phase (if relevant) H C.1.1 Focus on what is sent to the IUT  ` p H   C.1.1.1 Test event variation in each stateh  p H  C.1.1.2 Encoding variation of the invalid event H  C.1.1.3 Individual invalid parameter value H   variation H  C.1.1.4 Invalid parameter value combination H   variation H C.1.2 Focus on what the IUT is requested to send H  C.1.2.1 Individual invalid parameter values C.1.2.2 Invalid combinations of parameter values H C.2Data transfer phase H  substructured as C.1 H C.3Connection release phase (if relevant)  F- ԌH  substructured as C.1   HHD.X  Behaviour tests: response to inopportune events by peer implementationX   `  HHX D.1XConnection establishment phase (if relevant)x HHX X D.1.1 Focus on what is sent to the IUTx  ` p HHX X  D.1.1.1 Test event variation in each stateh HHX X  D.1.1.2 Timing/timer variationh  p` H  D.1.1.3 Special encoding variations  `  H  D.1.1.4 Major individual parameter value variations H  D.1.1.5 Variation in major combination of parameter Lb"IIb"IIb"IIb"I values H D.1.2 Focus on what is requested to be sent by the IUT H   substructured as D.1.1 H D.2Data transfer phase H  substructured as D.1 H D.3Connection release phase (if relevant) H  substructured as D.1 HIf the test suite is to cover more than one layer, then a singlelayer test suite structure such as this could be replicated for each layer  h concerned. In addition, a correspondingly detailed structure could be produced for testing the capabilities and behaviour of multiple layers taken as a whole, including the interaction between the activities in adjacent layers. 10.3HTest purposes HThe test suite specifier shall ensure that, for each test case in an abstract test suite, there is a clear statement of the test purpose. It is suggested that these test purposes should be produced as the next refinement of the test suite, after its structure in terms of test groups has been defined. The test purposes could be produced directly from studying those sections in the relevant Recommendation(s)* which are appropriate to the test group concerned. For some test groups, the test purposes might be derivable directly from the protocol state table; for others, they might be derivable from the PDU encoding definitions or the descriptions of particular parameters, or from text which specifies the relevant conformance requirements. Alternatively, the test suite specifier could employ a formal description of the protocol(s) concerned and derive test purposes from that by means of some automated method. HWhatever method is used to derive the test purposes, the test suite specifier shall ensure that they provide an adequate coverage of the conformance requirements of the Recommendation(s)* concerned. There shall be at least one test purpose related to each distinct conformance requirement. HIn addition, it is possible to give guidance on the meaning of "adequate coverage" with reference to the above example. In order to express this, a shorthand notation will be used: the letter "x" will represent all appropriate values for the first digit in the test group identifier, and similarly "y" for the second digit, so that B.x.y.1 stands for B.1.1.1, B.1.2.1, B.1.3.1, B.2.1.1, B.2.2.1, B.2.3.1, B.3.1.1, B.3.2.1 and B.3.3.1. With this notation, a minimum "adequate coverage" for the above example is considered to be as follows:  F- ԌHHa)X  for capability test groups (A.1, A.2):p&  h`   ` ( HHX 1)Xat least one test purpose per relevant feature, class orư" HHX X subset;ư"  (`   `  HHX 2)Xat least one test purpose per relevant PDU type and each major variation of each such type, using "normal" or default values for each parameter;(#  `   `  HHb)X  for test groups concerned with test event variation in each state (B.x.y.1, C.x.1.1, D.x.y.1):%  `   `  HHX X   at least one test purpose per relevant state/event combination;X   `   `  HHc)X  for test groups concerned with timers and timing (B.x.y.2, D.x.y.2):!  `   ` H HHX 1)Xat least one test purpose concerned with the expiry of each defined timer;Ɛ$  H`   `  HHX 2)Xat least one test purpose concerned with very rapid response for each relevant type of PDU;   `   ` H HHX 3)Xat least one test purpose concerned with very slow response for each relevant type of PDU;Ɛ$  H`   ` ( HHd)X  for test groups concerned with encoding variations (B.x.y.3, C.x.1.2, D.x.y.3):ư"  (`   `  HHX X   at least one test purpose for each relevant kind of encoding variation per relevant PDU type;   `   ` H HHe)X  for test groups concerned with valid individual parameter values (B.x.y.4, D.x.y.4):Ɛ$  H`   `  HHX 1)Xfor each relevant integer parameter, test purposes concerned with the boundary values and one randomly selected midrange value;X   `   ` 8 HHX 2)Xfor each relevant bitwise parameter, test purposes for as many values as practical, but not less than all the "normal" or common values;Ơ#  8`   `  HHX 3)Xfor other relevant parameters, at least one test purpose concerned with a value different from what is considered "normal" or default in other test groups;(#  `   `  HHf)X  for test groups concerned with invalid individual parameter values (C.x.1.3, C.x.2.1):8"  `   `  HHX 1)Xfor each relevant integer parameter, test purposes concerned with invalid values adjacent to the allowed boundary values plus one other randomly selected invalid value;X   `   ` 8 HHX 2)Xfor each relevant bitwise parameter, test purposes for as many invalid values as practical;Ơ#  8`   ` ( HHX 3)Xfor all other relevant types of parameter, at least one test purpose per parameter;ư"  (`  F- Ԍ `  HHg)X  for test groups concerned with combinations of parameter values (B.x.y.5, C.x.1.4, C.x.2.2, D.x.y.5):$  `   `  HHX 1)Xat least one test purpose per "critical" value pair;   `   `  HHX 2)Xat least one test purpose per pair of interrelated parameters to test a random combination of relevant values.X   `  11.HGeneric test case specification 11.1HIntroduction  ` ( HThe test suite specifier is recommended to specify a generic test suite, particularly if there is an intention to produce more than one abstract test suite.  ( HA generic test suite shall consist of one generic test case for each test purpose. Each generic test case will be a refinement of its test purpose, which can be used as a common root of corresponding abstract test cases for different test methods.  X HIf a generic test suite is produced in advance of abstract test suites, then it will be a useful step in the design process. If the generic test suite is produced after the production of at least one abstract test suite, then it will provide a means of relating different test suites to one another and analyzing where there may be gaps in their coverage. 11.2HDescription of generic test cases HA generic test case consists of a test description of an "initial state" [of the test body] and a specification of the test body in a standardized test notation. The "initial state" includes not only the protocol state, but also any necessary information concerning the state of the SUT and the testing environment. HThe test body is that part of a test case in which verdicts related to the test purpose are assigned to foreseen outcomes. The test body:  X HHa)X  shall be defined in the style of either the Remote or Distributed test method; %  `   `  HHNote Generic test cases may use the full expressive power of these methods (see 12.3.3 for details), including the use of abstract local primitives.$H  `   ` ( HHb)X  shall assign verdicts to all possible outcomes; all outcomes yielding "pass" verdicts shall be explicitly identified, whereas all outcomes yielding "fail" or "inconclusive" verdicts shall be either identified or categorized (which may include categorization of default verdicts);ư"  (`   `  HHc)X  shall be described using a standardized test notation; the Tree and Tabular Combined Notation (TTCN) (defined in AnnexD) is recommended.$  `  11.3HRelation of generic to abstract test cases  ` H HFor a particular abstract test method there may be many abstract test cases that can be derived from a single generic test case. A major difference between an abstract test case and a generic test case is that the abstract test case includes specifications of a preamble and a postamble. The preamble starts from a chosen stable state and leads to the required initial state of the test F-  body. The postamble starts from the end of the test body and returns to a chosen stable state. HThe preamble and postamble may be realized in different ways depending on the degree of control and observation provided by the test method used, or  Hh on the variety of different possible stable states which the derived abstract test case can start from and end in. These abstract test cases are simply different ways of achieving the same test purpose. HIn addition, the test body of an abstract test case may be different from the corresponding generic test case if the test method used for the abstract test case is different from that used for the generic test case. HIf a generic test suite is produced, it shall be used as the means of relating corresponding abstract test suites for different abstract test methods. 12.HAbstract test methods 12.1HIntroduction HEach abstract test suite shall be specified in terms of the control and observation of events in accordance with one of the abstract test methods defined in this section. The chosen test method determines the points at which control and observation shall be specified and the categories of events which shall be used (e.g. (N1)ASPs, (N)PDUs). 12.2.HGeneral specification of the abstract test methods 12.2.1HEndsystem IUTs HFor IUTs within endsystem SUTs there are four categories of abstract test methods, one local, and three external ones which assume the lower tester is located remotely from the SUT and connected to it by a link or network. HFor generality, the test methods are described with reference to an IUT in which the highest layer is numbered "Nt" (for "top") and in which the lowest layer protocol to be tested is numbered "Nb" (for "bottom"). The IUT may implement protocols in layers lower than "Nb", but these are not of interest in the test method descriptions. The description applies to singlelayer IUTs by taking Nt to be equal to Nb. 12.2.2HAbstract local primitives HAbstract test suite specifiers may, if necessary, define a set of abstract local primitives (ALPs) to be used in specifying the test suite. An ALP is an abbreviation for a description of control and/or observation to be performed by the upper tester, which cannot be described in terms of ASPs but which initiate events or state changes defined within the relevant protocol Recommendation(s)*. ALPs may be used with any abstract test method for end system IUTs, but, for simplicity, will not be shown in any of the figures which illustrate those methods. HAny test case which uses an ALP shall be optional in the abstract test suite. 12.2.3HThe local test methods Abbreviation: L HIn this method:  h( HHa)X  the abstract test suite is specified in terms of control and observation of (Nbĩ1)ASPs and (Nb) to (Nt)PDUs;ư"  (`  F-Ԍ `  HHb)X  the abstract test suite is also specified in terms of control and observation of (Nt)ASPs;%  `   `  HHc)X  the abstract test suite may also be specified in terms of control and observation by the upper tester of abstract local primitives (ALPs);%  `   `  HHd)X  the method requires access to the lower and upper boundaries of the IUT and a mapping between the specified ASPs and their realization within the SUT;$  `   `  HHe)X  the requirements for the test coordination procedures are specified in the abstract test suite but no assumption is made regarding their realization;H!  `   `  HHf)X  the upper and lower testers are required to achieve control and observation of the specified ASPs and the required test coordination procedures. They are assumed to be integrated into the SUT.$  `  HThis method is illustrated in Figure1. 12.2.4HExternal test methods 12.2.4.1 The distributed test method Abbreviation: D HIn this method:  ` ( HHa)X  the abstract test suite is specified in terms of control and observation of (Nbĩ1)ASP"s and (Nb) to (Nt)PDUs;ư"  (`   `  HHb)X  the abstract test suite is also specified in terms of control and observation of (Nt)ASPs; the method requires access to the upper boundary of the IUT and a mapping between the (Nt)ASPs and their realization within the SUT;%  `   `  HHc)X  the abstract test suite may also be specified in terms of control and observation by the upper tester of ALPs;%  `   `  HHd)X  the requirements for the test coordination procedures are specified in the abstract test suite but no assumption is made regarding their realization;H!  `   `  HHe)X  the upper tester is required to achieve control and observation of the specified (Nt)ASPs and to achieve the effects of the required test coordination procedures; no other assumptions are made;$  `   `  HHf)X  the lower tester is required to achieve control and observation of specified (Nb)ASP"s and specified PDUs and to achieve the required test coordination procedures.$  `  HThis method is illustrated in Figure2.  ` ( HFIGURE 1/X.290, PART 2 FIGURE 2/X.290 PART 2  (H  The local test method The distributed test method HFIGURE 3/X.290, PART 2 FIGURE 4/X.290, PART 2 The coordinated test method The remote test method  ) 12.2.4.2 The coordinated test method Abbreviation: C HIn this method:  H( HHa)X  the abstract test suite is specified in terms of control and observation of (Nbĩ1)ASP"s, (Nb) to (Nt)PDUs and test management PDUs (TMPDUs);ư"  (`   `  HHb)X  (Nt)ASPs need not be used in the specification of the abstract test suite; no assumption is made about the existence of an upper boundary of the IUT;$  `   `  HHc)X  the requirements for the test coordination procedures are specified in the abstract test suite by means of a standardized test management protocol;H!  `  HHd)X  TMPDUs may be defined which correspond to ALPS;x  `  HHe)X  the upper tester is required to implement the test management protocol and achieve the appropriate effects on the IUT;(#  `   `  HHf)X  the lower tester is required to achieve control and observation of specified (Nb)ASP"s and specified PDUs (including TMPDUs).$  `  HThis method is illustrated in Figure3. 12.2.4.3 The remote test method Abbreviation: R  ` X HIn this method, provision is made for the case where it is not possible to observe and control the upper boundary of the IUT. Also in this method:  X( HHa)X  the abstract test suite is specified in terms of control and observation of (Nbĩ1)ASP"s and (Nb) to (Nt)PDUs;ư"  (`   ` H HHb)X  (Nt)ASPs are not used in the specification of the abstract test suite; no assumption is made about the existence of an upper boundary of the IUT;Ɛ$  H`   `  HHc)X  the abstract test suite may also be described in terms of control and observation of ALPs within the SUT;%  `   `  HHd)X  some requirements for test coordination procedures may be implied or informally expressed in the abstract test suite but no assumption is made regarding their feasibility or realization;%  `   `  HHe)X  abstractly the SUT needs to carry out some upper tester functions to achieve whatever effects of test coordination procedures and whatever control and/or observation of the IUT are implied, informally expressed or described by ALPs in the abstract test suite for a given protocol; these functions are not specified nor are any assumptions made regarding their feasibility or realization;%  `   `  HHf)X  the lower tester is required to achieve control and observation of specified (Nb)ASP"s and specified PDUs and should attempt to achieve the implied or informally expressed test coordination procedures in accordance with the relevant information in the PIXIT.$  `  HThis method is illustrated in Figure4. F-Ԍ HH12.2.5HSinglelayer, multilayer and embedded variantsxH  ` 8 HEach category of test methods has a variant which can be applied to singlelayer IUTs (abbreviation: S), and another which can be applied to multi layer IUTs (abbreviation: M), when the set of adjacent layers is to be tested in combination (as a whole).  8 HFor a multilayer IUT in which the protocols are to be tested layer by layer, an embedded variant of the test methods has been defined (abbreviation: E). HIf control and observation are applied to a means of access to the upper boundary of the entities under test within SUT, then the test methods are normal (and no E is added to the abbreviated name). If, however, control and observation are applied through one or more OSI* layer entities above the entities under test, the test methods are called embedded (and an E is appended to the abbreviated name). HNames of particular variants of the test methods shall be formed as follows: HFor example, DSE is the abbreviation for the "distributed single embedded" test method. 12.2.6HOpen relaysystems HFor open relaysystems, loopback and transverse abstract test methods are defined. They are given the abbreviated names: YL and YT, respectively. 12.3HSinglelayer test methods for singlelayer IUTs in endsystems HFor singlelayer test methods, the abstract model of the IUT is called the (N)entity under test. 12.3.1HThe Local Singlelayer test method HThe Local Singlelayer (LS) abstract test method defines the points of control and observation as being at the service boundaries above and below the (N)entity under test. The test events are specified in terms of (N)ASPs above the IUT, and (N1)ASPs and (N)PDUs below the IUT, as shown in Figure5. In addition, ALPs may be used as test events. Abstractly, a lower tester is considered to observe and control the (N1)ASPs and (N)PDUs while an upper tester observes and controls the (N)ASPs and ALPs. The requirements to be met by test coordination procedures used to coordinate the realizations of the upper and lower testers are defined in the abstract test suites, although the test coordination procedures themselves are not. 12.3.2HThe Distributed Singlelayer test method HThe Distributed Singlelayer (DS) abstract test method defines the points of control and observation as being at the service boundaries above the (N)entity under test and above the (N1)Service Provider at the SAP remote from the (N)entity under test. The test events are specified in terms of (N)ASPs above the IUT and (N1)ASP"s and (N)PDUs remotely, as shown in Figure6. In addition, ALPs may be used as test events. Abstractly, lower and upper testers are again considered to observe and control the behaviour at the respective points. The requirements to be met by the test coordination procedures are again defined in the abstract test suites, although the procedures themselves are not. HFor lower layers (13) where it may be unrealistic to specify observation and control of (N1)ASPs, the lower tester observation and control shall be specified in terms of (N)PDUs and, when necessary, changes in the state of the F- underlying connection. Note For example, the state of the underlying connection could be changed by setting up a new connection, or resetting or closing an existing connection. HThe observation and control to be performed by the lower tester can optionally be specified in terms of (N)ASP"s where this will reduce the size of the test case specification without loss of required precision. 12.3.3HThe Coordinated Singlelayer test method HThe Coordinated Singlelayer (CS) abstract test method is an enhanced version of the DS method, using a standardized upper tester and the definition of a test management protocol to realize the test coordination procedures between the upper and lower testers. The same standardized upper tester and test management protocol are not necessarily applicable to all test suites which use the coordinated method. HStandardized upper testers and test management protocols are applicable to a particular standardized abstract test suite for the coordinated test method and may not be applicable to other abstract test suites for the coordinated method. HThere is only a single point of control and observation, above the (N1)Service Provider at the SAP remote from the (N)entity under test. Test events are specified in terms of (N1)ASP"s, (N)PDUs and TMPDUs, as shown in Figure7. HFor lower layers (13) where it may be unrealistic to specify observation and control of (N1)ASP"s, the observation and control shall be specified in terms of TMPDUs, (N)PDUs and, when necessary, changes in the state of the underlying connection. HConcerning the test management protocol:  H HHa)X  the test management protocol shall be implemented within the SUT directly above the abstract service boundary at the top of the IUT;Ɛ$  H`   `  HHb)X  the IUT shall not be required to interpret TMPDUs, only pass them to and from the upper tester;(#  `   `  HHc)X  a test management protocol is only defined for testing a particular protocol and so does not need to be independent of the underlying protocol;   `   `  HHd)X  verdicts on test cases shall not be based on the ability of the SUT to exhibit any ASP or parameter of an ASP at the upper service boundary of the IUT, since this would contradict the definition of the coordinated method in that the upper service boundary of the IUT is not a point of control and observation in this method. However, it is recommended that the test management protocol be defined separately from the abstract test suites(s) in order to facilitate the task of the implementor of an upper tester. This definition (as with the definition of any OSI* protocol defined by ISO or CCITT) can refer to the ASPs of its underlying service (i.e. the ASPs at the upper service boundary of the IUT);$  `   `  HHe)X  TMPDUs which correspond to ALPs are optional and there is no requirement for the upper tester to support them.(#  `   `  HFIGURE 5/X.290, PART 2 FIGURE 6/X.290, PART 2 The LS test method The DS test method  F- HFIGURE 7/X.290, PART 2 FIGURE 8/X.290, PART 2 The CS test method The RS test method 12.3.4HThe Remote Singlelayer test method   HThe Remote Singlelayer (RS) abstract test method defines the point of control and observation as being above the (N1)Service Provider at the SAP remote from the (N)entity under test. The test events are specified in terms of the (N1)ASP"s and (N)PDUs remotely, as shown in Figure8. In addition, ALPs may be used as test events. Some requirements for test coordination procedures may be implied or informally expressed in the abstract test suites, but no assumptions shall be made regarding their feasibility or realization. HFor the lower layers (13) where it is unrealistic to specify observation and control of (N1)ASP"s, the observation and control shall be specified in terms of (N)PDUs and when necessary the state of the underlying connection. HIn addition, in order to overcome the lack of specification of behaviour above the (N)entity under test, where necessary the required behaviour of the system under test shall be specified in terms of the (N1)ASP"s or (N)PDUs which need to be observed by the lower tester. This form of implicit specification shall be taken to mean "do whatever is necessary within the system under test in order to provoke this required behaviour". Note Such implicit specification is necessary with this test method for any test case which requires an IUT initiated event which cannot be initiated by means of an ALP. Since ALPs may only be defined if the same effect cannot be achieved by ASPs, then any PDU which can be initiated by an ASP needs implicit specification to allow it to be initiated in this test method. HHowever, it is possible that some of the test cases in the abstract test suite cannot be executed (e.g. transmission and maintenance of busy conditions, transmission of consecutive unacknowledged Data PDUs, etc.). In such instances, it is left to the test laboratory and client to negotiate the method by which these tests can be accomplished. HEven with such implicit specification of control of the IUT, in this method it is possible to specify control but not observation above the IUT, except through the use of ALPs. This is a major difference between this and the DS and CS test methods. 12.4HMultilayer test methods for multilayer IUTs (LM, DM, CM, RM) HMultilayer testing, when the combined allowed behaviour of the multi layer implementation is known, involves testing all the layers of a multilayer IUT as a whole, without controlling or observing any of the interlayer boundaries within the IUT. HIn the Local Multilayer (LM) test method the points of observation and control are the service boundaries directly above and below the IUT. The test events shall be specified in terms of the (Nt)ASPs and ALPs above the IUT and the (N1)ASPs and (N) to (Nt)PDUs below the IUT. HIn the Distributed Multilayer (DM) test method the points of observation and control are at the service boundary above the IUT and above the (N1)Service Provider at the SAP remote from the IUT. The test events shall be specified in terms of the (Nt)ASPs and ALPs above the IUT and the (N1)ASP"s and (N) to (Nt)PDUs remotely. HIn the Coordinated Multilayer (CM) test method the point of observation and control is above the (N1)Service Provider at the SAP remote from the IUT. The F- test events shall be specified in terms of the (N1)ASP"s, the (N) to (Nt)PDUs and the TMPDUs. The test management protocol shall be designed to operate over the (Nt)Service, where (Nt) is the highest layer in the IUT. HIn the Remote Multilayer (RM) test method the point of observation and control is above the (N1)Service Provider at the SAP remote from the IUT. The test events shall be specified in terms of the (N1)ASP"s and the (N) to (Nt)PDUs, ALPs and the implicit specification of the control of the SUT behaviour when necessary. Some requirements for test coordination procedures may be implied or informally expressed, but no assumptions shall be made regarding their feasibility or realization. 12.5HSinglelayer testing of multilayer IUTs or SUTs (embedded methods) HIn embedded singlelayer test methods testing is specified for a singlelayer within a multilayer IUT, including the specification of the protocol activity in the layers above the one being tested but without specifying control or observation at service boundaries within the multilayer IUT. Thus in a multilayer IUT from layer (N) to (Nt), abstract test cases for testing layer (Ni) shall include the specification of the PDUs in layers (Ni+1) to (Nt) as well as those in layer (Ni). HThe Local Singlelayer Embedded (LSE) test method uses the same points of control and observation as the LM test method for the same set of layers. The test events shall also be specified in the same terms as for the LM test method. The difference is that the LSE test method concentrates on a singlelayer at a time, whereas the LM test method tests the multilayer IUT as a whole. HIn the Distributed Singlelayer Embedded (DSE) test method for layer (Ni) within a multilayer IUT from layer (N) to (Nt) the points of observation and control are at the service boundary above the IUT and above the (Niĩ1)Service Provider at the SAP remote from the IUT, as illustrated in Figure9(a). The test events shall be specified in terms of (Nt)ASPs and ALPs above the IUT and (Niĩ1)ASP"s and (Ni) to (Nt)PDUs remotely. Note For the top layer in the multilayer IUT, (Nt), this method is the same as the DS test method. HThe Coordinated Singlelayer Embedded (CSE) test method uses features of both the CM and DSE test methods. The test events shall be specified in terms of (Niĩ1)ASP"s, (Ni) to (Nt)PDUs and TMPDUs and the test management protocol shall be designed to operate over the (Nt)Service. This is illustrated in Figure9(b). HThe Remote Singlelayer Embedded (RSE) test method uses the same point of control and observation as the RS test method for the same layer, but differs from the RS test method in that (Ni+1) to (Nt)PDUs shall be specified in test cases for layer (Ni). HSuccessive use of a singlelayer embedded test method (from layer (N) to (Nt)) is called incremental testing of a multilayer IUT. HThe DSE/CSE/RSE methods are defined for a single layer under test in a multilayer IUT. This does not mean that there cannot be accessible service boundaries within the multilayer IUT, just that no such boundaries are used in the test methods. Thus, all layers between the layer under test and the highest layer for which PDUs are expressed as test events in the abstract test suite shall be regarded as being part of the multilayer IUT. Note DME/CME/RME test methods could theoretically be defined to test multiple layers as a whole within a larger multilayer IUT, but in order to avoid excess complexity, they are not detailed in this Recommendation. 12.6HRelay test methods F-Ԍ HThere are two abstract test methods for relay system testing:   HHa)X  the loopback test method (YL): used for testing a relay system from one subnetwork.$  `  HThis test method is illustrated in Figure10.  ` X HFor this test method there are two points of observation and control on one subnetwork at SAPs remote from the (N)Relay. For connectionoriented protocols, it requires that the two test connections are looped together on the far side of the (N)Relay, but it is not specified whether this looping is performed within the (N)Relay or in the second subnetwork. For connectionless protocols, it requires that the PDUs are looped back within the second subnetwork and addressed to return to the second point of observation and control.  XH HHb)X  the transverse test method (YT): used for testing a relay system from two subnetworks.Ɛ$  H`  HThis test method is illustrated in Figure11.  `  HIn this test method there are two points of observation and control, one on each subnetwork, at SAPS remote from the (N)Relay. FIGURE 9/X.290, PART 2 & The embedded test methodsă FIGURE 10/X.290, PART 2 & Loopback test method (YL)ă FIGURE 11/X.290, PART 2 & Transverse test method (YT) 12.7HChoice of test method 12.7.1HIntroduction HBefore an abstract test suite can be defined, it is necessary to study all the environments in which the protocol is likely to be tested and  h establish accordingly the abstract test method(s) to be used for the production of one or more abstract test suites. HAbstract test suite specifiers shall place a requirement in the abstract test suite Recommendation* defining which abstract test method(s) shall be supported as a minimum by any organization claiming to provide a comprehensive testing service for the protocol(s) in question. 12.7.2HIUT environments HThere is a relation between the test methods and the configurations of the real open systems to be tested. HSection 7.2, Part 1 of this Recommendation gives a full account of the classification of systems and IUTs. HWhen choosing a test method, the test suite specifiers should identify, if they have not already done so, whether the test suites they plan to produce are for IUTs which HHa)  are single or multilayer;p&H  h`  Hb)  belong to end or relay systems; F-Ԍ HHc)X  belong to complete or partial systems;x HHd)X  belong to fully open or mixed systems;x HHe)X  have service boundaries accessible or not;x  ` H HHf)X  are special purpose (i.e. to be used by a single application) or general purpose (i.e. to be used by several applications).Ɛ$  H`  12.7.3HApplicability of the abstract test methods  `  HThe possibility of developing an abstract test suite for a given abstract test method will depend on the protocol(s) being considered, together with the results of the study described in subsection 12.7.2. This applies to the possibility of developing test suites for a given combination of methods (e.g. incremental) or a given variant of a method (e.g. embedded). HSome considerations concerning the applicability of the methods to different layers are discussed in AppendixI of Part1 of this Recommendation.  X HOne or more appropriate abstract test methods shall be selected for the protocol being considered. HPriorities should be assigned to the various test methods which have been identified, according to the configurations which are most likely to be found in real systems. 12.7.4HIllustrative examples HAppendixIV provides an illustrative example of the choice of abstract test methods for given protocols. 12.8HTest coordination procedures HFor effective and reliable execution of conformance tests, some set of rules is required for the coordination of the test process between the lower tester and the upper tester. The general objective of these rules is to enable the lower tester to control remotely the operation of the upper tester or vice versa, in ways necessary to run the test suite selected for the IUT. HThese rules lead to the development of test coordination procedures to achieve the synchronization between the lower tester and the upper tester and the management of information exchanged during the testing process. The details and how these effects are achieved are closely related to the characteristics of the SUT, as well as the external test methods. HFor each abstract test suite using the coordinated, distributed or local test methods, a standard set of test coordination procedures should be developed. This is because those procedures depend on the functionality of the upper tester and definitions of test cases. HIn the case of the coordinated test methods (CS, CSE, CM) the test coordination procedures are realized by the standardization of a test management protocol. The test management protocol needs to be able to convey requests to the IUT to achieve the effect of a service primitive or ALP and to convey back to the lower tester the record of observations of effects equivalent to the occurrence of service primitives or ALPs. The upper tester should be an implementation of the test management protocol. It will be necessary to add test cases to the abstract test suite for the purpose of testing that the upper tester conforms to the requirements F- of the test management protocol specification. Such test cases do not contribute to the conformance assessment of the IUT. HWhen defining test cases for the local and distributed test methods, the test suite specifier shall record any constraints on the upper tester and/or test coordination procedures which may be necessary. 13.HSpecification of abstract test suites 13.1HTest cases 13.1.1HThe abstract test suite specifier shall select an appropriate standardized notation in which to define the abstract test cases. The Tree and Tabular Combined Notation (TTCN), defined in AnnexD, is defined for this purpose. 13.1.2HOnce the test notation and test method have been chosen, then the generic test cases can be expanded into abstract test cases. There are two main kinds of change required to convert a generic test case into an abstract test case. The first is to express the test body in terms of control and observation required by the test method, and, if relevant, include a description of the synchronization needed between upper and lower testers. The second kind of change is to specify the preamble and postamble.  X 13.1.3HSpecification of preambles and postambles may result in more than one test step for each. The preamble shall start in a stable state and progress to the desired state. Conversely, the postamble shall progress from the final state of the test body to a stable state. A small number of stable states shall be defined for the protocol concerned, always containing as a minimum the "idle" state. Each abstract test case shall be capable of being run on its own and shall therefore include test steps to start the preamble from the "idle" state and to end the postamble in the "idle" state. HHowever, other stable starting and final states for an abstract test case can be useful when efficient concatenation of abstract test cases is required. HFurthermore, in designing the test step structure within the abstract test cases, the test suite specifier can benefit from using the same test steps in several abstract test cases. 13.1.4HIn converting from generic test cases to abstract test cases, the test suite specifier shall ensure that the initial state for the test body is preserved, the pass paths through the test body are preserved and the assignment of verdicts to outcomes remains consistent. HIn order to maintain consistency, the following conditional requirements apply when assigning verdicts to outcomes:  ( HHa)X  if the behaviour of the preamble and the postamble are valid then the verdict assigned to a particular outcome shall be the same as that assigned to the corresponding outcome in the generic test case;ư"  (`   `  HHb)X  if the preamble results in the initial state of the test body not being reached, although there is no invalid behaviour, then the verdict shall be inconclusive;%  `   `  HHc)X  if the preamble results in the initial state of the test body not being reached, because of invalid behaviour, then the verdict shall be "fail but test purpose inconclusive" ("fail type3");%  `   ` H HHd)X  if the postamble exhibits invalid behaviour and the generic test case (or test body) verdict is "pass" or "inconclusive", then the verdict shall be "fail but test purpose passed" ("fail type2") or "fail but F- test purpose inconclusive" ("fail type3"), respectively;Ɛ$  H`   ` 8 HHe)X  if the generic test case (or test body) verdict is "fail" then invalid behaviour in the postamble shall not change the verdict ("fail type1").Ơ#  8`   ` 13.1.5HThe test suite specifier shall also ensure that each abstract test case explicitly identifies all the outcomes assigned the verdict "pass" and identifies or categorizes all the remaining foreseen outcomes, assigning to each individual outcome or category a verdict "fail" or "inconclusive". All unforeseen outcomes in the test body shall be assigned either HHa)X  the verdict "fail"; or'  `  HHb)X  the verdict "inconclusive".x  `  HThe test suite specifier shall ensure that the application of a) or b) is consistent throughout the abstract test suite. If a) is chosen, then any unforeseen outcome in the preamble shall be assigned the verdict "fail but test purpose inconclusive" ("fail type3"), and any unforeseen outcome in the postamble shall be treated as a protocol violation, leading to the appropriate type of fail verdict. HIf b) is chosen, then any unforeseen outcome in the preamble shall be assigned the verdict "fail but test purpose inconclusive" ("fail type3"), and any unforeseen outcome in the postamble shall be assigned the appropriate type of fail verdict. 13.2HTest suites HAn abstract test suite specification comprises a set of test cases and test steps. Preceding the test cases themselves shall be the following information:   HHa)X  abstract test suite name, date of origin and version number;8"  `   ` 8 HHb)X  names (and version numbers) of the protocol Recommendation(s)* for which test cases are provided;Ơ#  8`   `  HHc)X  names (and version numbers) of the service Recommendation(s)* whose primitives are specified as being observed;(#  `   `  HHd)X  name (and version number) of the Recommendation* defining the test notation, or a reference to an annex for the same if it is not standardized elsewhere;(#  `  HHe)X  name of target test method;x  `  HHf)X  description of the coverage of the test suite; for example, the functional subsets of the protocol Recommendation(s)* that are tested;$  `   ` H HHg)X  description of the structure of the test suite, in terms of test groups and their relationship to the protocol Recommendation(s)*:Ɛ$  H`   `  HHh)X  description of the test coordination procedures (if applicable in the test method);%  `   ` 8 HHi)X  statement of which test cases are optional and which mandatory for conformance to the abstract test suite Recommendation*;Ơ#  8`   ` 8 HHj)X  information to assist the test realizer and test laboratory in their use of the abstract test suite Recommendation* (see section14).Ơ# F-Ԍ 8`  14.HUse of an abstract test suite specification  `  HThe abstract test suite specifier shall provide information in the abstract test suite Recommendation* to assist the test realizer and test laboratory in their uses of the test suite. This information shall include, but is not limited to, the following:   HHa)X  a mapping of the abstract test cases to the PICS proforma entries to determine whether or not each abstract test case is to be selected for the particular IUT; the mapping should be specified in a suitable notation for Boolean expressions;%  `   `  HHb)X  a mapping of the abstract test cases to the PIXIT proforma entries, to the extent that they are known by the abstract test suite specifier, to parameterize the test suite for the particular IUT and to determine which selected test cases cannot be run with the particular IUT.!  `   `  HThe test suite specifier shall define a partial PIXIT proforma. This shall contain a list of all the ALPs used in the test suite (or test management protocol) and a list of all parameters for which the test suite requires values. If any of the required parameter values will be found in the PICS, the PIXIT proforma entry for each such parameter shall reference the corresponding entry in the PICS proforma. Note Other aspects of the PIXIT proforma are for further study.   HHc)X  a list of the abstract test cases in the order that shall be used in the Protocol Conformance Test Report (PCTR), together with any information which shall be preserved in the PCTR on the status of each test case; this is a contribution to the development of a PCTR proforma;%  `   `  HHd)X  any restrictions that there may be on the order in which the test cases may be executed;%  `   ` H HHe)X  reference to the description of the test coordination procedures (if applicable in the chosen test method);Ɛ$  H`  HHf)X  any necessary timing information.x 15.HTest suite maintenance  ` X HOnce an abstract test suite has been specified and is in use, it can be expected that errors and omissions in it will be detected by those who are using the test suite. The abstract test suite specifier shall in such circumstances progress the updating of the test suite via the relevant rapid amendment procedures. HIn addition, from timetotime, changes will be made to the protocol Recommendation(s)* to which the test suite relates. The abstract test suite specifier shall ensure that the test suite is updated as soon as possible after changes to the relevant protocol Recommendation* have been ratified.