List of Figures

      

List of Tables

      

ABSTRACT

      

1.0  INTRODUCTION

      

One of the Task 1 requirements is that "Acquisition personnel must be enabled to determine whether deliverables conform to the relevant CALS standards." In refining this requirement, Subtask 2 will "Establish criteria for reuse of CALS testing tools. Evaluate with respect to such factors as portability, user-friendliness, and adequate documentation." and Subtask 3 will "Assess testing tools against reuse criteria. Identify deficiencies and functional gaps. Formulate a plan to correct these problems, including cost estimates for correcting/augmenting these assets." Subtask 4 will "Assess alternatives for developing a CALS validation testing and data acceptance tools reuse repository, and develop an implementation plan with resource estimates." These reuse requirements specified in the SOW are consistent with the Department of Defense's (DoD) vision for software reuse [DOD92].

A general operational context for CALS test and validation tools and data administration tools is presented below (Figure 1.0­1).

Each domain has several stake-holders that have interest in generating, maintaining, and using the assets within the context of that domain. There are three main stake-holders for any given domain: Users, Creators, and Maintainers/Managers [STARS92]. An organization can play the roles of more than one stake-holder. For example, an organization may create, manage, and use the assets in a domain. These three functions encompass the life-cycle of domain assets. In the domain of CALS Test and Validation Tools, vendors (or other developers) of test and validation tools fulfill the Create function, libraries such as CALS Test Network (CTN) fulfill the Manage function, and the organizations that utilize CTN to achieve mission requirements fulfill the Use function.

The "Design for Reuse" paradigm refers to the design/creation of domain assets so that they can be reused by other applications. This reuse paradigm reflects heavily on the Create function. The "Design with Reuse" paradigm refers to the building of new applications by reusing existing software. This paradigm supports the Use function. The Manage function (typified by a library infrastructure) has to foster and support both paradigms by providing a brokering mechanism for the Create and Use stake-holders. The Manage function also emphasizes continuous refinement of domain assets.

This report concentrates on the reuse issues as seen from the perspective of Manage function of CALS test and validation tools domain. This perspective is relevant for setting up the CALS test and validation tools repository.

2.0  SUMMARY

      

To set up a CALS test and validation tools repository, it is necessary to document, a priori, qualification guidelines that will be used in acquiring reusable assets for the repository. These guidelines will be used as "benchmarks" in determining the usefulness of assets within the context of the CALS test and validation domain. The initial criteria developed will be continually refined as the knowledge of the qualification team matures.

Advances in hardware technology have made it possible for widespread use of Commercial­Off­The-Shelf (COTS) software in building new software-intensive systems (even in mission-critical applications). Although (re)use of COTS software in building new applications supports the notion of mega-programming (large-scale reuse), it raises some interesting questions about the measurement of reuse potential for a given COTS software (How can one assess if Sybase is more/less reusable than Oracle?). Even when large in­house/public­domain software systems are used in building new applications, the reused systems are in essence treated as "black-boxes." The "goodness" of a software system is measured in terms of required functionality provided by the software system within the performance parameters of the application system. Further, because of the size and complexity of these systems, it may not be feasible to assess the "goodness" of the internal design of the reused systems.

3.0  CONCLUSION

      

Qualification of assets ensures that only those assets that are relevant and useful to the CALS test and validation tools community will become part of the repository. Thus, qualification guidelines are an integral part of the CALS test and validation tools repository. Specific qualification guidelines have to be used in assessing the reusability of tools for each of the CALS standards (e.g., Standard Generalized Markup Language (SGML), Initial Graphics Exchange Specification (IGES), Raster). Each of these tools will also be assessed against some common criteria as well (ease of use, portability, maintainability, etc.). Novel assessment criteria have to be employed to measure "goodness" of large software systems.

4.0  REUSE ISSUES

      

1. Conformance

The essence of CALS standards is to enforce standardization of data (documents, graphics, etc.) by specifying data formats (SGML, IGES, etc.) to facilitate (re)use of data among several applications. Thus, each of the test tools will have to be evaluated to ensure that all facets of the applicable standards are applied consistently and correctly to the test data. The reusability of a test tool is heavily dependent on its efficiency in weeding out non-conformant data as well as on proper identification of conformant data.

2. Tools Reuse

Test Tools will be largely evaluated on the basis of their ability to test the conformance of data to their concomitant standards (as described previously). Additionally, these tools will have to be evaluated on the basis of some general reuse criteria:  usability, maintainability, portability, reliability, etc. A "high" measure of these indicators for a given test tool will indicate its applicability to a wide spectrum of user environments.

The following Integrated Computer Aided Manufacturing Definition (IDEF) diagram (Figure 4.0­1) presents a context for qualifying CALS test tools. The purpose of the qualification process is to evaluate COTS, Government­Off­The­Shelf (GOTS), or other assets in light of applicable CALS standards, the common reusability criteria, as well as the IWSDB architecture within which these assets will be employed.

5.0  REUSE CONCEPTS

      

Most of the reuse efforts to date have been centered around general-purpose reuse of software artifacts. Mathematical and graphical subroutine libraries are good examples of such efforts. Such libraries can contain hundreds or thousands of software artifacts (mostly source code) that can be reused across multiple applications in multiple domains. Powerful search and retrieval mechanisms are needed to identify candidate reusable artifacts. One key deficiency of general­purpose reuse libraries is the loss of contextual information. When a user retrieves a candidate artifact for reuse, the user still has to analyze the artifact within the context of the intended application (command centers, medical systems). The functional and performance requirements imposed by two distinct domains may be vastly different. Furthermore, the architectural context within which a given software component has to operate may also differ among domains and applications.

The advent of domain-centered reuse efforts is centered on the provision of contextual information within which reuse can occur. The premise is that, in well-matured domains, contextual information can be leveraged to provide large-scale reuse of previously developed software artifacts. It is important to stress that, within the realm of domain-centered reuse, the term software artifact means much more than software code. Non-code assets such as requirements, architectures, designs, test material, lessons learned, decision points, etc., are all considered to be reusable assets.

5.1  Qualification/Metrics Concepts

Several reuse infrastructures promote and facilitate the reuse of software artifacts. Typically, reuse infrastructures are exemplified by "reuse libraries." Such libraries may be internal to organizations or shared by the public at large. Most of these libraries follow certain criteria (either implicit or explicit) in determining the entry of an asset into libraries. In other words, the assets in a given library are qualified to have passed some pre-determined criteria. Some libraries may have levels of qualification for their assets (e.g., tested vs. non-tested).

The nature of qualification undertaken by libraries is inherently dependent on the intended use of those assets. Some libraries (such as ASSET) contain numerous reusable assets that can be used by several user groups from differing application domains. Such libraries have to measure the general "goodness" of assets so as make these assets reusable among several domains. The criteria that are used by general-purpose libraries have to be sufficiently generic enough to be commonly applicable across all the domains. Source-code assets constitute a large portion of such general-purpose libraries. Hence, the criteria used to evaluate software assets for general­purpose libraries center around code metrics. Usually a subset of metrics generated by metric collection utilities is employed to evaluate assets (AdaMAT, McCabe's, etc.). These metrics are used as indicators of general "goodness" of software assets such as maintainability, reliability, portability, usability, etc.

Domain-centric libraries [CARDS94] measure the "form-fit-function" of assets within the context of particular domains. Such libraries emphasize the reusability of assets to meet the domain requirements in varying application contexts within that domain. Such libraries employ criteria to assess the reuse potential of an asset within the context of a given domain (domain­specific criteria) as well as to measure reuse potential of that asset across several domains (common criteria). Such libraries also emphasize large-scale reuse of assets in support of mega­programming concepts. Mega-programming concepts emphasize the use of large "black­box" assets in the development of new applications. Thus, typical assets found in domain-centric libraries may be larger (message processing systems, database systems, etc.) than those found in general-purpose libraries (stack/queue functions, math packages, etc.). Thus, the common criteria employed in assessing the reuse potential of large system will be different than those employed by general-purpose libraries.

5.2  Common Criteria for Large Systems

Software artifacts have to be qualified or certified as reusable among multiple domains. The criteria that are used to evaluate software artifacts are dependent on a number of factors (type of artifact, environment, domain, etc.). These criteria are commonly applied to all the artifacts for general reusability across multiple domains.

Historically, very little work has been done in the area of qualifying large code and non-code assets in support of the concepts of mega-programming. Domain-centric reuse programs such as Central Archive for Reusable Defense Software (CARDS) [CARDS94] and Portable Reusable Integrated Software Modules (PRISM) [PRISM92] have pioneered new qualification concepts in this area.

It is highly conceivable that a majority of the test tools will be either COTS or GOTS. In the absence of access to code, it is not possible to employ metrics measurement tools (such as AdaMAT) to measure the "goodness" indicators of software (such as cyclomatic complexity, etc.). For large systems, even if the code were available, such code measures may be meaningless. Thus, the measurement of reusability indicators has to be done through some novel criteria. Table 5.2­1 enumerates a preliminary list of criteria for assessing reusability of large software systems. These criteria will be uniformly used in assessing the common reusability indicators of CALS test and validation tools. The following criteria can also be used in assessing reusability of CALS data administration tools (Task 2). Answers compiled for the following questions give indications of "ilities" of a tool such as maintainability, portability, usability, affordability (cost), etc.

5.3  Domain Criteria for Test and Validation Tools

For the success of any domain-specific reuse repository, it is essential to qualify software artifacts against a set of domain criteria. This step is necessary to ensure that the artifacts meet certain requirements (minimum/desirable) for their intended use. For example, performance and sizing requirements may be different for two separate domains. Alternatively, the requirements of a tool (even if it is the same tool) will differ based on the operational context (development, maintenance, end-user etc.). Thus, an artifact has to be qualified for each context that it is intended to support. An artifact may be very well qualified to meet the requirements of a particular domain and be more reusable in that context, but it may not adequately meet the requirements of another domain.

The domain criteria can be initiated only after properly scoping the domain of interest. To this end, it is important to scope the domain of CALS test and validation tools. There are several scenarios in which CALS test tools can be defined and utilized. For example, the domain of CALS test tools could span one or more of the following contexts (sub-domains):

Test and Validation of CALS data (Conformance).

Test and Validation of CALS data generators and receptors (Compliance).

Test and Validation of CALS data transformers (Interoperability).

Test and Validation of recipient vs. generated CALS data (Data Acceptance).

The choice of context will determine the qualification criteria to be developed and used in qualifying potential tools. In addition, these tools have to be integrated with data administration/acceptance tools within the context of an IWSDB. Such a scenario places additional compositional constraints. The test tools have to be assessed based on their ability to work with other systems/subsystems within the IWSDB. Furthermore, because compositional constraints are predicated on the IWSDB architecture, different architectures may entail different qualification factors.

A separate set of domain criteria reports has to be generated for each class (sub-domain) of CALS test and validation tools (e.g., Computer Graphics Metafile (CGM) test and validation tool qualification criteria, raster test and validation tool qualification criteria). First, for each standard, the appropriate context for testing has to be determined (Conformance, Compliance, Validation, etc.), and, based on the chosen context, the qualification criteria for that class of test tools has to be developed. The following paragraphs present an overview of each of the CALS standards followed by our initial set of qualification criteria, for testing conformance, specific to each of the standards. We are currently in the process of procuring demo/evaluation versions of several COTS test and validation tools. The qualification criteria will be applied to selected tools and refined as necessary. We intend to include the refined qualification criteria as part of the final report.

5.3.1  Overview of CALS Standards

The following paragraphs are part of the CALS Assessment Report submitted earlier [OSD94] and are included here for the sake of completeness.

OVERVIEW:  MIL-STD-1840B ­ Automated Interchange of Technical Information

This is the "parent" CALS standard. It provides the rules for organizing files of digital data that conform to their respective 28000 series standard. MIL-STD­1840B specifies CALS delivery, format, and organization including header file requirements, data file types, etc.

MIL-STD-1840B explains the standards involved with the exchange of digital information for the presentation of technical information. It includes descriptions of the information that should be included with an exchanged file as well as the type of media and packaging guidelines.

This standard is applied wherever information is exchanged. There are no restrictions to its application. It can be used for the exchange of American Standard Code for Information Interchange (ASCII) text files, product definition data files, raster image files, or vector graphics files.

The transfer unit declaration file is an ASCII file that provides all the needed information to uniquely identify the contents of the transfer unit. It provides such information as where the files came from, where they are going, what kind of data is included, and how much data is being exchanged. The records in the unit declaration file are an identifier string followed by a ":" and a space. All of these records must be present in the file and in the order stated by the standard. The standard also defines the convention used in naming the transfer unit declaration file. An example of a unit declaration file is shown below:

Version: MIL-STD-1840B, 0, 19921103.

SRCSYS: Almost Heaven WV, 1 Fairmont, WV, 26554.

SRCDOCID: 1 document.

SRCRELID: NA.

CHGLVL: ORIGINAL, 0, 0, 19940415/1100:00.

DTEISU: 19940415/1100:00.

DSTSYS: ABC System, Wright-Patterson AFB, OH, 45433.

DSTDOCID: NONE.

DSTRELID NA.

DTETRN: 19940310/0800:00.

DLVACC: Contract Data Requirements List (CDRL) Item 1 of SOW 2001.

FILCNT: R1.

TTLCLS: Unclass.

DOCCLS: Unclass.

DOCTYP: MIL-R-28002B Test File.

DOCTTL: Raster Test Data.

TRANSACTTYP: PRODUCT DATA.

Each data file in a transfer unit must contain header records that are relevant to the type of data being exchanged. The data file header records are identifier strings followed by a ":" and a space. All data file header records are of a fixed length depending on the data file type. There are 14 data types which can be exchanged under MIL-STD-1840B. An example header record is shown below:

SPECVERSION: MIL-R-28002B 19921214.

SRCDOCID: 1 document.

DSTDOCID: NONE.

DTYPE: 1.

RORIENT: 090,270.

RPELCNT: 001505,001268.

RDENSITY: 0300.

DOCCLS: Unclass.

Notes: NONE.

The medium of delivery is 9-track magnetic tape unless the involved parties mutually consent that an alternate method of exchange, such as floppy disks, shall be used. Alternate methods of exchange could include media such as floppy disks, digital audio tape (DAT), magneto-optical disks, etc. The electromagnetic media will be packaged such that it is protected against dirt, moisture, and electrostatic discharges. The 9-track tapes are to be placed in a PPP-B-636 conforming box with PPP-C-1842 cushioning material.

The package shall also contain a packing slip that shows the names and volume numbers of the shipped reels or disks and a list of the transfer unit declaration files. The outside of the package will contain a warning message explaining that because the package contains magnetic media, the package must be handled in a fashion that will not harm its magnetic contents. The package must also have a label containing the following information:

Company name/Government Organization.

Date.

MIL-STD-1840 Version.

CAGE (Commercial And Government Entity) code.

Volume Identification.

Density of media.

Media Number.

Point of Contact.

This specification defines the CALS­compliant subsets of the IGES for the interchange of vector graphics. The focus of this standard is Computer-Aided-Design (CAD) information, primarily the digital representation of geometry.

MIL-D-28000 is the military specification for the digital representation of product definition data using IGES as specified by the American Society of Mechanical Engineers (ASME) standard Y14.26M. MIL-D-28000 IGES is organized into five classes by application area to meet the delivery need of that application. These classes are:

Class I  Technical illustrations subset (2-D geometry + text).

Class II  Engineering drawing subset (3-D with 2-D views).

Class III  Electrical/electronic applications subset (3-D + entity relationships).

Class IV  Geometry for numerical control (NC) manufacturing subset (wire frame and surface models).

Class V  3-D Piping application protocol.

The third class, Electrical/Electronic applications subset, should be in a class of its own because of its differences in integrity. The first four classes specify the entities needed for specific application subsets. In this way, the recipient of a MIL-D-28000 data file may specify the class of data needed without becoming an IGES expert. The class V application protocol is for the exchange of product data for 3D piping system models only.

IGES is a vector­based neutral format for the exchange of product model data. IGES is the most widely used product data exchange specification in the US. It was designed specifically for CAD/Computer Aided Manufacturing (CAM) systems for the exchange of geometric models and drawing. The data format is in ASCII with a fixed line length. The unit of information in IGES is an entity. Entities include solid models, curves, surfaces, and annotation.

Most CAD systems have some type of IGES translator, and even some non-CAD systems support the IGES specification. The support for MIL-D-28000 (subsets) is not as widespread as the support for the full IGES standard. The greatest stated support of the subsets comes from the commercial flavoring software and syntax checking software. MIL-D-28000 Class II, engineering drawings, is the most commonly supported class.

The MIL-D-28000 specification is being developed jointly by the DoD, federal agencies, and private industry under the direction of the DoD. The CALS Digital Standards Office is responsible for distributing the draft specification for comment, receiving, and tracking the comments, for holding the comment coordination meetings, for updating the draft specification with the comments, and for supplying the revised document to the Office of the Assistant Secretary of Defense (OASD) for approval.

The IGES/Product Data Exchange Specification (PDES) Organization (IPO) maintains a list of IGES versions. When a change is suggested, a Request For Change (RFC) is submitted to IPO for review. A technical committee will update the IGES version and publicly release the new version as well as documentation. The following is a list of IGES versions and its changes:

Flag section is for compressed ASCII and Binary IGES files only. Start section is a viewable text header. MIL-D-28000 requires written information about the IGES file. Global section contains information needed for processing the file such as IGES file name, preprocessor version, units of measurements, version of the IGES specification, and drafting standard. Directory entry section organizes the IGES file by dividing it into 10 fields per line with 8 characters per field. Parameter data section contains geometric and other information needed to define the entity in the CAD database. Terminate section is the last line in the IGES file. The terminate section indicates the number of lines in each section of the IGES file.

The SGML was developed to address the problems of information exchange and management challenges associated with electronic printed output and exchange of textual information. The term markup originated from the typesetting instructions handwritten on authors' manuscripts. These instructions described the desired appearance of the text on a printed page. A similar scheme was adopted with the evolution of electronic publishing in order to provide instructions to an automated printing device. The International Standard Organization (ISO) participated in the development of the SGML standard and adopted it in 1986 as ISO 8879. This standard completely defines the terms and syntax to specify the structure and content of a document. ISO 8879 does not provide functionality for specifying appearance or formatting requirements.

The DoD application of SGML is specified in MIL-M-28001. ISO 8879 defines SGML as a meta-language. Many different SGML languages satisfying the grammar and syntax specified in ISO 8879 can be obtained by choosing from among different features and options afforded by ISO 8879. MIL-M-28001 specifies the DoD CALS SGML implementation by choosing certain SGML features and setting certain parameters. MIL-M-28001 also uses SGML to provide a capability for specifying the formatting of CALS SGML documents.

An ISO 8879 SGML document contains three parts: the SGML declaration, the Document Type Definition, and the Document instance. A CALS SGML document contains these parts and may also contain a Formatting Output Specification Instance (FOSI). The parts of a CALS SGML document and how each relates to the others are described next.

SGML Declaration

The SGML declaration defines what characters will be allowed in the rest of the document and how they will be encoded. Because many different encoding schemes are available, the SGML Declaration is needed to ensure document transportability between different publishing systems. Both ISO 8879 and MIL-M-28001 specify the use of the American Standard Code for Information Interchange (ASCII). The SGML declaration also specifies how certain characters are to be interpreted. For example, the reference syntax, which is defined in the SGML declaration for the character "<", is the start-tag open delimiter in the document instance. Other special rules and definitions are also included in the SGML declaration. MIL-M28001 specifies the SGML declarations for CALS SGML applications.

Document Type Definition

The Document Type Definition (DTD) defines the content and structure of a document. This structure defined in the DTD should not be confused with the parts of an SGML document, i.e., SGML declaration, DTD, document instance, and FOSI. It refers to how the information in a class of documents such as technical manuals, books, memos, etc., are related, and defines any dependencies. The document sections are declared in the DTD as element declarations and then referred to as elements. Each element declared is given a unique name called a generic identifier (GI). These GIs are used as the "tags" to markup the document's information content and are also used to verify the content and structure of the document. Markup of a document provides an unambiguous definition of its structure and content, allowing automated data processing software to process the document in a predictable manner. If the DTD defines an index element and the document is properly tagged, a word processing program designed to read SGML documents will be able to automatically generate the index for the entire document. Also defined in the DTD are entity declarations. Entities may be used to define data once and to reference it many times. Entities are typically defined in the document type declaration subset and referenced in either the document instance or the DTD itself. DTDs may be used for any class of documents that have the same content and structure. CALS documents are to follow the CALS DTD for SGML documents, 38784.

Document Instance

The document instance is the information content of the document, marked up in accordance with a DTD. This markup may include declarations, tags, and entity references. The first line of a document instance contains the document type declaration. This declaration identifies the DTD required to correctly interpret the document.

Formatting Output Specification Instance

The FOSI specifies the desired appearance of the information content of the document. This output formatting description capability is not contained in ISO 8879 but is found in MIL­M­28001. MIL­M­28001 contains an output specification (OS) DTD that defines how FOSIs are to be developed and interpreted. Document-wide formatting requirements such as page layout and hyphenation rules are specified in the FOSI, along with how document elements such as paragraph titles, tables, and lists are to be formatted. The FOSI, which is based on the DTD, provides the composition and imaging characteristics to be applied to the SGML tagging of a document instance to present the text material in paginated or screen presentation form.

The MIL-R-28002B specification identifies the requirements to be met when raster graphics data represented in digital, binary format are delivered to the Government. This data format is used for the storage and interchange of scanned engineering drawings as well as technical manuals and computer­generated illustrations.

The digital representation of raster graphics data will be one of the following types:

Type I  - Untitled Raster Graphics Data.

Type II - Tiled/Untitled Raster Graphics Data.

Type I data is CCITT T.6 Group 4 encoded for the entire single image and is enclosed in a MIL­STD-1840 header. The CCITT T.6 Group 4 encoding of raster data is defined in Federal Information Processing Standard (FIPS) PUB 150, Telecommunications: Facsimile Coding Schemes and Coding Control Functions for Group 4 Facsimile Apparatus. CCITT T.6 Group 4 encoding is basically an algorithm for condensing a bitmapped image file.

Type II data is defined in the Office Document Architecture (ODA) Raster Document Application Profile (DAP) that is found in Appendix A of the MIL-R-28002B specification. The notation used for the interchange format of the ODA document is called Abstract Syntax Notation One (ASN.1). ASN.1 is defined in two documents: IS 8824, Information Processing Systems ­ Open Systems Interconnection ­ Specification of Abstract Syntax Notation One (ASN.1) and IS 8825, Information Processing Systems ­ Open Systems Interconnection ­ Specification of Basic Encoding Rules for Abstract Syntax Notation One (ASN.1). The MIL-STD-1840 header is wrapped around the ODA-style document. The Type II raster data may be tiled or it can be a single untiled raster image such as Type I. The information for the data structuring is contained in the ODA parameters.

Each tile in a ODA Raster DAP file can be coded differently. The content information in a tiled or untiled MIL-R-28002B Type II raster file can be represented in the following formats:

T.6 encoding ­ Most Significant Bit (MSB):  The entire raster image is not tiled and is encoded in accordance with the CCITT T.6 algorithm using the down bit order sequence (MSB to Least Significant Bit (LSB)).

T.6 encoding:  The entire raster image is not tiled and is encoded in accordance with the CCITT aT.6 algorithm using the up bit order sequence (LSB to MSB).

Null background:  Background without any raster content.

Null foreground:  Foreground raster information.

Dividing a raster image into tiles and encoding them in different formats significantly reduces the amount of digital data that is needed to store the image.

Also, different types of information are represented with different picture element densities when stored in the digital raster format. For technical manuals and illustrations, the picture element density must be 300 pels per inch. For large-format engineering drawings, the picture element density must be 200 pels per inch.

A CGM is a functional specification (semantic standard) for encoding computer graphics images for redisplay on one or several supported device types. It is composed of three encoding (syntactic) standards:  one, a character encoding for minimal size and ease of transmission; two, a binary encoding for speed of access; and three, a clear text encoding for human readability and easy editing. It is primarily oriented toward stroke­drawn graphics, but includes raster bitmap encodings. CGM is both an American National Standard (ANSI X3.122-1986) and an ISO international standard (ISO8632/1987) that were replaced by ANSI/ISO 8632:1987 in 1991. Amendments took place to the standard to make CGM more compatible with graphics, applications, and electronic publishing standards. These amendments were incorporated into a new CGM standard, the International Standards Organization (ISO)/International Electrotechnical Commission (IEC) 8632:1992, that has also been adopted by ANSI as ANSI/ISO 8632:1992 and in the FIPS PUB 128.

MIL-D-28003A is a proposed military specification defining an application profile for delivery of two-dimensional technical publication illustrations using the CGM standard. CGM application profile requirements defined by MIL-D-28003 address the first of several classes of conforming basic meta-files, a conforming basic metafile generator, and both a minimum-level and a publication-level conforming basic metafile interpreter. Generator and interpreter requirements are specified to provide necessary controls over the creation and validation of conforming metafiles and to remove implementation dependencies. The metafile definition itself includes the allowable output primitives and attributes that may be used to compose the illustration. Although the CGM standards that MIL-D-28003 implements define a complete and unambiguous syntax, they intentionally under specify the semantics, or meaning, of the CGM primitives and attributes to accommodate a wider range of existing systems, and to make the standard more adaptable to the requirements of diverse applications and users. The CALS application profile provides an unambiguous interpretation of those semantics appropriate for illustration of technical publications.

5.3.2  CALS Qualification Criteria

All of CALS data, irrespective of the specific standard it has to conform to (such as IGES, SGML, etc.), has to conform to general standards for Automated Interchange of Technical Information as specified in MIL-STD-1840B. Hence, the data has to be checked for conformance of MIL-STD-1840B in addition to the applicable specific CALS standards. Usually, separate tools may have to be employed to test for conformance to MIL-STD-1840B. There are three areas of testing for meeting the MIL-STD-1840B standard:

1. Does the tool check for proper naming convention?

2. Does the tool check the data file for required content information?

3. Does the tool check for format of the content?

1. Does the tool check for proper naming convention?

2. Does the tool check for proper header records?

3. Does the tool check for file type?

1. Does the tool validate sequential magnetic tape media?

2. Does the tool validate data on alternative media (floppy, etc.)?

5.3.2.1  SGML Domain Specific Criteria

Conformance tools for the validation of the components of the SGML file set are termed as SGML Parsers. While it is reasonably trivial to evaluate these tools against general qualification criteria, the same cannot be said about them from the SGML file point of view. It is difficult to assign domain­specific reuse criteria for the SGML parsers as can be done with the other standards like CGM and IGES. Such an assignment, if made, would prove to be too general in the SGML domain context. On the other hand, the parsers themselves can be validated against a set of detailed scripts that test and evaluate the performance of the parser itself.

The Computer Systems Laboratory of the National Institute of Standards and Technology is evolving the procedures for establishing a validating program for Federal Information Processing Standards 152, Standard Generalized Markup Language parsers. The testing methodology is based on ANSI X3. 190-1992, Text and Office Systems ­ Conformance Testing for Standard Generalized Markup Language Systems. The Computer Systems Laboratory specifies the necessary conformance test specifications, test methods (test suites, test tools, and technical procedures), validation procedures, and testing laboratories for testing SGML parser product compliance. The testing information provided by the SGML Conformance Testing Service will be published in Certificates of Validation, Validation Summary Reports, Registered Validation Summary Reports, and Validated Products Lists.

The SGML Validation Service will use testing laboratories to provide the testing service to evaluate conformance to FIPS 152. The SGML Test Suite consists of over 1000 test scripts that evaluate the performance of the parser to individual aspects of SGML elements and routines. For a parser to conform with FIPS 152, it must pass all of these test scripts. The output from running the test scripts must be compliant with ANSI X3.190-1992. The SGML product must output the parser results from running the test scripts in the RAST (Reference Application for SGML Testing) format. RAST interprets all markup to allow for machine comparison of test results for documents conforming to ISO 8879. RAST indicates in a standard way when a tag, processing instruction, or data is recognized by the parser, replacing references and processing markup declarations and marked sections appropriately. The SGML Test Suite contains a carefully chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the general rules from the standard.

Upon careful consideration, it has been decided that the SGML conformance testing tools (parsers) that need to be used in a CALS Test and Validation tools repository must be ones that have passed the exhaustive conformance testing requirements of the SGML Validation Test Suite at the National Institute of Standards and Technology. This form of parser validation will ensure that the parser being used will serve as an established one, and further criteria for qualification into the CALS Test and Validation Tools repository are not necessary.

5.3.2.2  CGM Domain Specific Criteria

A tool for the conformance checking and validation of CGM files must have the capacity to check, verify and report important aspects of a CGM file. The tool must allow a careful analysis of the conformance to the rules of the CGM standard. Some of the essential domain­specific criteria for CGM validation tools are:

2. Does the tool verify ANSI/ISO 8632 and FIPS 128 specifications?

3. Does the tool verify CALS MIL-D-280003 application profile?

4. Does the tool report type, quantity, and location of CGM elements within the metafile?

5. Does the tool restrict checking to specific pictures in the metafile?

6. Does the tool allow for different levels of error checking detail?

7. Does the tool provide on-screen reports of private line types, markers, hatches, non­standardized Escapes and application data elements?

These errors occur when the Class/ID is not one of the standardized ones. The error indicates an encoding error.

These errors occur when the element is coded with either too little or too much data.

These errors occur when state conditions (appearance of an element in a particular section of the metafile) requirements are not met.

These errors occur when Required CGM elements (BEGIN METAFILE, END METAFILE, METAFILE VERSION, METAFILE ELEMENT LIST etc.) are missing.

These errors encompass over 150 range checks on CGM files.

These errors result when basic structure conflicts occur.

These are errors in state conditions pertaining to the CGM Application profile (Escape 301,303 in Metafile Descriptor and Escape 302 in Picture Descriptor).

These elements report errors in profile elements, CALS CGMs allow -301, -302, -303 and Generalized Drawing Primitive (GDP) Elements only.

Errors occur when profile parameters are out of the standard allowable in the CALS CGM Application Profile.

Errors occur when profile data limits exceed the limitations specified in the CALS CGM Application profile.

Errors are in color indices, background color, and etc.

Because the CGM has been designed to promote and advocate an open form of graphics interchange, it is sometimes very loosely defined. In addition to the capacity to handle errors, a CGM conformance tool in a test and validation tool repository must have design rules embedded to facilitate greater feedback to the user of the repository of the nature and features of the tool and the file being tested. A tool should contain the ability to evaluate the "goodness" of metafiles in addition to the rigorous conformance checking. The rules can be broadly divided into three categories:

General Rules.

Domain Rules.

Optional Guideline Rules.

The first two categories of rules establish successful and open interchange. Failure of metafile generators to follow these rules, in most cases, will lead to a file that is missing needed information or that contains misleading or incorrect information. Interpreters, other than those specifically matched by the generator, will in all probability not render the picture correctly if these rules and guidelines are not followed. The third category contains rules that will improve the chances that a third parties' interpreter will succeed with a given generator's metafile output. These are more subjective in nature when compared with the first two categories. The ability of a conformance tool to detect and report these rule violations greatly enhances the "worth" of the tool and should definitely be considered as qualification rules for CGM test and validation tools.

Syntactic Correctness:  A tool must detect and report these types of errors.

Semantic Completeness and Correctness:  A tool must check that metafiles are semantically complete and correct. The degree of complete and correctness of the tool can be seen from the error reporting capability of the tool.

Source Identification:  The source of the metafile must always be identified. The METAFILE DESCRIPTION element is used for this. A tool must recognize this in the metafile and report it.

5.3.2.3  Raster Domain Specific Criteria

There are two types of CALS raster formats:  Type I and Type II. Hence, different criteria apply for testing and validation of these formats. MIL-R-28000B specifies the applicable standards for both Type I and Type II raster formats.

1. Does the tool check for all aspects of CCITT T.6 Group 4 encoding scheme?

1. Does the tool test for conformance of ASN.1 representation?

2. Does the tool test for of conformance of Office Document Interchange Format (ODIF) representation?

3. Does the tool test for conformance of document structure and content?

4. Does the tool test for conformance to Document Application Profile?

5.3.2.4  IGES Domain Specific Criteria

IGES is a vector­based neutral format for the exchange of product model data. IGES follows ASME Standard Y14.26M and is a set of basic elements for computer CAD storage and transport.

IGES data specification should follow under one of the following data classes:

Technical illustrations subset.

Engineering drawing subset.

Electrical/electronic applications subset.

Geometry for NC manufacturing subset.

3D piping application protocol.

The National Institute of Standards and Technology (NIST) will be establishing the validation procedures for parsers/verifiers according to FIPS PUB 177 (IGES 4.0). These results of validation would fall into our classification of parsers in a test tool repository specific to IGES parsers and verifiers.

To be more specific, all IGES data will pass through a software checking sequence to be validated for the specified criteria. There are two major components used to validate IGES data. The first component, the Parser, checks the structure and syntax of the IGES file. The second component, the Verifier, checks the structural relationships and mathematical definitions of the IGES entities. The results of the analysis are found in a report file generated by the software.

The Parser component checks the general structure and syntax of an IGES file. Messages are produced indicating any structural or syntax errors found in the file. By checking the general structure of the IGES file, the parser ensures that all five of the required IGES file sections (Start, Global, Directory, Parameter, and Terminate) exist and are in the correct order. The parser also checks that the sequence numbers in each section are correct. If any of these criteria are not met, a fatal error is reported, and the file is not processed. If the file is found to be structurally correct, the parser will parse each section of the IGES file, check the data type and syntax of each field, and ensure that all the fields are properly delimited.

The Verifier component checks the structural relationships and mathematical definitions of the IGES entities. The verifier module also generates tables and statistics pertaining to the content of the IGES file.

The parser/verifier produces a report containing all messages generated by the parser and verifier components. This file also contains detailed statistics on the content and structure of the IGES file. This report file will be placed in the same directory as the original IGES file and will have the same name as the IGES file being checked, but the extension will be changed to ".ver". Furthermore, a summary of the results is displayed on the terminal from which the software is run.

In general, all digital product data files complying with the 28000 specification shall conform to the identified subset or application protocol class. The specific subset or application protocol class shall be identified in the start section of the files. All other specific requirements shall comply with Section 3 of the MIL-D-28000A handbook.

The following is a list of specific criteria for IGES testing tools.

General Criteria

Does the tool test for all aspects of FIPS 177?

Does the tool test for all aspects of MIL-STD-28000?

Does the tool test for all aspects of ASME (ANSI) Y14.26M?

The tool should check for the following specific IGES errors:

Invalid entity referenced.

The specification restricts which entities may reference which other entities. This includes Annotation, Wire Frame Geometry, Solid Geometry, Definition, and Structure.

Missing or invalid pointers.

This error check will check for occurrences of missing or illegally defaulted pointers.

The detection of invalid mathematical definitions.

This error check will check for occurrences of mathematical definitions where two or more parameters conflict with each other.

Missing or invalid data.

This error check will check for any occurrences of missing or invalid data.

Missing or invalid flags.

This error check will check for any occurrences of missing or invalid flags.

Missing or invalid line, color, and font values.

This error check will check for any occurrences of missing or invalid line, color, and font values.

CALS specific detectable errors:

Points with symbol pointers.

This will check for occurrences of points containing symbol pointers. They are not allowed in the CALS subset.

Drawings with annotation pointers.

This will check for drawings with annotation pointers. They are not allowed in the CALS subset.

Entities with associative pointers and illegal form.

This will check for entities containing associativity pointers. They are not allowed in the CALS subset.

Views with clipping planes.

View entities with clipping planes are not allowed in the CALS subset.

Nested transformations.

Nested transformations are not allowed in the CALS subset.

Missing properties

This checks for occurrences of missing properties required by the CALS subset.

Problems in the global sector.

This checks for incorrect global data parameters for the CALS subset.

Non-zero matrix, view, Z depths, and label display pointers.

These errors check for non-zero matrix, view, Z depths, and label display pointers in a file. None of the mentioned are allowed in the CALS subset.

Illegal colors, line fonts, entity types, composite curves, copious data, flags, levels, and entity relationships.

This error checks for any occurrences of illegal colors, line fonts, entity types, composite curves, copious data, flags, levels, and entity relationships for the particular CALS subset.

Illegally defaulted pointers and global values.

The CALS specification requires that certain global data must be present in the file as well as certain pointers that may be defaulted in standard IGES be present. This error will check for occurrences of illegally defaulted pointers and global values.

The following should be included in a listing of the general output file:

Check for file syntax.

Check summary and statistics such as file information, entity occurrences, and status flag.

Check entity analysis.

6.0  CALS TEST TOOLS ACQUISITION METHODOLOGY

      

The acquisition of CALS test and validation tools will be carried out in three distinct phases:  Identification, Screening, and Qualification. The purpose of product identification is to compile a list of potential CALS test tools and obtain the requisite information prior to product screening. The goal of product screening is to prioritize the list of potential products and perform a "paper-analysis" so that more detailed and time-consuming evaluations can be performed with a high acceptance rate. If a product successfully passes the screening phase, it becomes a candidate for the qualification process based on pre-determined criteria. The qualification process will lead to a recommendation from the evaluation team as to whether to accept the CALS test tool to the repository.

The evaluation team may also recommend some enhancements to the CALS test tools for acceptance. If the enhancements are approved, the product may pass through an adaptation phase. However, adaptation of test tools can be carried out under certain conditions:  ease of modification (availability of well-documented source-code, libraries, etc.), availability of resources, appropriate licenses, etc.

Eventually, all CALS test tools will be integrated within the context of IWSDB architecture. Figure 6.0-1 summarizes the steps involved in the acquisition and qualification of CALS test and validation tools.

APPENDIX A:  REFERENCE LIST

      

[ASSET92] Criteria and Implementation Procedures for Evaluation of Reusable Software Engineering Assets, ASSET, March 1992.

[CARDS93] Library Operation Policies and Procedures, Volume III, Library Development Handbook, Central Archive for Reusable Defense Software (CARDS), STARS-VC-B005/001/00, 29 Oct. 1993.

[CARDS94] Technical Concept Document, Central Archive for Reusable Defense Software (CARDS), STARS-VC-B009/001/00, 28 Feb. 1994.

[DOD92] DoD Software Reuse Vision and Strategy, CrossTalk, Oct. 1992.

[OSD94] OSD CALS Initial Assessment Report, OSD CALS IWSDB Project, DAAB10-89-D-0503, A001, 31 May 1994.

[PRISM92] Qualification Methodology Report, Portable, Reusable, Integrated Software Modules (PRISM) Program, 14 Jul. 1992.

[STARS92] STARS Reuse Concepts, Volume I, Conceptual Framework for Reuse Processes (CFRP), Version 2, STARS-UC-05159/001/00, 13 Nov. 1992.

APPENDIX B:  ACRONYM LIST