Submitted by

In support of

Contract DAAB10-89-D-0503

And in compliance with

CDRL Sequence Numbers A004 and A007

SOW Numbers 3.1.1, 3.1.2

TABLE OF CONTENTS

   

List of Figures

List of Tables

1.0  INTRODUCTION

2.0  OVERVIEW OF TESTING TOOLS

3.0  Technical Discussion

APPENDIX A:  EVALUATED TOOLS

APPENDIX B:  SGML SOFTWARE TOOLS

APPENDIX C:  BIBLIOGRAPHY

APPENDIX D:  CALS ACRONYMS AND ABBREVIATIONS

List of Figures

      

Figure 1.0-1 1840 and 28000 Series Military Specifications

List of Tables

      

Table 2.1-1 MIL-STD-1840B Conformance Checking Tools

Table 2.2-1 MIL-D-28000 Test and Validation Tools

Table 2.3-1 MIL-M-28001 Test and Validation Tools

Table 2.4-1 Test Tools

Table 2.5-1 MIL-D-28003 Test and Validations Tools

1.0  INTRODUCTION

      

This document outlines a study of test and validation tools used for the MIL-STD-1840 and MIL-STD-28000 series of standards. The tools are a means of establishing the conformance of digital data to the respective military standard. The establishment of a set of comprehensive tools to ensure compliance of contractor digital data deliverables to the Continuous Acquisition and Life-Cycle Support (CALS) formats will be the ultimate goal. The testing tools are a critical component of the CALS data acquisition process, because they ensure that only those components that meet the CALS guidelines are able to enter the system, for example the Integrated Weapon Systems DataBase (IWSDB) or a CALS reuse repository. The tools will serve as a "front door" in serving as a quality check upon which future functionality of those items allowed into a database or repository is based. A brief description of the standards follows.

The primary objective of this task is to provide the weapon system's Program Manager and acquisition personnel with the ability to effectively determine whether digital deliverables conform to their relevant CALS standards. This report discusses the current CALS Core Standards for technical information exchange and the test and validation tools being used to test compliance/conformance to the standard.

To ascertain the acceptance level that might be attainable given the test and validation tools in use today, we will outline procedures that investigate several areas designed to assist the acquisition/program manager in better determining which tools are available in a specific operating environment and to provide a pilot CALS Test and Validation Tool Reuse Repository. The repository would provide up-to-date information on CALS Test and Validation Tools from a reuse perspective, without duplicating capabilities and services presently being provided by other entities.

This report involves the results of surveying existing Commercial-Off-The-Shelf (COTS), Government-Off-The-Shelf (GOTS), and public domain CALS test and validation tools and establishing how they are being used today. Also, we will discuss the CALS Core Requirements standards and describe the levels of testing observed at the various Government agencies visited.

At the heart of the CALS initiative has been the exchange of technical product information in digital format. A key objective was and is the ability to enter data once and have it accessed and used over and over as a weapon system progresses through its life-cycle. The only reasonable way to discuss meeting the objectives of CALS is through the utilization of certain Core Technical Data Exchange Standards.

The creation of an IWSDB requires the capability to exchange technical information among diverse systems. Standards allow this to occur by defining neutral data formats that can be used to define data in a comprehensive, unambiguous manner enabling interoperability among Government and industry systems of many types. Because CALS is conceptually built around the ability to exchange digital data files between linked systems or to access data in a physically distributed but logically integrated database, CALS Standards for product data interchange provide the common rules that allow the whole process to move forward.

The following section describes the core CALS standards being used today for exchanging technical product data in neutral format. Figure 1.0-1 shows the 1840 and 28000 series of Military specifications.

1.1  Overview

This section details the standards, their architecture, the current status and implementation issues.

1.1.1  MIL-STD-1840B - Automated Interchange of Technical Information

MIL-STD-1840B serves as a central standard for the CALS environment. It supersedes and enhances MIL-STD-1840A and standardizes formats for the exchange of digital information between organizations or systems in order to facilitate the development and logistic support of defense systems throughout their entire life cycle. MIL-STD-1840B defines the format of the data to be exchanged in the CALS environment as well as the mechanisms and the parameters of those mechanisms, required for the exchange to take place. This standard is applied wherever information is exchanged. There are no restrictions to its application. It can be used for the exchange of American Standard Code for Information Interchange (ASCII) text files, product definition data files, raster image files, or vector graphics files.

Information is exchanged as a transfer unit. The method of encoding for the exchanged information is specified by contract. A transfer unit consists of a transfer unit declaration file, the appropriate transfer unit data file header records, and at least one transfer unit data file. The transfer unit declaration file is an ASCII file that provides all the needed information to uniquely identify the contents of the transfer unit. It provides such information as where the files came from, where they are going, what type of data is included, and how much data is being exchanged. All of these records must be present in the file and in the order stated by the standard. The medium of delivery is 9-track magnetic tape unless the involved parties mutually consent that an alternate method of exchange, such as floppy disks, shall be used. Alternate methods of exchange could include media such as floppy disks, Digital Audio Tape (DAT), magneto-optical disks, etc. The electromagnetic media will be packaged such that it is protected against dirt, moisture, or electrostatic discharges.

The MIL-STD-1840B standard defines, by reference, the formats and structures of the data files used for the transfer and archival of technical data in digital form. It clearly defines the formats, standardized header records, contents of the files used for the exchange of data as well as requirements during shipment for the labeling, protection, packaging, and the marking of media. The provisions for "protection" of media require that electromagnetically inscribed information transfer media such as encoded magnetic tapes and disks be protected against dirt, moisture, and electrostatic discharge damage during shipment.

This standard is not designed to be usable for specific applications, but is not restricted in any way in its application. This military standard is intended for technical information that includes product data, product acquisition and implementation data, and product support data. Product data includes engineering drawings, specifications, as well as new and evolving digital data forms that provide the data in a platform independent form directly usable by computer applications. Product acquisition and implementation data includes parameters and data necessary to manufacture and/or acquire an entire defense system. Product support information includes training and maintenance manuals with their associated illustrations needed to maintain a defense system in a required state of readiness. The scope of this data covers the entire life cycle of a weapon system.

MIL-STD-1840B provides general requirements for Technical Publication, Product Data, and Electrical/Electronic Application Data File document types. The Technical Publications document type includes files that contain MIL-M-28001 Standard Generalized Markup Language (SGML), Document Type Declaration with no text, Formatting Output Specification Instance (FOSI), SGML text entity, MIL-D-28000 Initial Graphics Exchange Specification (IGES), MIL-R-28002 Raster, MIL-D-28003 Computer Graphics Metafile (CGM), Page Description Language (PDL), gray scale or color illustration, or special word data.

In any typical application, the hardware and software must prepare the files for transfer on the sending system by adding header information and writing the files to the selected media type. The hardware and software on the receiving system must process the files by reading the files from the selected media type and stripping the added header information. The media must also be labeled, protected, packaged, and marked appropriately during shipment in accordance with the MIL-STD-1840B standard.

1.1.1.1  Architecture

MIL-STD-1840B is composed of the following six sections:

1.1.1.2  Status and Planned Extensions

MIL-STD-1840B was released on November 3, 1992. It supersedes MIL-STD-1840A which was released 5 years earlier on December 22, 1987. The next version of this standard is expected in late 1995.

MIL-STD-1840B is required for the delivery of technical CALS data. It is approved for use by all agencies of the Department of Defense (DoD). It has been implemented and used successfully by numerous vendors. The use of MIL-STD-1840B will undoubtedly continue to grow because this standard is at the heart of the CALS interchange environment. The Department of Defense will continue to require that CALS procurements adhere to the requirements of this standard.

The future revisions of MIL-STD-1840B will address solid modeling for system design, interactive retrieval and use of technical information, and other potential computer applications for defense systems of the future. The interactive retrieval and use of information will be necessary for the implementation of the Contractors Integrated Technical Information Service (CITIS) and the IWSDB. It is expected that expert systems will be included in future revisions.

MIL-STD-1840 continues to recognize the format required for IGES information but does not attempt to address the Standard for the Exchange of Product data model (STEP) or Product Data Exchange standard using STEP (PDES) formats. This will be done in future revisions of MIL-STD-1840.

Many of the standards and specifications required or referenced by MIL-STD-1840B are evolving significantly due to rapidly advancing technologies. These will have to be further implemented in future revisions of this standard.

Efforts are currently under way to form a recognized working group with international participation. A Target Capability (TCAP) document is being developed for circulation among the CALS Industry Steering Group (ISG). It recommends that the following areas be addressed in the next revision of MIL-STD-1840:  

1.1.1.3  Advantages

MIL-STD-1840B, a standard for interchanging digital technical data, is a core standard for the CALS environment. In addition to the advantage of being required for the delivery of CALS data, this standard has the advantage of having an essential function which facilitates the sharing of technical data within and among autonomous organizations.

This standard clearly defines the mechanisms for exchanging digital data with formats defined in other CALS specifications (MIL-D-28000, MIL-M-28001, MIL-R-28002, and MIL-D-28003). The reference to and use of other standards and specifications allow this standard to evolve, as those standards and specifications evolve, in order to take advantage of advances in current technologies or to use new technologies.

MIL-STD-1840B contains specific detailed instructions for using 9-track magnetic tapes as a medium for exchange of digital data. It contains flexible provisions that rely on agreements between sender and receiver for using other media such as diskettes, Write Once Read Many-times (WORM) optical disks, and Compact Disk Read Only Memory (CD-ROM) for the exchange of technical data. This reliance on agreements provides MIL-STD-1840B with the advantage of being accommodating to changing user needs as well as being able to adapt to new requirements and/or guidelines.

Another advantage of MIL-STD-1840B is that it clearly defines the formats, standardized header records, and the contents of the files used for the exchange of data as well as requirements for the labeling, protection, packaging, and the marking of media during shipment. The standard also addresses electronic product data, new packaging of data for electronic trade business transactions, and electronic product data technology.

1.1.1.4  Implementation Issues

MIL-STD-1840B is required for the delivery of CALS technical data. However, the media for delivery must still be selected. This standard specifies the media for exchange of CALS information and includes provisions for using nine-track magnetic tapes, diskettes, WORM optical disks, and CD-ROM.

Although given recognition as viable CALS media, actual specifications for CD-ROM and WORM optical disks are not clearly defined in this standard. The generic description of requirements leaves room for portability problems in the areas of placement, arrangement, and structuring of files on media other than 9-track magnetic tapes. Informal agreements between sender and receiver must be made to ensure complete portability when CALS data is interchanged on any medium other than 9-track tapes. In addition, other media such as the Bernoulli removable disk drives are not considered, thereby leaving the impression they are not to be used.

Corresponding hardware and software must be present at the preparing (source, sending) site and the receiving (destination) site to accommodate the selected media for the information interchange. The files on the sending system must be prepared for transfer by adding header information to them and writing them to the selected media type. The files must be processed on the receiving system by reading them from the selected media type and stripping the added header information. The configuration management functionality of MIL-STD-1840B requires careful management within the software to assure that correct information is added to the file headers.

The CALS Test Network (CTN) was established to perform end-to-end testing of the exchange of digital data using MIL-STD-1840B. "CALS-compliant" systems must have the capability of reading and generating data or media conforming to MIL-STD-1840B. Because MIL-STD-1840B is at the heart of the CALS interchange environment, DoD will continue to require that CALS procurements adhere to the requirements of this standard.

MIL-STD-1840B was developed by the National Institute of Standards and Technology (NIST) under the direction of the Office of the Secretary of Defense (OSD) CALS Office. All of the CALS standards including MIL-STD-1840B are currently being managed by the CALS Digital Standards Office (CDSO) at the Wright-Patterson Air Force Base, Dayton, Ohio.

1.1.2  MIL-D-28000A - Digital Representation for Communication of Product Data (IGES)

This specification defines the CALS compliant subsets of the IGES for the interchange of vector graphics. The focus of this standard is Computer-Aided-Design (CAD) information, primarily the digital representation of geometry. IGES is a vector-based neutral format for the exchange of product model data. IGES is the most widely used product data exchange specification in the US. It was designed specifically for CAD/Computer Aided Manufacturing (CAM) systems for the exchange of geometric models and drawings. The data format is in ASCII with a fixed line length. MIL-D-28000 IGES is organized into five classes by application area to meet the delivery need of that application. These classes are:

1.1.2.1  Application Subsets

The first four classes specify the entities needed for specific application subsets. In this way, the recipient of a MIL-D-28000 data file may specify the class of data needed without becoming an IGES expert. The Class V application protocol is for the exchange of product data for 3D piping system models only.

Most CAD systems have some type of IGES translator, and even some non-CAD systems support the IGES specification. The support for MIL-D-28000 (subsets) is not as widespread as the support for the full IGES standard. The greatest stated support of the subsets comes from the commercial flavoring software and syntax checking software. MIL-D-28000 Class II, engineering drawings, is the most commonly supported class.

The first four classes of MIL-D-28000A specify the entities needed for specific application subsets. In this way, the recipient of a MIL-D-28000A data file may specify the class of data needed without becoming an expert on the IGES. The MIL-D-28000A application subsets specify the entities allowed in that class through a list of ASME Y14.26M entities given in table form. Limits on the entities are given through notes on that table. Rules are also given for the entity construction. Guidance is provided for MIL-D-28000A file construction requirements for each section (start section, global section, directory entry section, parameter data section, and terminate section) of an IGES file for each class.

The four application subsets defined within MIL-D-28000A are described in the following paragraphs.

1.1.2.2  Application Protocols

An Application Protocol (AP) is a way to transfer defined product data through IGES. An AP documents the user requirements for an application in a graphical model called an Application Reference Model (ARM). The requirements in the ARM are then represented by specific IGES entities in a given AP (the AIM). APs enable IGES to be used to transfer product data reliably until PDES/STEP is available from the commercial CAD vendors. APs provide a defined and more reliable method for transferring product data through IGES.

An AP is composed of the following elements:

APs are very specific in nature. For example, the 3D Piping AP (Class V) exclusively supports the exchange of product data for 3D piping system models. It does not support piping engineering drawings. A user wishing to transfer an engineering drawing of a piping system would have to use an Engineering Drawing AP. Also, only CAD/CAM systems supporting piping will be able to support the piping AP. A CAD/CAM system that does not support piping does not have the appropriate constructs within its database to either output data in the Piping AP or input the data reliably. APs will provide increased information transfer, but with a much narrowed scope in the information that is transferred.

1.1.2.3  Status and Planned Extensions

MIL-D-28000A is based upon an underlying American National Standard (ASME Y14.26M - 1989) and the IGES Version 5.1, both of which were developed by the IPO. As such, the Office of the Assistant Secretary of Defense (OASD) CALS Evaluation and Integration Office (EIO) cooperates with the voluntary IPO through the IPO's CALS/IGES Special Interest Group (SIG). The CALS/IGES SIG reviews proposed changes and suggests new changes at the IPO meetings. The CALS/IGES SIG is a source of IGES technical expertise and is instrumental in ensuring the quality of revisions or amendments to MIL-D-28000A.

A new application protocol, the Engineering Drawings Application Protocol (AP), is being developed for MIL-D-28000A. It is being created by members of the Navy Industry Digital Data Exchange Standards Committee (NIDDESC) and the IGES/PDES Organization. The AP will be submitted to the OASD CALS EIO for inclusion in the next amendment or revision of MIL-D-28000.

1.1.2.4  Implementation Issues

The method by which the senders of a MIL-D-28000A file will produce application subsets is a possible concern for the implementation of MIL-D-28000A. The preferred method is for the originator's CAD system's IGES translator to produce the MIL-D-28000A file. An alternative method is for the CAD system to produce the IGES file that is subsequently run through commercial flavoring software to produce the MIL-D-28000A compliant file. This method must be performed very carefully to prevent any loss of the file's underlying structure, and is not suited for the transfer of application protocols.

Even perfect transfer of the application subset does not ensure that all of the information in the original CAD model will be translated. For example, a CAD system may recognize objects such as resisters and capacitors in its internal database, but because IGES has no standard way to represent such objects, these objects may be transferred as a grouping of points, lines, and curves that represent the object. The concept that a group of entities represent an object is not necessarily conveyed by the subset to the receiving CAD system. MIL-D-28000A displays an awareness of this problem by specifying that "it is the intent of this specification to evolve in the direction of application protocols to ensure quality data exchanges." The application protocol work is being developed within the IGES/PDES Organization to transfer objects within an application area instead of merely a geometric representation with little standardized intelligence attached. The 3D Piping AP is the first AP developed by the IPO.

Most IGES entities are general purpose in nature. They can be combined to create constructs needed for product data transfer, such as a circuit in an electrical application, but they do not rigorously define how this is done. This can be a problem in transfer, because unless the receiving system knows how the IGES entities were combined to create the construct, the receiving system may not be able to interpret it. The basic data will be translated, but all the information needed to translate product data for the application will not be available.

MIL-D-28000 does not contain any rationale for why a specific set of entities was chosen for an application subset over another set of entities. This can make extensions to the subsets laborious, and raises many questions from the user and vendor community.

1.1.2.5  Testing and Validation

The CTN was established to demonstrate the CALS standards, test their effectiveness, and identify needed improvements to the standards. Currently, the CTN tests only the CALS data interchange standards. The CTN uses naturally occurring tests and structured tests. After the test results have been reviewed and approved by the CTN, the test reports are put on the CALS Bulletin Board for public view. The CTN testing performed by Industry participants and lead testbeds is directed by the CALS Test Network Office reporting to the Standards and Technology Division of the CALS EIO.

The CTN has test packets for testing MIL-D-28000 Class I and II, which currently need to be upgraded to the MIL-D-28000A version. Test cases for Class V are currently under development. The CTN does not have test packets for Classes III or IV. The main contents of the CTN test packets are IGES files to process into the CAD system, scripts to follow in creating the data on the CAD system for subsequent output, and plots to show the expected results. The test results from the naturally occurring tests are called "Quick Short Test Reports" (QSTRs) and are available from the CTN or the CTN Bulletin Board.

1.1.3  MIL-M-28001B - Markup Requirements and Generic Style Specifications for Electronic Printed Output and Exchange of Text (SGML)

This standard defines CALS requirements for automated, page-oriented, ASCII-text-based documents. Included in 28001 are SGML for source data, output specifications for composition processing, and page description language options for composed document display.

The SGML was developed to address the problems of information exchange and management challenges associated with electronic printed output and exchange of textual information. The ISO participated in the development of the SGML standard and adopted it in 1986 as ISO 8879. This standard completely defines the terms and syntax to specify the structure and content of a document. ISO 8879 does not provide functionality for specifying appearance or formatting requirements.

The DoD application of SGML is specified in MIL-M-28001. ISO 8879 defines SGML as a meta-language. Many different SGML languages satisfying the grammar and syntax specified in ISO 8879 can be obtained by choosing from among different features and options afforded by ISO 8879. MIL-M-28001 specifies the DoD CALS SGML implementation by choosing certain SGML features and setting certain parameters. MIL-M-28001 also uses SGML to provide a capability for specifying the formatting of CALS SGML documents.

An ISO 8879 SGML document contains three parts; the SGML declaration, the Document Type Definition (DTD), and the Document instance. A CALS SGML document contains these parts and may also contain a Formatting Output Specification Instance (FOSI). The SGML declaration defines what characters will be allowed in the rest of the document and how they will be encoded. The SGML declaration also specifies how certain characters are to be interpreted along with other special rules and definitions. The DTD defines the content and structure of a document. It refers to how the information in a class of documents such as technical manuals, books, memos, etc., are related, and defines any dependencies. Markup of a document provides an unambiguous definition of its structure and content, allowing automated data processing software to process the document in a predictable manner. Also defined in the DTD are entity declarations. The document instance is the information content of the document, marked up in accordance with a DTD. This markup may include declarations, tags, and entity references. The FOSI specifies the desired appearance of the information content of the document. This output formatting description capability is not contained in ISO 8879 but is found in MIL-M-28001. MIL-M-28001 contains an Output Specification (OS) DTD that defines how FOSIs are to be developed and interpreted.

"SGML markup" (or an "SGML document instance") consists of unformatted text with inserted SGML "tags" that correspond to the elements and attributes of the DTD. These tags identify elements of the text (e.g., titles, paragraphs, tables, footnotes) defined in the document's DTD. The "marked up" document (or SGML document instance) can then be "parsed" using special software to determine if the document's tagging conforms to the DTD.

Unlike MIL-M-28001A which contained the DTD for documents conforming to MIL-M-38784B and 12 DTDs based on that DTD, MIL-M-28001B does not contain any DTDs to be used for delivering data to the Government. Like MIL-M-28001A, MIL-M-28001B will provide the so-called "template" DTD in Appendix A. Its chief function is to serve as a "toolkit" for the construction of DTDs by providing SGML coding that can be incorporated or modified in DTDs being developed. The template DTD also provides a set of elements and attributes for use in new DTDs.

A "declaration subset" is used to define a new DTD in terms of changes to an existing DTD. The implementation of the changes in a declaration subset results in a complete and different DTD for the corresponding military specification. The use of such "declaration subsets" in creating new DTDs this way actually allows tighter control over the number of distinct DTDs. While DoD wishes MIL-M-28001B to be implemented in a wide variety of applications, DoD is also quite concerned with the uncontrolled proliferation of DTDs.

New DTDs must be developed for all applications of automated technical publications for which existing CALS DTDs are inadequate. New DTDs should be constructed using those elements and attributes of the "template" DTD as defined in Appendix A of MIL-M-28001B whenever possible. This appendix provides general guidance for development of DTDs. The DTD for a given class of documents such as technical manuals will either be provided in the governing specifications for such documents or else be completely specified within the governing specifications.

1.1.3.1  Applications

The development of those CALS DTDs contained in earlier versions of MIL-M-28001 and based on the DTD for MIL-M-38784B conforming documents was a coordinated effort of the Navy, Army, Air Force, and Industry. MIL-M-38784B has been superseded by MIL-M-38784C, and the DTD for documents conforming to MIL-M-38784C is provided in Appendix B of MIL-M-38784C. In addition, numerous other DTDs were developed, including those DTDs developed by the Air Force under the Technical Manual Specifications Standardization (TMSS) program, as well as a Work Package DTD developed for Naval Air Systems Command (NAVAIR). These DTDs will be included or specified in the appropriate functional specifications.

Currently, MIL-M-28001B is concerned with the digital interchange of paper-based manuals. However, efforts are underway to define the digital interchange of "paperless," i.e., screen medium technical publications. The MIL-D-87269 specification uses SGML to specify a revisable database for the support of interactive electronic technical manuals.

1.1.3.2  Architecture

MIL-M-28001B is composed of the following six sections and 3 appendices:

1.1.3.3  Status and Planned Extensions

It is in the interest of both DoD and industry to agree on the widest applicable set of conventions for the preparation and interchange of publications for defense and non-defense use. MIL-M-28001B has identified applications that provide a more comprehensive specification for the interchange of ASCII data. Such applications include the specifications for an Output Specification, Electronic Review, and Partial Document delivery. These specifications represent the chief enhancements of the MIL-M-28001 specification achieved in its "B" version. They are briefly discussed below.

1.1.3.3.1  Output Specification

In order to format an SGML source file, associated formatting information must be provided. This associated formatting information must define formatting characteristics such as a page model, font and family characteristics, point size, indenting, etc. In addition, these formatting characteristics must be responsive to certain SGML tags. For example, a "paragraph" tag may trigger a change in the line leading, or a "chapter title" tag may trigger "bolding" and "center" functions. The Electronic Publishing Committee of the CALS Industry Standards Working Group formed a "MIL-M-28001 Output Specification Ad Hoc Group" to develop a standard language for providing the associated formatting information of SGML instances. It was decided to use SGML itself for this purpose and provide the associated formatting information in the form of an OS DTD. Appendix B, Section 50 of MIL-M-28001B contains the CALS paper medium OS DTD.

The OS DTD defines a finite set of formatting characteristics used to rigorously describe the composition processing functions to be performed with respect to the tags of a SGML source file. A FOSI is an instance of the OS DTD. The FOSI defines values for the formatting characteristics defined in the OS DTD for every SGML element used in the document DTD, taking into account every context in which the SGML element has a unique formatting requirement. For example, a title of a Technical Manual (TM) chapter is formatted differently than a title of a TM subparagraph. The objective of the FOSI is to rigorously define the format style of the document to be produced from the SGML tagged source file, as required by the appropriate functional specification (MIL-M-38784C, etc.).

A FOSI should be developed for each DTD to describe all default formatting characteristics necessary to compose and publish a document authored according to that DTD. The FOSI should be delivered with the SGML source tagged file. Because all FOSIs will be written with respect to the standard OS (paper medium), vendors will be able to develop software that can accept and process FOSIs and interface with the publishing software. However, such automatic processing of a FOSI is not a requirement of MIL-M-28001B.

1.1.3.3.2  Electronic Review

Section 80 of Appendix C of MIL-M-28001B provides a mechanism that enables an electronic review and comment capability for SGML tagged source files. This capability allows reviewers located in diverse environments to make and exchange comments electronically on multiple copies of a document file over a network. The comments may then be sorted, processed, and incorporated into the document by the file "owner."

The mechanism for electronic review of SGML tagged source files consists of certain SGML constructs that are incorporated into a DTD for a given document type. These SGML constructs have been defined as generally as possible to take into account the many kinds of reviews such as internal contractor reviews, Government reviews, contractor/Government reviews, specification reviews, etc.

Plans for future extensions of electronic review include both a CALS graphics comment capability using SGML tagged for the comments, and a capability to link SGML text and CALS graphics files for related changes. Efforts will also be made to develop a more precise addressing mechanism for indicating the location within document elements of a proposed change.

1.1.3.3.3  Partial Documents

Partial document delivery is used to transmit source SGML tagged source data either as an interim deliverable or as an update package containing data for a document that has been previously delivered. Its purpose is to minimize the retransmittal of unchanged data or to indicate incomplete data. Partial document delivery is not intended to address the issues of page integrity or fidelity, nor is it intended to include specific change pages. The intent of this methodology is to allow the delivery of certain portions of a source document such that the receiving system can identify the location of the information in the original document and perform the appropriate addition, deletion, or replacement operations. Both the manner in which this is accomplished and the effect of the change on composition depends on the receiving system.

1.1.3.3.4  CALS SGML Registry/CALS SGML Library

One of the Ad Hoc Groups of the Electronic Publishing Committee is investigating the requirements for the development and maintenance of a CALS SGML Library (CSL).

It is envisioned that a CALS SGML Registrar will administer the CALS SGML Registry (CSR), a central registry office where DTDs and FOSIs will be registered. The CSR will maintain a CALS SGML Library that will be an on-line database containing all SGML elements and attributes that have been defined, with cross references to DTDs and governing military specifications. It is anticipated that all CSR/CSL requirements will be specified in a future revision of MIL-M-28001.

The CALS SGML Registry will require adherence to basic guidelines for acceptance of SGML tags/attributes. Some of these guidelines are:

1.1.3.4  Advantages

MIL-M-28001B provides SGML applications or "conventions" that can be applied in the automation of document production and management. These applications include the Output Specification, Electronic Review, Partial Document Delivery, and the "template" DTD tagset of elements and attributes. These innovations that are oriented toward DoD and industry needs will provide a comprehensive specification for the interchange of ASCII data. They constitute both a far-sighted adaptation and a wide-ranging application of ISO 8879 to the rapidly changing technology of the printing and publishing industry.

1.1.3.5  Implementation Issues

MIL-M-28001 has undergone extensive revisions since its initial publication on February 26, 1988. MIL-M-28001B contains various specifications and/or recommendations for the interchange of data. Some of these have not been thoroughly tested, including:

Currently, there are no tests for vendor products claiming conformance to MIL-M-28001. Such MIL-M-28001 product conformance testing will depend upon the product's function. For instance, conformance testing of SGML parsers entails the correct interpretation of ISO 8879. Conformance testing of "auto-taggers" or "authoring stations" would be limited to determining the parseability of the instances generated, and again would involve correct ISO 8879 interpretation. With few exceptions, there is no disagreement regarding the correct interpretation of ISO 8879.

However, conformance testing of CALS SGML publishing systems involves MIL-M-28001 compliance, but MIL-M-28001 does not rigorously define system requirements. For example, while MIL-M-28001 specifies the requirements of a FOSI, it does not require a system to automatically process such a FOSI. Therefore, MIL-M-28001B conformance testing of a publishing system would likely be limited to tests for CALS data acceptance and valid document formatting.

1.1.3.6  Extent and Nature of User and Vendor Support

The vendor community is aware of the evolving nature of MIL-M-28001. Some vendors are waiting until the standard is finalized, while other vendors are undertaking full implementations at the present time. A large vendor community is represented on the Electronic Publishing Committee. For the CALS environment, vendors supporting MIL-M-28001 should not "hard-code" their systems to process only a single DTD or FOSI. Certainly, most users will be processing a variety of technical publications that must conform to multiple DTDs and will require a system that can be configured to adapt to new and changing requirements as they arise.

Currently, there are various types of SGML software products on the

market. These include:

1.1.4  MIL-R-28002B - Requirements for Raster Graphics Representation in Binary Format (Raster)

The MIL-R-28002 specification establishes requirements for a standard interchange file format and raster encoding scheme for raster data. MIL-R-28002 (Type I and Type II as defined in NISTIR 88-4017) was first issued in December 1988. It was revised by MIL-R-28002A in November of 1990 and again by MIL-R-28002B, December 14, 1992. This specification identifies the requirements to be met when raster data represented in digital, binary format is delivered to the Government.

Raster graphics involves the digital processing, storage, exchange, and reproduction of images. This technology supports the binary representation of a two-dimensional image as an array of picture elements, also known as pels. Each pel of the array contains information about that portion of the image. This information may include its lightness, darkness, gray-level, and color. The quality of the image depends directly on the density of pels within the array, also known as resolution density or pel transmission density. A high resolution density provides a high quality image, but at the expense of higher storage costs. Data compression, in which an encoding scheme is used to represent redundant data bits of information, can alleviate this storage problem to some extent. MIL-R-28002 restricts such compression to Group 4 encoding as defined in Consultative Committee on International Telephone and Telegraph (CCITT) Recommendation T.6 (Federal Information Processing Standard (FIPS) PUB 150) in order to conform with developing industry standards.

MIL-R-28002 permits two types of digital representation of raster data, called Type I and Type II in the specification. The Type I file format is used for raster data contained in a single monolithic block of compressed data and is called untiled raster data. The Type II file format is an Open Document Architecture (ODA) document (as specified by ISO 8613 ODA) conforming to a special application profile for raster. Type II may be tiled raster data or may consist of a single compressed block of data as in Type I, but with all ODA parameters and data structuring included.

Tiled raster data consists of an image that is subdivided into non-overlapping regions known as tiles where each tile is treated as a separate pel array. This method is especially useful for mechanical drawings in which there are large open areas of space. Within a single image, tiles are equal in size, and their dimensions, specified in terms of pels, have certain limitations. Tiles can be compressed and manipulated to obtain an optimal raster file. MIL-R-28002 specifies that individual tiles be digitized and the data compressed in accordance with Group 4 encoding defined in Recommendation T.6 (FIPS PUB 150).

Type I raster data interchange is intended to be used in procuring data for systems that use only untiled raster data representations. Examples of such systems include typical technical documentation systems, the Air Force Engineering Data Computer-Assisted Retrieval System (EDCARS) and the Army Digital Storage and Retrieval Engineering Data System (DSREDS). A set of graphics attributes specifying the details necessary for processing and reproducing the image must be included in a header record at the beginning of the raster file. These attributes include the size of the original image, the scanning resolution, the image orientation (portrait or landscape), the starting position on the page, and the spacing between the pels and also between the lines containing the pels. These attributes are used in reproducing the image and apply to both Type I and Type II raster data files.

Type II raster data interchange is intended to be used in procuring data for systems that need the flexibility to use tiled or a mixture of tiled and untiled raster data representations. Tiled representations are best applied in systems handling large format drawings or illustrations typically associated with engineering design. The subdivision of a drawing into tiles allows the use of only those portions of an image required at a given time by the application. This can result in reduced requirements for workstation memory and display. The attributes required for Type I are also required for Type II data and are encoded in the ODA data stream as specified by the ODA Raster DAP (explained below). For Type II data, additional attribute information must be included to cover the size of each tile, the number of tiles in the array (image), the method of tile ordering, and the method of tile coding. This information is stored in the header record of an image file during the scanning process and is essential for reproducing the image.

1.1.4.1  Typical Applications

MIL-R-28002 was created for the storage and interchange of scanned engineering drawings but applies to other documents as well, such as technical manuals and illustration in raster form. Appendix A of MIL-R-28002B contains the ODA Raster Document Application Profile (DAP). The DAP specifies an interchange format suitable for transfer of structured documents between equipment designed for raster processing. The documents supported by the Raster DAP are based on the paradigm of an electronic drawing or illustration. Such documents contain one or more pages. Each page consists of an image in the form of a bi-tonal raster graphic content. The features of a document that can be interchanged using the Raster DAP fall into the following categories:

1.1.4.2  Architecture

MIL-R-28002 identifies the requirements to be met when raster data represented in digital, binary format is delivered to the Federal Government. This specification identifies the storage and transmission format of raster data and tiling conventions for document pages and large format engineering drawings interchanged as raster images. All digital raster data files complying with MIL-R-28002 shall conform to either the Type I or Type II binary formats defined in the specification.

As specified in MIL-STD-1840, a set of graphics attributes specifying the details necessary for processing and reproducing the image is contained in a header record at the beginning of a raster file. These attributes include an indication of the raster data type, the size of the original image, the scanning resolution, the image orientation (whether it be portrait or landscape), the spacing between the pels, the spacing between the lines containing the pels, and the bit ordering. These attributes are used in reproducing the image and apply to both the Type I and Type II raster data formats.

Type I data is CCITT T.6 encoded data for an entire scan representation enclosed within MIL-STD-1840 header information. The CCITT T.6 encoding of raster data is defined in FIPS PUB 150, "Telecommunications:  Facsimile Coding Schemes and Coding Control Functions for Group 4 Facsimile Apparatus." Type I has no support for tiling, but has the virtue of simplicity.

Type II data is a MIL-STD-1840 header wrapped around an ODA document as specified in the ODA Raster DAP. The ODA document may consist of a single compressed block of data as in Type I, or it may be tiled. ODA parameters and data structuring must be included.

1.1.4.3  Status and Planned Extensions

MIL-R-28002B was published in December 1992. As a result of international harmonization activities, changes were made to the March 1993 version of the ODA Raster DAP. The "application comments" attribute will now consist of two fields to be compatible with other International Profiles, and the value for the "ODA version" attribute was changed. DoD decided that these changes were significant enough to warrant an amendment to MIL-R-28002B. The amendment will contain changes to Appendix A (the ODA Raster DAP) and is not expected to impact the MIL-R-28002B implementation of the DAP. Therefore, current implementations of MIL-R-28002B will be upwardly compatible with the amendment. This amendment is expected before the end of the 1993 fiscal year.

NIST is planning to issue the ODA Raster DAP as a FIPS PUB by the end of 1994. Once the FIPS is in place, MIL-R-28002 will no longer contain the ODA Raster DAP as an appendix, but will reference the appropriate FIPS.

1.1.4.4  Implementation Issues

The development of Type I data capabilities has been evolving for some time and has stabilized. The CTN digital raster data interchange testing for Type I data has aided many present and potential DoD contractors in their efforts to develop hardware and software that are technically capable of accomplishing MIL-R-28002 Type I interchanges. In particular, the CTN Engineering Data Transfer Test with the Engineering Data Management Information and Control System (Navy) (EDMICS) using MIL-R-28002 (May 1992) reported that MIL-STD-1840A tapes generated by EDCARS and DSREDS were successfully converted from CALS (MIL-R-28002) to EDMICS native format and visually displayed on EDMICS. In principle, this demonstration supported the viability of tri-service data interoperability via CALS media. CTN has published several test reports that describe Type I testing results for a variety of vendor implementations. NIST has begun Type I conformance test development.

Type II data, on the other hand, is a considerably newer and more complex environment. ODA implementors maintain that a detailed understanding of the ASN.1 Basic Encoding Rules (required by ODA) is not required to understand MIL-R-28002B Type II. A vendor or user need only implement the Raster DAP, not the entire ODA standard, to achieve a MIL-R-28002 Type II implementation. Users may employ a library of ASN.1 routines to avoid having to fully understand encodings at the bit or byte level. The (NIST) report, NISTIR 5108, "Raster Graphics:  A Tutorial and Implementation Guide," is an excellent resource for those needing a better understanding of MIL-R-28002 and how to implement it.

A system that can receive (read) and output (write) MIL-R-28002B Type I and Type II data will be considered a compliant CALS implementation. The internal raster data format of the system need not use ODA. Cost comparisons between the implementation of data translators verses designing the system's internal format based on ODA should be performed to determine what is the best MIL-R-28002B (Type II) implementation strategy for each Navy raster system.

The CTN Raster Test Bed at the Lawrence Livermore National Laboratory (LLNL) and David Taylor Model Basin, Carderock Division of the Naval Surface Warfare Center (CDNSWC) have been involved with the testing and validation of raster image data provided by Industry and DoD. InterLinear Technology, in support of LLNL, has developed a Test tool called "ODATOOL" to evaluate CALS raster image data formatted according to the MIL-R-28002A. CTN has tasked InterLinear to update the ODATOOL in accordance with MIL-R-28002B. InterLinear offers commercial products to write, convert, and read ODA files. At present, only the Modular Electronic Document Information Solution (MEDIS) systems by InterLinear Technology support MIL-R-28002 Type I and II.

1.1.5  MIL-D-28003A - Digital Representation for Communication of Illustration Data:  CGM Application Profile (CGM)

MIL-D-28003A is a proposed military specification defining an application profile for delivery of two-dimensional technical publication illustrations using the CGM standard. CGM application profile requirements defined by MIL-D-28003 address the first of several classes of conforming basic meta-files, a conforming basic metafile generator, and both a minimum-level and a publication-level conforming basic metafile interpreter. Generator and interpreter requirements are specified to provide necessary controls over the creation and validation of conforming metafiles and to remove implementation dependencies. The metafile definition itself includes the allowable output primitives and attributes that may be used to compose the illustration. Although the CGM standards that MIL-D-28003 implements define a complete and unambiguous syntax, they intentionally under specify the semantics, or meaning, of the CGM primitives and attributes to accommodate a wider range of existing systems, and to make the standard more adaptable to the requirements of diverse applications and users. The CALS application profile provides an unambiguous interpretation of those semantics appropriate for illustration of technical publications.

The Computer Graphics Metafile standard is a published international standard (International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 8632), a Federal Information Processing Standard (FIPS 128), and has been adopted by the American National Standard Institute (ANSI). The CGM standard is being developed and maintained through a coordinated effort of ISO SC24 and ANSI X3H3. The U.S. and international standards are identical.

The overall intent of the CGM standard is to provide the lowest level of drawing functionality required to capture and reproduce a picture, in a way that is portable across applications and devices. The CGM standard specifies two-dimensional graphics data interchange in a file format that can be created independently of device requirements and translated into formats needed by specific output devices, graphics systems, and computer systems.

A metafile is a device-independent, application-independent storage format for the exchange of the data that makes up a picture ("picture data"). The metafile definition in ISO/IEC 8632 includes a definition of output primitives and attributes that may be used to compose an illustration, but in an intentionally under-specified semantics (meaning). This was done to accommodate a wide range of existing systems, and to make the standard more adaptable to the requirements of diverse applications and users. The application software wishes to store a picture on a metafile. To do this, it has to write all the information required to a file. The software that performs these writing actions is called the generator. The software that reads a metafile back into an application is called an interpreter.

A profile addresses implementation conformance requirements for the generator and interpreter. For generators, the graphical characteristics of the picture are mapped onto a set of CGM elements that define the pictures within the accuracy and latitude defined by the implementation requirements in the profile. For interpreters, the graphical characteristics of the CGM elements are rendered into a graphical image or picture within the latitude defined by the implementation requirements in the profile. Without a profile, one can address only the syntactical correctness of the data stream. With a profile, one can address and test that the picture is correct. A profile provides a way of standardizing and publicly specifying the options that a vendor supports and how they are to be supported.

The three CGM encodings meet different needs, but all may be interchanged without loss of information. The binary encoding facilitates rapid graphic data processing. The character encoding is compact and transportable. The clear text encoding is human readable and editable. Only the binary format is approved for use by MIL-D-28003.

ISO/IEC 8632 CGM is an upwardly compatible standard format, developed in three versions that offer steps in capability. Version 1 includes elements of ISO 8632 CGM:1987. Version 2 is also based on ISO 8632:1987, but adds capabilities through Amendment 1:1990. Version 3 incorporates ISO 8632:1987, Amendment 1 and Amendment 3:1991. This has resulted in major increases in capabilities that are represented by ISO/IEC 8632:1992, the current version of the CGM standard. ISO/IEC 8632:1992 provides a clear definition of Version 1, 2, and 3 metafiles.

The Military specification, MIL-D-28003, "Digital Representation of Illustration Data:  Computer Graphics Metafile," first published December 20, 1988, was superseded by MIL-D-28003A, published November 15, 1991. Amendment 1 to MIL-D-28003A was published August 14, 1992.

MIL-D-28003 is the CALS Application Profile for CGM, developed to provide more information and requirements for specific application areas that relate to the CALS environment. It is designed for a multi-system environment that involves a wide range of applications. It is also designed to increase the conformance requirements for CGM, to guarantee final results when the pictures are drawn. All CALS CGMs conform to ISO/IEC 8632. It is important to note that CALS applications that use CGM conform to the MIL-D-28003 Application Profile, and not simply to the CGM standard.

1.1.5.1  Typical Applications

MIL-D-28003A is intended for use in computer graphics applications in the following situations:

FIPS 128 in conjunction with MIL-D-28003 should be used when the representation of graphical information in digital form is to be used in technical illustrations and publications, and when the use of a general-purpose, graphical interchange mechanism is required. The current version of FIPS 128 is FIPS 128-1, May 11, 1993. This FIPS adopts the redesignated version of the CGM standard known as ANSI/ISO 8632.1-4:1992.

ISO/IEC 8632 is the recommended standard to:

1.1.5.2  Architecture

The CGM standard (ISO 8632-1987) (FIPS PUB 128), "Computer Graphics - Metafile for the Storage and Transfer of Picture Description Information" is composed of 4 parts. MIL-D-28003A utilizes Part 1 and Part 3 of the standard's four-part architecture.

1.1.5.3  Status and Planned Extensions

Work is in progress to produce several International Standardization Profiles (ISPs) for CGM. The profiles will be based on ISO 8632:1992 and Amendment 1 which specifies profile rules and a model profile. It is proposed that there will be three profiles based on current profiles (including MIL-D-28003 the CALS CGM AP), which range from basic scientific and technical graphics to advanced presentation and visualization.

The Industry Steering Group/Drawing and Graphics Committee is proposing an Amendment 2 to MIL-D-28003A that will include metafile descriptions, clarification of restricted text, order of precedence, and rules for profiles. The overall intent of this amendment is to permit the majority of CGM software procured under MIL-D-28003 to meet the requirements of MIL-D-28003A.

ISO/IEC 8632 CGM:1992 Amendment 1 defines rules for writing profiles. This amendment includes rules for profiles, a model profile, and conformance requirements. ISO/IEC 8632 CGM:1992 Amendment 1 has been approved and will be published in late 1994. It contains rules information that will be used to revise the CALS Application Profile (MIL-D-28003) for CGM. The "Model Profile" of Amendment 1 is similar to MIL-D-28003A. The revision of 28003A, probably MIL-D-28003B, would be upward compatible with 28003A.

ISO/IEC 8632 CGM:1992 Amendment 2 Application Structuring contains structure and intelligent graphics information. Amendment 2 has been given a Draft International Profile status and is being balloted over a six-month period. This amendment contains only change pages; therefore, one should have the full standard in order to use and understand the amendment.

The latest, 1992, edition of the international standard for CGM provides critical capabilities for the CALS environment, which include:

The purpose of this edition of the standard is to extend CGM to fulfill requirements of engineering drawings, the preparation of graphic arts quality presentation materials, cartography, and technical publishing. To a large degree, this work was directly in response to CALS requirements.

Of particular interest to the CALS environment is the work under way within the CGM standards organization (X3H3) to provide "intelligent graphics." "Intelligent Graphics" is defined as adding information to graphics that is not graphical information, but that attaches application intelligence or semantics to the graphics. Requirements are associations of comments with graphics elements, associations of comments with element groups (hierarchical), native format editing, version control, and text-to-graphics links. This requirement was originally introduced by the "electronic Review" work of the CALS ISG Electronic Publications Committee, where SGML-tagged documents are used to provide a commenting capability. The CGM additions will allow for SGML tags to be attached to objects within the CGM file.

Possible future extensions to the international standard that could be of considerable interest to CALS include the formulation of an object-structured grammar. This has been requested from, and will be of major use to, CGM users in commercial aviation (intelligent graphics); CALS electronic review (review comments in graphics and stronger links to text); and hypermedia (smart objects in graphics databases).

1.1.5.4  Advantages

The CGM contains device-independent, digitally-encoded vector and raster graphics data. CGM files are easily transferred and displayed on a wide variety of hardcopy devices (dot-matrix, ink-jet, electrostatic, and laser printers, pen plotters, and film cameras). CGM files can also be easily previewed on an extensive range of softcopy terminals. In comparison to Raster, CGM is easily modifiable, generally of much smaller size, and not dependent upon resolution of the output device. In comparison to IGES (2D data), CGM is faster to interpret and display, and again more compact. The selection of which of the CALS graphic standards (raster, IGES, or CGM) that best fits the application, should be the result of the thorough examination of the processes involved in the application.

1.1.5.5  Implementation Issues

Although MIL-D-28003 has been available for some time, vendors have not fully implemented it, thereby causing confusion with users. Vendors who state they are "CGM-conforming" may not be MIL-D-28003 conforming. Some vendors actually have a problem with importing and exporting the same image; if a CGM file is imported and then immediately exported, it is changed. In other cases, CGM generators provide functionally satisfactory CGMs, but fail to provide the required identification elements.

Standards testing, conformance testing, and interoperability testing are essential steps towards achieving successful interoperability. Standards testing examines the design, construction, and details of the standard, and tests to see if it is correct and well-defined. Conformance testing is a way of determining if a CGM product or metafile correctly implements the CGM standard and MIL-D-28003A the CALS CGM AP. Interoperability testing demonstrates that a given pair of systems (or products) can interoperate. In other words, standards testing assures that the standard is well-defined; conformance testing ensures that the product is "built correctly"; and interoperability testing ensures that there are no hidden problems in system interfaces resulting from laxity in defining the specification.

Standards testing and a small part of interoperability testing have been assigned to the CTN. The CTN tests and evaluates the effectiveness of the CALS standards and specifications. CTN also tests data interchange files for compliance with the CALS specifications. This is, in a small sense, a kind of interoperability test of the standard, but it is in no way a full interoperability test. No organization is performing full interoperability testing of MIL-D-28003 in the sense of testing interoperability between pairs of systems. Conformance testing for MIL-D-28003, on the other hand, has been formally assigned to the NIST.

NIST offers three CGM Test services; metafile testing, generator testing, and interpreter testing. The purpose of the test services is to determine the degree to which the metafile, generator, or interpreter conforms to FIPS 128-1 and MIL-D-28003A. A certificate of validation is issued for metafiles, generator implementations, or interpreter implementations that have been tested and are in compliance with FIPS 128-1 and MIL-D-28003A. Both the generator and interpreter test suites are available as part of the validation service or may be purchased separately. NIST provides the validation test service on a cost-reimbursable basis.

Metafiles should be tested when a specific picture or set of illustrations is being acquired. Conformance of a metafile does not necessarily imply conformance of the generator or interpreter. Generators should be tested to ensure that only correct metafiles are proposed. If the generator has been certified, then it is not necessary to individually test metafiles created by the generator product. Interpreters should be tested to ensure that any conforming CGM can be read and produce the correct picture.

The CTN CGM Test Packet consists of several computer graphics metafiles together with procedures for conducting a CGM interpreter test and for evaluating the resultant images. The Air Force CALS Test Network Test Bed at Lawrence Livermore National Laboratory has been testing Version 1 CGM interchange for several years, but has only recently acquired a CGM generator library that will permit the construction of Version 2 and Version 3 CGMs.

The Application Profile provides common usage across a user community, in this case the community CALS and CALS-related organizations. An advantage of having profiles is that profiles provide a way of standardizing and publicly specifying the options that are supported and how they have been implemented. Another advantage of Application Profiles is that profiles address implementation conformance requirements on generators and interpreters. Without a profile you cannot test the generator or interpreter, which is a step toward assuring successful interchange.

1.2  Summary

This report will provide the results a survey of existing COTS, GOTS and public domain test and validation tools for the above stated standards. Criteria for the qualification of the tools have been established with underlying principles governing reusable software assets. The tools will be analyzed and evaluated against domain specific and general reuse criteria to arrive at a comprehensive set that will be present in a repository of CALS test and validation tools. This report gives a complete view of the tools, their descriptions, their functionality, the availability, and the results of the evaluation against the common and domain-specific reuse criteria established.

Testing tools are a critical component of the CALS process, because they ensure that only those components that meet the CALS guidelines are able to enter the system. This directly supports the implementation of CALS and attainment of key CALS needs. The ability to quickly confirm or deny the compliance and usability of deliverables in digital data format drives directly at the worth of the tool. It also establishes the credibility of the CALS test and validation tools repository. Considering the potential volume of digitized data and reuse of such data by the CALS community, it is imperative that vendor and Government-supplied digital data conform to applicable standards. This conformity must be obtained in language, graphics, and applicable type-of-weapon defense platforms, as well as in the CALS 28000 series specifications and MIL-STD-1840.

In short, the CALS test and validation tools repository must be accessible for the CALS community to conform and validate vendor deliverables. Those deliverables that fail to meet compliance standards will not be permitted to enter the proposed IWSDB. Additionally, both the Government and the user community must be assured that the test results using the tool or tools are valid and accurate.

As a preliminary exercise in conducting our survey of existing CALS test and validation tools, the work performed under the CTN and the NIST was reviewed. The CTN is a DoD-sponsored confederation of voluntary participants from industry and Government. The primary objective of the CTN is to evaluate the effectiveness of the CALS standards (specifications) for technical information (data) exchange. The CTN is utilized to demonstrate the technical capabilities and operational characteristics of each emerging standard in conjunction with software products (COTS) that claim to support and conform to each standard.

Because standards for technical data interchange are one of the key enablers of CALS objectives, one must address the meaning of "conforming" to the standard. Products marketed that claim to conform to the CALS Standard or to be CALS compliant should be subject to a formal testing scenario that would assure the marketplace and public and private sectors that the standard is in fact being met. Companies offer COTS products that claim to perform that validation. Commercial/public test tools are programs that are used to test data formats versus a known standard or to execute test suites to validate that the CALS COTS products are producing data that conforms to the standard.

2.0  OVERVIEW OF TESTING TOOLS

      

In the following sections, we will look into the test and validation tools that have been identified from our survey. In essence, the tools considered will be tools that ensure the conformance to the respective standard. Analysis regarding complimentary software and ancillary tools that do not perform conformance checks for the standard will be minimal, as they do not provide the necessary function of ensuring conformance to the standard that is necessary for an entity to enter an Integrated Weapon Systems DataBase. These tools, nonetheless, have a function in aiding the tools qualification team responsible for the maintenance of the weapon systems database. They provide a means to cross check the results of and ensure the "degree of confidence" in entering the asset into the database system.

2.1  MIL-STD-1840B Testing

MIL-STD-1840B is the "parent" CALS standard. It provides the rules for organizing files of digital data that conform to the respective 28000 series standard. MIL-STD-1840B specifies CALS delivery, format, and organization including header file requirements, data file types, etc. This standard is applied wherever information is exchanged. There are no restrictions to its application. It can be used for the exchange of American Standard Code for Information Interchange (ASCII) text files, product definition data files, raster image files, or vector graphics files. Table 2.1-1 represents a list of identified tools used for checking conformance to MIL-STD-1840B.

TapeTool, available free of charge from CTN, is the most widely used tool for the validation of data that is exchanged according to the MIL-STD-1840 standard. CTN claims, based on a survey of the CALS community, that it is impossible to validate MIL-STD-1840 data properly with any other tool. In October 1993, Version 1.2.10 of TapeTool was released. With this revision of the tool, more data that is valid is passing the validation process. Before this revision, some data that was valid according to MIL-STD-1840 was being rejected. TapeTool v1.2.10 complies with MIL-STD-1840A. A beta version of TapeTool that complies with MIL-STD-1840B is now available through CTN. The common criteria and domain-specific evaluation for TapeTool Version 1.2.10 can be reviewed in Appendix A under 1840B evaluations.

This tool is for handling 9-track tape data in accordance with the MIL-STD-1840 standard. This tool could not be evaluated against our reuse criteria because an evaluation copy was never received.

This version of TapeTool was specifically written for Texas Instruments and is not as complete as the CTN TapeTool. This version was found to have software bugs and hence is not recommended for use. It does not provide the functionality and quality of testing that is provided by the CTN TapeTool. Please note that the evaluation for the TI TapeTool is encompassed by the evaluations of the CTN TapeTool.

2.2  IGES Testing Tools

MIL-D-28000A This specification defines the CALS compliant subsets of the IGES for the interchange of vector graphics. IGES testing is being used to improve the accuracy of data interchange. Errors in the data exchange can require extensive re-work by the CAD system user at the receiving system to portions of the CAD file to correct the translation errors. By identifying those errors in the test cycle, the sender information can correct the errors or identify alternate path. Organizations that extensively use data exchange can experience productivity gains by eliminating much of this re-work step. The two most common types of testing are loop testing and end-to-end testing. These are covered in detail in the "MIL-D-28000 Usage Guide" published by the Carderock Division, Naval Surface Warfare Center (formerly David Taylor Research Center) for the CALS Test Network, Wright-Patterson Air Force Base, Dayton, Ohio, on September 23,1993.

In our evaluation of the tools against the reuse criteria, we felt it necessary to evaluate all the tools from the survey with the common qualification criteria and restrict the domain specific reuse criteria to the Parsers/Verifiers in the IGES domain. This action is justified because the criteria established for domain specific reuse pertained to the specific evaluation of key conformance issues only in the domain. The specific qualification criteria can be found in the Final Criteria and Test Procedures Report.

To test IGES data for CALS conformance, a Parser/Verifier is used. In some cases, the file must be edited or reworked before using a Parser/Verifier. Therefore, tools for IGES testing include both Parser/Verifiers and Viewers. The following was divided in such classes. A list of MIL-D-28000 test and validation tools is shown in Table 2.2-1.

2.2.1  Tools (Parser/Verifiers)

IGES Data Analysis (IDA)

IDA offers a magnitude of software tools that claim to be CALS compliant. IDA's IGES Tool kit is composed of three intuitive and easy to use products for the identification, repair, and flavoring of IGES files. IGESVIEW is a graphic viewer for the display and manipulation of IGES files in their graphical form. The IGES Parser/Verifier is a detailed analysis utility for checking conformance to the IGES specification and to the CALS subsets. IGESXpert is a powerful IGES file editor for the examination, modification, and repair of IGES files. These products can be purchased separately or as an integrated package.

The IGES Tool kit offers a complete solution for testing and debugging high-end engineering applications. IDA is one of the companies that developed a Parser/Verifier that is now considered the industry standard for checking IGES file's conformance to the IGES specification. IGES Parser/Verifier performs a rigorous syntax, structure, and semantics check on an IGES file, producing several detailed file statistic, warning, and error message reports. The software performs two types of analysis. The first type checks the syntax and structure of the IGES file, while the second type checks the structure and semantics of the file. A CALS conformance option can be purchased for the Parser/Verifier, providing additional checking to verify an IGES file's conformance to the CALS IGES subset. Files can be checked against Class I, II, III, or IV as defined in MIL-D-28000A. CALS conformance checking is performed in conjunction with checking all versions of the IGES standard. Any errors are reported with the same detailed references to the original file.

The common criteria and domain specific evaluation for IDA's Parser/Verifier can be reviewed in Appendix A under IGES evaluations.

2.2.2  Viewers/Translators

Arbor Text, Inc.

Arbor Text, Inc. developed an advanced electronic publishing system designed to meet the publishing requirements of all phases of the CALS initiative. IGES2draw is an IGES viewer/translator that converts a Multi-drawing IGES file into Arbor Text IslandDraw format. Draw2IGES does just the opposite. It will convert Arbor Text IslandDraw format to IGES version 3 or 4.

Arbor Text will operate on a large variety of platforms such as SUN Sparc, DEC Alpha, IMB Risc, and just recently they released an MS Windows version.

The common criteria evaluation for Arbor Text, INC. IGES2draw/Draw2IGES can be reviewed in Appendix A under IGES evaluations.

Autodesks AutoCAD version 12 supports IGES translator version 5.1. The IGES translator was developed to run with version 12 on a 386/486 IBM compatible or a SUN SPARCstation running Solaris 2.1 O/S.

The AutoCAD IGES Translator compiles with the latest release of IGES, Version 5.1. It imports IGES files that comply with IGES 5.1 and earlier versions, and exports IGES 5.1-compliant files. The translator also imports files that comply with MIL-D-28000 and enables the export of MIL-D-28000 files for CALS Classes I, II, and IV.

Autodesk participates in groups developing data exchange standards, such as the ISO. Autodesk is also a member of the United States Technical Advisory Group, which represents technical positions of the U.S. Government and industry to the ISO; and of the IGES/PDES Organization-the U.S. counterpart to ISO-composed of Government representatives, CAD vendors, and CAD customers. Autodesk is a member of the CALS Test Network and United States Product Data Exchange Association, and participates internationally in PROSTEP (Germany), CADDETC (U.K.), and the Standards Association of Australia.

Carberrys CADLeaf Plus 4.0 utility is a simple-to-use UNIX based viewer, redliner, and translator. CADLeaf Plus is a family of software products designed for engineering and publishing users. Through a single integrated interface, several stand-alone modules are combined to provide the ability to view most any graphics file format, convert the file to any other format, and add notes/corrections (mark-up) to the file. Files supported by CADLeaf Plus include all popular raster and vector-based CAD formats. This includes most of the CALS compliant formats, IGES, Tag Image File Format (TIFF), and CGM.

The Viewer will allow users to view, pan, and zoom IGES and other files. A powerful feature of the Viewer is the ability to convert popular CAD formats to raster formats for use in many desktop publishing packages or vice versa.

The redliner allows work groups to view and comment on drawings as they would through a normal documentation cycle. In addition to the functionality of the viewer, the redliner adds the ability to make markups in the form of text, boxes, lines, circles, polygons and freehand drawings to the image file. These can be added by an unlimited number of reviewers, each with his own assigned color. Files with markups can be translated into different file formats for use in various CAD packages.

The Batch Translator will allow the user to batch process the conversion of graphics files from one format to another. This is useful when many files need to be converted because now one can select all the files and leave the CADleaf Batch program to do all work unattended.

If needed, an additional product can be purchased which allow conversion from raster (including CALS raster) format graphics files to vector format files (such as IGES). The raster to vector option uses very complex algorithms to determine the translation of a raster image components into vector circles, lines, and text.

The common criteria evaluation for Carberrys CADLeaf Plus 4.0 utility can be reviewed in Appendix A under IGES evaluations.

IGESView and CALSView are somewhat similar products. CALSView is a powerful tool used for viewing not only IGES data, but other 28000 series standard files. IGESView can be bundled with the IGES tool kit, whereas CALSView is a separate product. Both products provide viewing, redlining markup, and file conversion. However, CALSView is a little more extensive in document control, data delivery processes, and geometric measurement.

IGESXpert is also part of the IGES tool kit. IGESXpert allows users to quickly and easily browse and edit IGES files. Using the OSF/Motif graphical user interface, the user can manipulate data by simply pointing and clicking with a mouse. Its mass editing capability allows IGES files to be edited and processed quickly when the same known problems are encountered on a regular basis. In addition, entities can be quickly and easily selected for editing.

The common criteria evaluation for IDA's IGESView and CALSView can be reviewed in Appendix A under IGES evaluations.

PrePARE and PreVIEW are two products developed by Rosetta Technology. PrePARE accepts MIL-D-28000 Class I, II, II IGES data, MIL-R-28002 raster format data, as well as other industry standard CAD, such as raster and plot data formats, including DXF, TIFF, CCITTG4, RLC, HPGL, Calcomp 960. The software processes these file formats into PreView viewable file formats. PrePARE can also be used to convert vector data such as CAD or plot file formats into MIL-R-28002 compliant data format.

PrePARE can be run on one central workstation or PC to process files for an entire network of PreVIEW users. PrePARE is currently supported on a magnitude of hardware platforms.

PrePARE is currently installed throughout the CALS Test Network. It is being used to verify third party CAD and Raster Scan vendors compliance with MIL-STD-1840A, MIL-D-28000, and MIL-R-28002 standards.

PreVIEW displays MIL-R-28002, and other CAD, raster and plot file format files processed via its companion product PrePARE. PreVIEW provides graphical access to these files, enabling them to be displayed on personal computers and workstations for review and annotation.

PreVIEW provides intelligent viewing capabilities, including new view generation and accurate dimensional measurement verification of MIL-D-28000 files, as-well-as visual measurement of non-CAD derived files such as MIL-R-28002 and plot file formats. PreVIEW provides end-users with interactive communications as an efficient way for reviewers of design data to send written and graphic comments back to the originating design group, thereby facilitating concurrent engineering.

PreVIEW does not allow modification of the original database insuring that the original intent of the data is protected at all times. The product provides for graphical drawing comparison between multiple revisions of the same data file, which provides end-users with an accurate, graphical overview of drawing changes between revisions. PreVIEW files can be output to MIL-D-28003, HPGL and Postscript formats. PreVIEW, much like its counterpart PrePARE, supports a large variety of hardware platforms.

This information was obtained from TRW's CALS COT's handbook dated 1991. No common qualification criteria has been defined as no response from vendor for evaluation copy or literature on this product was received.

2.3  SGML Testing Tools

MIL-M-28001B defines the CALS requirements for automated, page-oriented, ASCII-text-based documents. The work performed by the NIST and CTN formed the basis for this section of the survey. A list of the MIL-M-28001 test and validation tools is shown in Table 2.3-1. The main type of tools considered are SGML parsers. The function of an SGML parser is to recognize and validate the structure and syntax of SGML documents. The purpose of the validation function is to verify that a document's content is organized and stored in conformance with the document's DTD. As mentioned earlier, the survey looks more closely at tools that validate and conform data to the respective standard and not at tools that perform other major requirements to the SGML domain like document authoring, conversion tools, editors, etc.

NIST is in the process of establishing a validation service for SGML parsers. The testing methodologies are based on ANSI X3.190-1992, Text and Office Systems - Conformance Testing for Standard Generalized Markup Language Systems. NIST is responsible for establishing conformance testing programs for Federal Information Processing Standards (FIPS 152). NIST will specify the necessary conformance test specifications, test methods including test suites, test tools and technical procedures, validation procedures and testing laboratories for testing product compliance. The SGML test suite is a package of over one thousand six hundred test scripts. For a parser to conform to FIPS 152, it must pass all of the test scripts. The test suite contains a carefully chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard. It must be noted that these test scripts encompass all the different entities and requirements for the CALS MIL-D-28001 standard.

In light of the extensive test suite in the parser validation service being provided by the National Institute of Standards and Technology, alignment with NIST is recommended. Results of the parser validations can be used for the domain specific evaluation of SGML test and validation tools. We strongly recommend that this approach be followed for the SGML domain because of the extensiveness and detail specified in the SGML standard. Each of the test scripts in the test suite encompasses all aspects of the FIPS 152. For these reasons, the SGML tools have been validated against the common reuse criteria alone. The results of this evaluation are provided in the appendix under SGML Tools.

SGMLS Version 1.0 is a public domain SGML parser based on the ARCSGML parser materials written by Charles F. Goldfarb (author of the highly respected SGML Handbook used very extensively in the SGML community) conforming to International Standard ISO 8879 - Standard Generalized Markup Language. SGMLS has error reporting capability, shell mode, and some ability to batch files. SGMLS is packaged with both source code and binaries. It can be run on either a UNIX or DOS based platform.

SGMLS parses and validates the SGML document entity in filename and prints on the standard output a simple ASCII representation of its Element Structure Information Set. (This is the information set which a structure-controlled conforming SGML application should act upon.) Note that the document entity may be spread amongst several files; for example, the SGML declaration, document type declaration, and document instance set could each be in a separate file.

The following are just some of the features available with SGMLS:

This product was created specifically for a UNIX based platform. It is public domain. This parser is popular in the European SGML community. The evaluation of this parser is in the appendix under SGML Tools.

ARCSGML is a tool kit for use in developing conforming SGML parsers, systems, and applications. It was originally written to validate the 1983 working draft of the SGML standard, and was subsequently revamped to track the standard through its final phases of development, culminating in the amendment.

A parser built with ARCSGML is employed as a co-routine with an application, called a "Text Processor (TP)" in the code comments. All memory management, I/O, and error handling occur in separate modules that can be revised to suit individual environments without affecting the core parser. Message texts are segregated to facilitate translation to languages other than English.

The application can either interface directly to parser control blocks or be isolated from them by an Application Programming Interface (API). APIs for both C and REXX are included. (REXX is an easy-to-use interpreted language with powerful parsing and string handling functions that are well-suited for converting SGML documents to other formats -- such as word processor files.)

ARCSGML was developed entirely on a succession of IBM PC computers running PC-DOS, but has proven to be portable to other hardware and software platforms. Preparation for the SGMLUG distribution of ARCSGML (ARCSGML 1.0) was done with Borland C++ 2.0 on DOS 4.01 (but only the C compiler of the integrated development environment was used, not C++). Microsoft C 5.1 was used on previous versions (command line compiler only).

The code includes internal trace routines for debugging. These can be excluded from production versions by defining "FINAL" and omitting TRACESET from the build. The distributed executable markup validator (VM2.EXE) excludes the trace code.

Although programs derived from ARCSGML have been tested extensively with modern SGML test suites, the development of ARCSGML predated the establishment of test suite guidelines.

Exoterica's XGML parser is now being bundled with the Omnimark specialized programming language for analyzing, modeling, and processing structured text information. The XGML parser provides uniquely powerful fault tolerance and error recovery features. It supports translations based on simple and complex pattern analysis and context sensitive parsing. The parser's transparent integration with Omnimark makes it difficult to evaluate the parser alone, even though it gives the Omnimark tool unique development abilities. The Omnimark program can be completely aware of the SGML implications of the current processing ensuring SGML conformance. The parser is always aware of the current state of the document structure and can provide feedback to the programmer on context and surrounding elements. In a recent development, Lotus Development Corporation's Word Processing Division has licensed Exoterica's SGML parser for use in their products. This parser has been singled out for evaluation against common reuse criteria, even though it is combined with an SGML application, because the parser existed as a stand alone piece of software previously and was used extensively by the CALS Test Network for their quick short test reports on SGML data.

MarkMinder and Intellitag are software that have in-built parsers that validate SGML files against a DTD. These two software packages are not standalone parsers. MarkMinder uses the SGMLS parser, and the Intellitag uses its own. These appear in the list of all SGML tools in Appendix B.

Although this evaluation of SGML Tools restricts itself to the parsers alone, it must be noted that numerous Commercial Off the Shelf SGML tools entertain different audiences of customers. Although these tools are for publishing, authoring, and other SGML relevant functions, they usually contain some sort of parser that validates SGML files on-the-fly for the tools. In Appendix B, a table containing all the available tools with different SGML functions is provided as a reference.

2.4  Raster Validation and Testing

Two formats exist for the CALS raster graphics standard. These two formats are defined as Type I, which is for untiled raster, and Type II, which can be for untiled or tiled raster. These two types create two categories for CALS raster validation. The tiling in Type II divides the image up into smaller portions to reduce the amount of digital data needed to convey the image. Validation of Type II data requires a much more sophisticated tool than Type I due to its open system format as specified by the Office Document Architecture Raster Document Application Profile (ODA Raster DAP). ODA data is encoded using Abstract Syntax Notation One (ASN.1) which is defined in IS 8824, "Information Processing Systems - Open Systems Interconnection - Specification of Abstract Syntax Notation One (ASN.1)" and IS 8825, "Information Processing Systems - Open Systems Interconnection - Specification of Basic Encoding Rules for Abstract Syntax Notation One." A test tool for Type II raster data must be able to parse the Office Document Interchange Format (ODIF) data stream as well as test all features of CCITT T.6 Group 4 encoding specified by FIPS PUB 150 and MIL-R-28002B. A Type I test tool needs to validate only data encoded according to FIPS PUB 150 and MIL-R-28002B.

2.4.1  Type I Raster Validation Tools

ValidG4

ValidG4 is a CCITT Group 4 validator for CALS raster files. It is public domain software and is available through the CTN at no cost. It can be used on UNIX, DOS, and microVAX platforms. Its source code is available. No documentation is available for ValidG4; rather, it is limited to a single screen of help information that can be called upon at execution. ValidG4 is run through a command-line interface where the file to be analyzed is named, and also switches can be set to augment the analyzing process. It is limited in its analytical features. Also, its error messages provide little information pertaining to the error if the file is rejected.

DecompG4 is a CCITT Group 4 decompresser that can also be used to validate as well as analyze the CALS raster files. It takes a CALS raster file and decompresses it into a plain bitmap. DecompG4 is public domain software and is available through the CTN at no cost. It can be used on UNIX and DOS platforms. Its source code is available. A rough draft of documentation has been written. The four-page documentation provides a good explanation of how to use DecompG4 and how to interpret its output. The output from DecompG4's validation features can vary from a simple rejection/confirmation to a detailed bit-by-bit analysis using the reference elements a0, a1, a2, b1, and b2 which are used in Electronic Industry Association (EIA)-538, "Facsimile Coding Schemes and Control Code Functions for Group 4 Facsimile Equipment." This kind of detailed analysis formerly required hand decoding.

2.4.2  Type II Raster Validation Tools

ODA-CT

ODA-CT (ODA-Conformance Tester) was jointly developed by Department of Communications, Canada (DOC) and the National Computing Centre Limited, United Kingdom (NCC) under the direction of Interoperability Technology Association for Information Processing, Japan (INTAP). This tool was developed to serve conformance testing needs around the world.

ODA-CT is a menu driven application. It is designed to confirm that a product claiming to conform to ODA supports the base and functional standards for ODA. It can perform two types of tests:  1) conformance of the data stream and 2) the ability of a system to receive and generate conforming data streams. This tool can generate reports on several levels for both tests.

ODATOOL is an application shell that is wrapped around COTS products developed by InterLinear Technologies, Inc. (ILT). ILT has the ODA Tool Kit that complies with ISO-8613 and ISO-8824/25 standards. This tool contains an ASN.1 compiler. Also, ILT has the TRIF Verification Utility (TVU), a testing utility for Tiled Raster Interchange Format (TRIF) files. It is designed to create, parse, read, and write CALS Type II raster data files. The tools which currently exist comply with the MIL-R-28002A standard. ILT is currently augmenting the tools to comply with MIL-R-28002B.

2.5  CGM Testing Tools

A CGM is a functional specification (semantic standard) for encoding computer graphics images for redisplay on one or several supported device types. It is composed of three encoding (syntactic) standards:  1) a character encoding for minimal size and ease of transmission; 2) a binary encoding for speed of access; and 3) a clear text encoding for human readability and easy editing. It is primarily oriented toward stroke drawn graphics, but includes raster bitmap encodings. CGM is both an American National Standard (ANSI X3.122-1986) and an ISO international standard (ISO8632/1987) which were replaced by ANSI/ISO 8632:1987 in 1991. Amendments took place to the standard to make CGM more compatible with graphics, applications, and electronic publishing standards. These amendments were incorporated into a new CGM standard, the ISO/IEC 8632:1992, that has also been adopted by ANSI as ANSI/ISO 8632:1992 and in the FIPS PUB 128. MIL-D-28003A is the proposed military specification defining the application profile for CGM.

CGM testing is also conducted at both NIST and AFCTN. A list of MIL-D-28003 test and validation tools is shown in Table 2.5-1. We will restrict our attention to the tools that check for conformance alone. Syntax and semantic errors, particularly in Binary Encoded files like the CALS Application profile, can often make the file unparseable and unusable. It is absolutely necessary that metafiles are semantically correct and syntactically complete.

Developed by Advanced Technology Center, MetaCheck/MetaCALS is a DOS-based CGM checker. It checks for conformance with ANSI/ISO 8632, FIPS PUB 128, and MIL-D-28003 standards. It accepts CGM files in three formats:  character, binary, and clear. MetaCheck/MetaCALS, available as an option, checks for conformance to the CALS CGM Application Profile. Metachek produces an expanded report documenting how well the given CGM also conforms to the requirements of the application profile. It reports on all 64 types of MIL-D-28003 compliance errors. It provides output tracing of CGM elements at five levels of detail and reports type, quantity, and location of CGM elements within the metafile. It also reports on the use of Escapes, GDPs, and Application Data Elements as well as other barriers to interchange of metafiles, 184 in all. MetaCheck/MetaCALS is an efficient product used extensively at both NIST and CTN. The domain and common criteria evaluation appear in Appendix A under CGM tools.

This tool is available free of charge from the CTN. Preliminary review does not indicate that this tool covers all the needed aspects for CGM file validation. This tool is used sparingly in the validation process but usually in combination with MetaCheck/MetaCALS. There is very little or no documentation included with ValidCGM. The domain and common criteria evaluations appear in Appendix A under CGM tools.

2.6  Test Tools Evaluation Conclusions

The test and validation tools are a means of establishing the conformance of digital data to the respective military standard. With the ultimate goal of the establishment of a set of comprehensive tools to ensure compliance of contractor digital data deliverables to the CALS formats, the tools are being reviewed under specific and general criteria. A comprehensive set of criteria applicable to the domain of each of the standards will be applied and the tools so evaluated. These tools will form the essence of the CALS test and validation tools repository. A qualifying matrix will be applied to the tools to ensure their utility and value to the test and validation tools repository. Because these tools will ensure the conformance of the data entering the shared Integrated Weapon Systems DataBase, it is most important that the tools be analyzed and reviewed carefully. The review and analysis is under progress for most of the tools.

3.0  Technical Discussion

      

This section deals with related issues pertinent to the survey conducted of the test and validation tools.

3.1  MIL-STD-1840B Testing at NIST and CTN

CTN

CTN conducts MIL-STD-1840B conformance testing and validation as part of the Quick Short Test process. CTN examines the outer markings on the package as well as the packaging itself ensuring that they conform to the standard. The tape format and content are also checked against the standard. CTN outlines its method of MIL-STD-1840 validation by dividing the process into four categories:

Any comments concerning the delivered data package and its conformance to the standard are recorded in the sections labeled above.

3.2  IGES Testing

NIST

The Computer Systems Laboratory at the NIST has recently licensed the CAD-CAM Data Exchange Technical Centre's (CADDETC) IGES testing software tools with the intent of beginning a test service for FIPS PUB 177 (IGES 4.0). NIST is planning to evaluate the testing tools before any date is set for the open validation services. NIST is currently seeking CAD/CAM systems with IGES processing software to aid in this evaluation. The NIST testing services will include the validation of IGES data files, IGES preprocessors, and IGES post processors.

CADDETC has developed conformance testing services for IGES 4.0 (IGES version 6.0 is scheduled to be released in July 1994). CADDETC has been accredited by NAMAS, a United Kingdom accreditation body. Because IGES does not require that the entire specification be supported by a processor, the service verifies that the vendors support all the parts of IGES that they claim to support. A test report is produced, which is made public only with the client's permission.

CTN tests the CALS specifications and standards with the goal of identifying needed changes and improvements to those documents. One type of test the CTN performs is an informal test that results in a QSTR. The QSTRs provide feedback to the CTN on user interpretations of the CALS standards. Participants take part voluntarily and gain a quick assessment of their implementation of the MIL-D-28000 specification. The QSTRs briefly summarize the nature of the test, the test results, and the hardware and software tested.

The QSTRs involve a four-step process and are generally turned around (results sent back to requester) within 24 hours. The four-step process involves the following:

The CTN feels that there is currently no valid tool for testing IGES. The methodology used today is to read in the IGES CAD file to an IGES viewer and rotate the file first 90 and then 180 degrees. If the file breaks apart (graphics obviously become discontinuous, for example), then the part is failed. If the file remains intact, it passes.

3.3  SGML Testing

NIST

NIST is in the process of establishing a validation service for SGML parsers. The testing methodologies are based on ANSI X3.190-1992, Text and Office Systems - Conformance Testing for Standard Generalized Markup Language Systems. NIST is responsible for establishing conformance testing programs for FIPS. NIST will specify the necessary conformance test specifications, test methods including test suites, test tools and technical procedures, validation procedures, and testing laboratories for testing product compliance.

The SGML validation service of NIST will use testing laboratories to provide a testing service to evaluate conformance of SGML parsers to FIPS standards. Results of validation from other testing laboratories also may be accepted by NIST, which will issue a registered Validation Summary Report of the product, providing all of the NIST requirements for NIST validation are met. The SGML test suite is a package of over one thousand six hundred test scripts. For a parser to conform to FIPS 152, it must pass all of the test scripts. The test suite contains a carefully chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard. It must be noted that these test scripts encompass all the different entities and requirements for the CALS MIL-D-28001 standard.

The validation testing offered by NIST does not warrant that the product will perform according to the vendor's assertions. It will certify the correct operation of the parser within the vendor product and check for conformance to FIPS 152. The basic goal of the SGML validation is to identify parsers that can be procured by federal agencies and used to develop applications that will conform to the FIPS goals of portability and interoperability. The goal is not to detect inconsistencies of the vendor software, but to test that the parser is capable of handling all the requirements of the standard.

The results of validation testing are documented in a Validation Summary Report (VSR). The VSR will contain pass/fail results in various categories and aspects of the test suite, information given by the supplier concerning identification of the product, and the operating environment under which the testing was done. This information will be supplied to NIST by the testing laboratory. A NIST certificate of validation is issued if the SGML parser demonstrates conformance to the respective FIPS based on the SGML test suite. The validation certificate identifies the parser tested, the version of the test suite, the FIPS tested, the test suites types executed, and the environment in which the parser was tested.

The Air Force CALS Test Network is to provide for testing and capability prototyping of the established CALS standards and specifications. The CTN evaluates, reviews, and demonstrates the functional interchange of digital data. Test reports are published and are distributed to interested parties. Vendors submit the SGML files to the CTN on 1840 formatted tapes and diskettes. These files are validated with the tools existing at CTN and viewed across different platforms. Any inconsistencies are reported to the vendor wanting to have the data set checked.

3.4  Raster Testing

NIST

The Computer Systems Laboratory at NIST currently has in place detailed testing and validation procedures for Type I and Type II CALS raster files. These procedures are outlined in NISTIR 4848, Raster Graphics Validation. Type I data is tested for the CCITT T.6 Group 4 compression algorithm specified by FIPS PUB 150. Type II data is tested for the CCITT T.6 Group 4 compression algorithm specified by FIPS PUB 150 as well as ODIF data stream specified by the ODA Raster DAP.

For NIST to validate raster graphics data files produced by a system, the client to be tested must first submit a completed Raster Graphics Validation form and payment to NIST. The cost of validation is $6,000. NIST then provides the client with a test suite of files consisting of uncompressed bitmap files and compressed Group 4 files. The client processes these files, converting the bit mapped files to Group 4 files and the Group 4 files to bit mapped files. The client then sends the processed files to NIST who then converts the files back to their original formats and does a bit-by-bit comparison. The CCITT T.6 Group 4 encoding algorithm is a precise method of encoding data. If the files do not match up bit-for-bit, then an error must have occurred in the encoding/decoding process. By comparing the files, the testing process evaluates the performance of the interpreter and the generator as well. NIST records the results of the testing in a Validation Summary Report. If a client's implementation passes all the tests, a certificate of validation will be issued to the client, and a Validation Summary Report will be placed in the Validated Products List.

The tools that NIST uses for its Type I validation process are ValidG4 and DecompG4. The tool of choice is DecompG4. For Type II, the tool ODA-CT is still under evaluation. As of yet, no actual Type II validation is taking place.

The Air Force CALS Test Network (AFCTN) has testing procedures in place for Type I raster data. AFCTN issues a Quick Short Test Report for data that is submitted for validation. The tools used by CTN are the same as those used by NIST (ValidG4 and DecompG4). The results of the testing are published in the Quick Short Test Report. Also included in the Quick Short Test Report are the results of displaying the raster images through a viewer. However, because a viewer cannot isolate and define where an actual error in CCITT T.6 Group 4 encoding may have occurred, it cannot be considered as a "testing tool." If an error were to occur in the encoding of a Group 4 raster file, it should be pointed out where that error occurred and how it can be corrected. The AFCTB does not use the results of the viewers for its pass/fail criteria.

3.5  CGM Testing

NIST

CGM conformance testing is a way of determining if a CGM product correctly implements the CGM standard and its associated application profile. NIST has established a validation service for testing conformance of CGM. Conformance testing is an essential step towards achieving interoperability between systems. NIST concentrates on conformance testing only and does not expand to tests of interoperability across various systems.

The conformance testing for CGM is broken up into three parts:  testing of the Metafile, Generator (program creating the metafile), and the Interpreter (program capable of reading the metafile). It is important to do the conformance test in all three of these components so that there is a good deal of integrity in standards conformance. NIST is involved in the conformance testing of all these individual parts.

The vendor requests NIST to validate the vendor's metafile. The metafiles are validated for syntax with the validation software at NIST. A validation summary report is provided to the client reporting all the testing procedures, errors, etc., and indicates data conformance or not. A certificate of validation is then issued to the vendor.

The tool developed by Advanced Technology Center, MetaCheck with the MetaCALS option, is used by NIST for validation and syntax checking of the metafile. MetaCheck is virtually a monopoly in syntax checking for CGM files. In its evaluation of Metafile checkers, NIST also came across the tool AFCTB ValidCGM. NIST feels that all aspects are not covered well by the CTN tool, and it found that MetaCheck with the CALS option was a much better analysis utility.

NIST has also developed a process for the testing of generators and is finalizing on interpreter conformance tests. The vendor must complete a questionnaire regarding the functionality of the generator to be tested. Using the questionnaire, NIST develops a vendor-specific test specification. The vendor must generate these specific test patterns on the CGMs and send them back to NIST. These CGMs are tested at NIST for metafile conformance. The next stage is for NIST to go to the vendor site and re-create the CGMs to validate the vendor's authenticity by conducting the same tests with different parameters. A validation summary report and certificate are then issued to the vendor. In case of errors, a registered report is issued; otherwise the vendor comes back again after the rectifications are made in the generator.

NIST is still to complete its evaluation procedure for Interpreter conformance testing. In essence, this will involve sending out a set of about 200 files to the vendor to generate the interpreted view. These files are specific to all the different picture and graphical elements in a CGM. In case the vendor does not support a certain element, he must indicate the same to NIST. This note will be made by NIST in the registered report. NIST is aware of the results the files produce, and these results are cross checked for accuracy. NIST also wantonly introduces known errors into a couple of the files to establish interpreter filtering capability and correctness.

A validation certificate will be issued to the vendors after each stage is cleared. The process of validation for a generator and interpreter is quite involved and takes much longer than the validation of the metafile itself. So far, no organizations have had all three of their components tested by NIST.

The conformance testing done at CTN is very different from the conformance testing carried out by NIST. The testing at the CTN constitutes testing of the metafile only. A vendor sends out a metafile generated from an application and requests CTN test the conformance of the file to the CALS 28000 series standards. The extent of conformance checking of the metafiles at NIST and CTN is exactly the same. Both of these organizations use the same utility to check the metafile. The tool is MetaCheck with the CALS option by the Advanced Technology Center. If a file has cleared the syntax checker, it is viewed across a series of different platforms and applications.

Opinions vary, though, as to the method and meaning of conformance testing between NIST and CTN. NIST believes that conformance testing is not just semantics and syntax of the metafile alone, but also establishing the performance of the generators and interpreters of these metafiles. Ideally, this is a definite prelude to interoperability testing and porting to other machines. Conformance testing according to the CTN is just testing of the actual metafile and ensures vendor data conformance and no more.

3.6  Comments on Testing Levels at NIST and CTN

We must view the functionality of testing at NIST and CTN separately. NIST is a service that vendors of software who create or read CALS compliant data must make use of. The companies manufacturing software can afford to take the time and have extensive checking performed on their software so that there are minimal bugs realized once the software is in the market. On the other hand, a vendor having to use CALS compliant data must verify that it is in fact compliant before using it. A vendor is most likely to go to the CTN for the fast turnaround time and no cost policy for processing.

NIST views the CTN as a vehicle of interoperability testing. Conformance testing must be done prior to the kind of tests conducted by the CTN. A vendor does not have to repeatedly have the data files generated by his product tested at CTN if the generator and interpreter have been tested for conformance by NIST, because all attributes of the standard are checked for in the NIST validation procedure.

Although the NIST view is to exhaustively test all three components of the file cycle, there are some advantages in the views expressed by CTN. The turnaround time is drastically less at CTN. If the file does not conform to standards, the reasons are supplied in the evaluation reports. Once a file supplied conforms to the standards, it is taken through about 10 viewing software tools to check the way the file is displayed. If there is any problem in viewing the file, the respective vendor of the interpreter is notified. This cyclic process of dialog between the vendor and CTN has led to the eradication of the errors in the software creating and reading the CALS Standard files.

APPENDIX A:  EVALUATED TOOLS

      

      

APPENDIX C:  BIBLIOGRAPHY

      

"CALS Standards Overview, Revision 2." Carderock Division, Naval Surface Warfare Center. CDNSWC/CISD (18)-94/02 May 1994. Paramax Systems Corporation.

"TRW CALS COTS Product Guide." 31 Jan. 1991. TRW Space and Defense.