| Metrics Working Group |
A Metrics Model For NATO USE
A Report to the Conference of
National Armament Directors
| October 2000 | Version 1 |
A Metrics Model For NATO Use
A report to the Conference of National Armament Directors.
The NCMB was tasked at the Spring '99 CNAD to deliver a report and recommendations on Final State Metrics, which is to say the measures and targets by which NATO could assure compliance with its strategic objectives. The report was to focus on the contribution of CALS tools and techniques to the achievement of these aims.
The NCMB subsequently tasked a Metrics Working Group (MWG), led by the US, to deliver a report to the October 2000 CNAD on its findings. The MWG, comprising members of the NCMB, the NCO and Industry, met on 3 occasions, reporting its progress to CNAD through the NCMB. In the course of its work, the MWG worked closely with other working groups, particularly the Life Cycle Working Group (LCWG), in order to co-ordinate it's findings. This summary details the major findings and recommendations contained in the detailed report.
Subject to CNAD acceptance of the report, the tasking of the MWG is considered completed. The recommendations of the report detail how the findings of the group should be taken forward in this very significant area of activity.
The MWG determined that the principal role of metrics is to establish targets and indicators of performance by which the satisfactory conduct of acquisition and support processes can be measured and improved. Such measures will fall in the three principal areas, or domains, in which CALS tools and techniques act:
· Environmental domain: the managerial, structural and cultural ability and competence of an organization to absorb, apply and review technology and processes in order to achieve strategic goals.
· Technology and Infrastructure domain: the communications, technology, security and application capabilities on which an organizations process are built.
· Process domain: the working methods, procedures and systems by which an organization's required outputs are delivered and strategic outcomes achieved. This is the domain in which most direct business benefit is delivered.
The MWG determined that these domains are interdependent, and that metrics play a vital part in ensuring that investment is focused and prioritized between these domains in order to deliver the maximum business benefit.
The MWG determined that metrics must be scalable (that is, capable of being applied at a number of organizational viewpoints), and that they must be appropriate to the viewpoint selected (that is, capable of delivering effective management information and control of the Environment, Technology or Process to which they are applied).
The LCWG have recommended a process-focused approach to Life Cycle Management, based on the standard ISO 15288. The MWG ensured that its recommendations are complementary to this standard, by developing a metrics model that maps time- or phase-based metrics to the processes outlined in the standard. The body of this report describes in detail the methods by which this is achieved, and provide illustrative metrics.
The MWG determined that metrics play a key role in assessing organizational effectiveness. Whilst metrics are key within a single organizational boundary, for any form of partnership activity, at project, national or international level, it is essential that all partners have an understanding of shared capability and maturity. Where processes, and the information upon which they act, cross organizational boundaries, the efficiency and speed of those processes are dictated by the partner with the lowest technological capability.
In order to determine where such bottlenecks exist requires primarily that an organization, or a partnership, has a sophisticated understanding of its current capabilities and performance in each of the CALS domains. The Metrics Model developed by the MWG recommends a number of complementary techniques to undertake this analysis. Metrics may then play a key role in establishing a business improvement program to address deficiencies uncovered, by helping to shape the goal (the changed characteristics of the To-Be position from current state), and by determining the measures of performance/achievement in the transition.
The MWG recognized from the outset of the task that it would be impractical to suggest definitive metrics for NATO at this time. To do so would require that NATO nations share a common definition of their processes and expected outcomes. Much work is currently underway in precisely this area, through the Life Cycle Working Group, the Life Cycle Costing Group, the NATO Standardization Agency, etc. and it is assumed that if nations can harmonize on a common Work Breakdown Structure for acquisition, for example, they could also agree definitive performance and target metrics. Until this time, the metrics suggested by the MWG in the White Paper are necessarily generic.
In spite of this caveat, the MWG has provided indicative and/or predictive metrics for each CALS domain against the developed framework, and has made recommendations on the likely resources that will be required to complete the model. The MWG concludes that the fully developed model will be of significant assistance in prioritizing investment to improve the performance of organizations at a number of levels, and in assessing the capabilities of candidate partners.
In this spirit, the aim of the MWG was to specifically answer the CNAD question: "what are the CALS final state metrics?" The MWG addressed this question by providing a definition of final state metrics as follows"
Target Final State Metric for CALS within NATO may be expressed as:
"Improved operational availability of NATO assets within an acceptable cost and time frame by delivering a fully integrated secure digital data environment shared across the equipment acquisition and support chain over the Lifecycle of the DS which is:
Although "operational Availability" is a tangible metric, It did not satisfy the MWG as the "solution". Therefore, the MWG continued to analyze the issue of metrics ultimately developing the "CALS Maturity Model".
The CALS Continuum
This model graphically demonstrates the relationship between technology and the improved processes which it enables.
Through the development of this model, the TWG realised that metrics, to be successful, must be applied in three areas: environment, technology, and process. The TWG concluded its work by detailing the application of this model across these three areas and by providing example metrics for measuring the benefits realised by applying CALS concepts.
