Within each domain within which CALS operates, the MWG assessed the range of currently available methods and tools that can provide an objective assessment of the penetration of CALS. The purpose of this approach was to establish an objective mechanism for `scoring' an organization against metrics within the Maturity model, and thus provide an indication of where improvement should be focused. Information about tools in use was primarily sought from nations, but industry views and products were also sought and researched.
5.1. Applying the Model - the Environment domain:
Although, on the surface, the most intangible of the 3 domains explored, organizational theory has been exhaustively studied for many decades, and has been a primary focus of business change programs. This focus has resulted in a rich toolkit to assist in business analysis. Many of the tools are industry-specific (e.g. directed at specific manufacturing or service-industry sectors, such as automobile manufacture or banking), and thus too narrowly focused for the purposes of this exercise. The MWG determined that the common feature of the tools examined which enable a more generic business analysis is a view on the organizational and cultural characteristics of successful enterprises, whether private or public sector, as measured by their performance outcomes (share price, policy implementation, etc.). In effect, they attempt to establish `Best Practice'. Whilst each tool takes a different approach to assessment, there appears to be general consensus that the most significant areas of competence are:
· Strategic goal setting and Business Planning
· Leadership and Communication
· Staff competence and their alignment to strategic and business goals
· Resource management
· Review and action mechanisms
These factors are described as the enablers of business results.
Assessment may be by external review, by self-assessment, or a combination of the two, but each model examined applies both a methodology to assessment (thus ensuring a measure of consistency), and a scoring mechanism which serves to highlight areas of significant strength or weakness in each of the competence clusters.
Such models are not prescriptive about organizational structures - they may be flattened, hierarchical, or matrix - or about team organizations, which may be traditionally structured around a single function, or multi-disciplinary, integrated teams which are focused on a project. Neither do they suggest any model of leadership, resource management or strategic planning and review; rather, the scoring mechanisms are predicated on evidence of strength or weakness in each of these areas, and their impact on results and outcomes. Most of these tools have some form of weighting system, based on empirical evidence, for each of the areas examined, and most also have some form of overall scoring regime. Whilst this may appear somewhat arbitrary, it does give a broad and useful metric, and a framework in which a more detailed analysis and comparison of results can be formed.
The MWG determined that the most significant benefit of this approach is that it allows both direct comparison between organizations with quite dissimilar functions (and between public and private organizations), and can also act as the springboard for business improvement within an organization.
The model of which many members of the MWG had direct experience was the European Foundation for Quality Management (EFQM) Business Excellence Model (BEM), which embraces the principles outlined above.
Figure 4: The EFQM Business Excellence Model (BEM)
This model is based upon a business evaluation model from the United States, the Baldridge Model, named after a US Commerce Secretary, and contains the illustrative weightings attached to each of the enablers in the model, and the consequent business results. The inclusion of this model is for illustrative purposes only, indicating an approach that is common to most models examined. A notable feature of all such models is that they can be applied at virtually any level of viewpoint, or organization boundary, and are well supported by a wide range of complementary business analysis techniques. Within the UK Defence Procurement Agency, for example, the model works in harmony with the Investors In People (IIP) standard and the Electronic Business Assessment (eBAT) tools, each of which focuses on business improvement in specific areas of the BEM Enablers.
The MWG does not at this time make a firm recommendation on any particular model or methodology for business assessment in the Environment domain. Nonetheless, the value of the approach is self-evident, and is capable of providing indicative metrics of the environmental maturity of spheres of activity in both the government and industrial sectors at any level deemed appropriate. It may be used for comparative analysis, if so desired, though its major benefit perhaps lies in its ability to provide a platform for performance improvement. The value of the approach to NATO would, of course, be considerably enhanced if a common tool and methodology were employed, thus enabling fair comparison.
Embarking on a Business Analysis should be viewed as a Change Program in its own right. Assuming that no such regime already exists within an organization, the resources required to adopt, implement and act upon the results of a common business assessment methodology are considerable. Many models are in the public domain, and therefore do not attract licensing or other ancillary costs. However, the staff and managerial resources and commitment involved may be substantial, and the assessment process itself is expensive, whether conducted as a self-assessment or externally assessed. Evidence provided to the MWG suggests that the initial costs of communicating the purpose and method of assessment alone may equal perhaps a half working day for each member of the organization to be assessed. Establishing the mechanisms to collect and collate evidence for the assessment are a further overhead and the assessment itself may well be accompanied by an extensive interview program. The costs of acting upon weaknesses exposed during the assessment will, of course, relate to the nature and severity of those weaknesses, but will invariably require some form of dedicated project-based activity. Finally, the entire exercise will require substantial and visible commitment of senior staff effort.
However, evidence provided to the MWG also suggests that the costs are very much at the front-end - that is, after the set-up costs, and once the practice of review becomes embedded in the culture of an organization, the ongoing costs for second and subsequent assessments are substantially reduced, and should, in theory, become negligible, as the evidence required to perform an assessment becomes a natural by-product of an organizations business practices.
It is impossible to specify direct business benefits which arise from employing a Business assessment tool, as these will be derived from the improvement program(s) that arise from the assessment itself. Nonetheless, the evidence provided to the MWG suggests that there are very considerable collateral benefits to be gained from simply adopting such a managerial approach, in terms of improved communication throughout organizations, and a renewed focus on the significance of relating and aligning the outputs and skills of an organizations workforce to its strategic goals and policies. This is unsurprising, given all such models are based on the characteristics of demonstrably successful organizations. It should also be pointed out that even if an assessment were to reveal that an organization is a world-class performer, with strengths in every field examined, this would in itself be of value in pinpointing models of Best Practice.
Whilst the CALS community lays no claims to ownership of the management and organizational theory and practice on which these models are based, the importance of this domain to successful implementation of CALS tools and techniques is that of an organizations ability and competence to transform the data within an organization into information, and to competently manage the information itself in pursuit of its strategic goals. Until an organization achieves this level of competence, it will be unable to fully realize any investment it makes in either technology or process improvement.
· You can not guess what your environment is, YOU HAVE to KNOW
· There are a number of well tested methods available to help
· However, you must use them to determine what is to be improved and to measure improvement
· The tool will also help identify best practices
· Use of these tools will help determine investment opportunities
· CNAD accept that these tools are useful and will add value
· Selection of one tool would be most practical
· Reduces learning curve
· Standardizes measurement within NATO
· Most easily identifies best practices
· CNAD to champion their use
· NATO should have a body of expertise for
· Understanding and use of these tools
· Maintaining a linkage to appropriate standards where applicable
· Establishing appropriate bench marks
· Serving as a repository of best practices
5.2. Applying the Model - the Technology & Infrastructure Domain:
The Technology & Infrastructure domain has the most direct relationship with the Maturity Model shown at Figure 2. It is also the most fundamental of the CALS domains. Without some method of capturing, storing, disseminating and re-using information in a timely manner, the processes that act upon that information are inherently inefficient, no matter how well structured the organization that employs them. It is unarguable that the basis of information capture, storage, dissemination and re-use must be digital.
There is a vast catalogue of technologies and applications available, some (particularly in the communications area) based on neutral International standards, many proprietary. Moreover, this is an almost uniquely dynamic industry, with extremely short technology cycle times, which are not aligned to customer investment cycles and decision points. These factors conspire to make the investment profile for technology and infrastructure projects extremely complex.
The MWG determined that Metrics, in this domain, can only be assessed by capability and maturity with respect to the model. Such metrics are key within a single organizational boundary, but for any form of partnership activity, at project, national or international level, it is essential that all partners have an understanding of shared capability and maturity. Put crudely, where processes, and the information upon which they act, cross organizational boundaries, the efficiency and speed of those processes are dictated by the partner with the lowest technological capability. This is the bottleneck that must be addressed.
Following the principles established for each of the domains, the MWG looked for the principle characteristics of maturity and capability. A concept that has often been used, and which complements the Maturity Model at Figure 2, is a step model of capability. This is illustrated at Figure 5.
Figure 5: The Technology Development Steps
The model illustrates that there are recognizable characteristics of the progression toward the Virtual Enterprise that is the goal of the Maturity model. These may be summarized as:
· Paper based information
· Information is digital
· Digital information is exchanged
· Digital information has structure
· Standards are used
· Fully integrated secure shared digital environment - the Virtual Enterprise
The MWG determined that this progression summary, whilst crude, provides simple and powerful `Acid-Test' metrics for this domain. There are clearly degrees of sophistication underpinning these questions - the exchange of information may be as simple as a transfer by diskette, or a dedicated, secure high-bandwidth telecommunications channel. Nonetheless, the principles are sound.
As in the other domains, the narrower the focus, the easier the task of assessment and the clearer the options available to improve any shortfalls discovered. As the focus moves out, the same Metrics apply, but will have to take account of pockets of excellence and areas of weak performance within the target viewed to form an overall view.
A particular issue to be considered will be the impact of Legacy systems on the overall result. These systems may be at a stage where it is not considered practical or cost-effective to invest in further development to improve the score on the maturity model. If these systems are self-standing, then they may be safely excluded from the scope of assessment, or noted as a limitation on the result. However, where legacy systems form a key component of shared processes between partners then very careful consideration will need to be given to the overall impact of a decision not to invest further in the system. The investment issues behind such decisions between partners are extremely complex; it is generally difficult to justify capital investment in one business area, where the business benefit of the investment may well appear in another. Such issues are therefore likely to be a key indicator of the concept of `partnership'. Care must be taken at this juncture to avoid sub-optimisation.
Whilst Security is a constraint in each of the CALS domains it most evident in the Technology domain, where the management of system security is a key design element of systems, and may result in significant escalation of costs and severe limitations in use of such systems.
The Final State Metric, which is a shared goal between governments and Industry, is predicated on the assumption that these security issues can be overcome. However, it must be recognized that the nature of the business of NATO, national Defense Ministries and Departments, and the Defense Industry itself pose extreme difficulties. It is hard to conceive of any other sphere of activity where the requirements of national security and commercial confidentiality coincide to such an extent.
Security per-se is not within the CALS domain although it must be considered in all aspects of information creation, storage and use. CALS can assist projects by pointing to products that satisfy national security requirements, and the national authorities that assess and accredit system design and implementation. However, the ownership of technical security policy in most cases lies outside of defense ministries. Moreover, the basis of security policy is, in many instances, enshrined in national constitutions and supporting legislation. A CALS viewpoint can certainly help to focus the debate and the need for change and harmonization throughout the NATO alliance in support of strategic goals.
One specific instance is the stated aim of `reducing the logistics footprint'. One of the key enablers which has already been identified in meeting this goal is the increased integration of civilian components into the operational logistics environment, principally by adopting the `factory to foxhole' concept. This would mean, in effect, that industry would subsume many of the consumer logistics functions in a stabilized theatre of operations at the earliest possible stage, thus reducing the military burden. However, this implies reliable and secure communications across which information flows could pass into industrial processes within nations. This would mean that either secure military communications would have to be extended to civilian components for the duration of an operation (requiring a substantial spare bandwidth capacity and a set of dual processes within Industry), or that industry builds its processes and information management mechanisms on a secure and deployable communications network. Either of these options suggests a significant cost and technical overhead.
There are several bodies within NATO and within nations working these issues and the CALS community would wish to assist in the debate at any and all appropriate levels.
The assessment of technological and infrastructure maturity is a key factor in deciding the ability of an organization to operate efficient processes. At an individual project level, the NATO CALS community offers products and assistance to help focus the relationship between project information management needs and the technology and infrastructure required to support those needs. This assistance includes selecting appropriate technologies, creating business cases to support investment, and employing/contracting against appropriate technical standards.
As the focus moves out to a wider level, the same CALS principles apply: however, the complexity and scale of analysis, the investment required, and the number of stakeholders involved, expand dramatically
· The Technology and Infrastructure domain represents the basic building block of military/industrial capability for through-life information management
· This domain is a key enabler of partnership activity.
· There is increasing pressure to extend this domain into the operational environment
· There is no central focus for this within NATO, although there are numerous activities looking at pieces of this issue.
· Without a central focus, there is a danger of building isolated islands of incompatible architectures
· Maturity and capability are the key metrics that should be established
· The CALS community offers significant practical assistance and tools to develop capability at the project and program level
· There is a significant change in effort and resources required when applied at the national and international level.
· National Security and commercial confidentiality are major constraints on this domain, and need further analysis. This is beyond the scope of this project.
· Support the maturity analysis process
· Initially at the project/program level
· Champion at the national and international level
· Establish a central group for Technology and Infrastructure with a focus on augment existing work efforts
5.3. Applying the Model - The Process Domain:
The Process domain represents the `cutting edge' of capability, as it is the area where capability manifests itself in tangible business and strategic outputs and outcomes. There has been considerable investment in modelling processes to enable a shared understanding of their input, output and efficiency. One such model, which is highly relevant to NATO, is ISO 15288, which has been recommended by the Life-Cycle Working Group (LCWG). This model recognises the importance of process to life cycle information management. The MWG determined that this approach is wholly consistent with its approach to the other CALS domains, and that the same principles of maturity and capability apply.
The ISO model is predicated on the principle that there are key processes running concurrently throughout the life cycle of all Defense systems, and that the activities and outcomes of these processes can be described from the key stakeholder viewpoint of these processes. Whilst the MWG recognized the importance of this approach, it needed to develop the model (shown at Figure 3) to reconcile the process approach with a more traditional time or phase-based approach for describing the life cycle. This clarified both the importance and role metrics plays in planning and measuring transition from a current position to a desired outcome. The MWG also determined that, from a CALS perspective, the greatest end-user benefit was to be gained by assisting in focusing investment through time. In a resource-limited world, investment (particularly in technology) is invariably incremental, in that capability is added as it becomes essential to the efficient management or use of the defense system to which it is applied. The MWG sought a method to reconcile potentially conflicting stakeholder viewpoints and investment demands by focusing on agreed goals (or outcomes) through time. This focus will determine the combination(s) of technology and process that are most likely to achieve a successful achievement of a shared goal.
The MWG placed the concurrent processes within ISO 15288 against a generic phase model (AAP-20 -- PAPS) to determine the key metric for that phase. For example, the key metric during the Concept phase is the time needed to finalize a decision to proceed against an operational or design concept. ISO 15288 argues convincingly that some element or degree of each of the core processes that it identifies will operate in this phase, but clearly some processes will be more dominant than others, and some stakeholders will `own' more of these processes than others through time. A combination of stakeholder analysis (a ranking of interest) and objective setting (time to decision point) will determine the tools and technologies which are most likely to deliver the objective.
Figure 6 demonstrates this concept, using a simple Traffic Light system for the relative importance of process within a phase (Red being most important, Green least). This analysis indicates those technologies and tools that provide the greatest level of support to these key processes. This does not suggest that this is an exhaustive list of the tools and technologies available that are relevant to this phase, to the identified processes, or to the stakeholders. Nor does it argue that there is no business benefit to be gained from wider investment. The aim of the analysis is to prioritise investment where contention for resources exists.
Figure 6: Process/Stakeholder Analysis - Concept Stage
As a project or program moves through phases, the relative importance of process will change, and the richness of technology (and thus investment choices) available to support those processes will expand. The range of indicative metrics, or characteristics, which might be selected as shared goals, will also expand. Figure 7 illustrates how these choices are likely to change as a weapon system enters into service.
Figure 7: Process/Stakeholder Analysis - Sustainment Phase
Using data supplied by current projects, the MWG was able to establish a direct causal relationship between the deployment of technology and specific performance metrics. Figure 8 shows contractual Performance and Sustainment Metrics from the F-117 program, which monitor data held in shared government and industry systems.
Figure 8: F-117 program - Performance & Sustainment Metrics
This approach is most easily applied and proven at an individual project level, where performance metrics are specific and predictive. The principles of this approach can be applied at higher levels of viewpoint although the granularity of these metrics are likely to be lower and indicative rather than predictive. However, absent other means, this is potentially a very powerful approach to providing an investment framework.
The LCWG is currently developing and staffing the concepts of ISO 15288 to the requirements of the NATO emphasis on Through Life management of Defense system information. The MWG has developed an outline concept and model that provides metrics that are aligned to these emerging concepts. However, because of the importance of stakeholder analysis in the developing model, there has not been sufficient time to staff the approach through other interested NATO bodies. Nor has the MWG gathered a broad enough range of supporting empirical evidence from projects and industry to prove that the model is sufficiently robust in its current state of development. Nonetheless, there has been considerable interest from other NATO groups (e.g. SILCEP) in assisting development of the model and validating it with empirical evidence.
It is essential that hard data be gathered from existing projects to validate the model. During the course of its activity, the MWG issued requests on nations and industry to provide information, but the result was disappointing. The responses received revealed that further work would be necessary to develop questionnaires that would minimize the information demands on industry and projects, and that some effort would be required to provide an effective analysis of the responses. This will require some specific professional assistance, particularly in the areas of questionnaire design and statistical analysis.
There is considerable data available in the public domain that supports further development of the model, but again will require some professional assistance to analyze effectively. If properly designed, any future information calls on both nations and industry should be readily answered from existing data.
An illustrative example, with metrics, is at Annex A.
· The most direct business benefit is found in the process domain.
· The most direct link between investment and outcome is in this domain.
· The CNAD sponsored LCWG has taken the ISO 15288 as the model for defense system life cycle processes.
· The MWG have developed a complementary model that reconciles process against time providing indicative metrics and focused investment.
· The MWG attempted to validate the model using data from existing projects. This needs further work and consultation.
· Professional assistance is needed (e.g. statisticians) to make best use of publicly available information, and to minimize the impact of any further call for information on projects and from industry.
· Support the process domain model
· Agree it should be further developed within NATO
· Support the need for professional assistance in data analysis
· Champion at the national and international level
· Assign responsibility to a central group
