Frequently Asked Questions

1st ISSRR
May 2001

Call for Participation

Proceedings (PDF)

Position Papers

Workshop Photos

ACSA Home

Workshop on Information-Security-System Rating and Ranking (WISSRR)

Introduction

The Workshop on Information Security System Scoring and Ranking, sponsored by the Applied Computer Security Associates (ACSA) and The MITRE Corporation, was held on May 21-23, 2001, in Williamsburg, Virginia. The goals of the workshop were to characterize the information security measurement problem domain, identify "good practices," focus needs, and determine potential research directions.

Workshop Scope

The workshop organizers struggled with the following questions: What are we talking about? What should we call what we're talking about? With respect to the first question, the Call for Participation emphasized metrics for information technologies and products. However, the position papers addressed a broader spectrum of information security metrics, as reflected in the characterization described below.

With respect to the second question, the workshop organizers recognized that considerable controversy exists regarding the term metrics: some seek to reserve it for the results of measurements based on scientific principles, but others use it to include results of assessments based on subjective judgments. While some position papers urged reliance on a dictionary or scientific definition, others observed that broader usage has been adopted in policies and practices. As some past discussions on metrics had been totally consumed with this discussion, the expression information security (IS)* was used in the workshop agenda to avoid long discussions on terminology. The asterisk (*) was used to mean any of the following terms: metric, measure, score, rating, rank, or assessment result (although not necessarily an exhaustive list). Therefore, IS* is defined below:

An IS* is a value, selected from a partially ordered set by some assessment process, that represents an IS-related quality of some object of concern. It provides, or is used to create, a description, prediction, or comparison, with some degree of confidence.

Although participants gravitated toward use of the terms IS metric or information assurance (IA) metric, we will use IS* in these Proceedings. Figure ES-1 illustrates the workshop characterization of IS*.

Ultimately, IS*s are intended to improve understanding or support decision making related to the IS posture of an entity. Several general problems were identified:

• Metrics are often ill defined; consequently, any definition of an IS* should include a specification of the process used to construct and evaluate it.

• The definition and evaluation of IS*s frequently become distanced from the ultimate use, so that metrics become ends in themselves. That is, the consumer of the metric is lost in the definition of the metric.

• IS*s are often used or reported in contexts other than those for which they were originally intended.

As a result, many security technologists have great misgivings about the topic of IS*s. The workshop attempted to categorize these misgivings and tried to understand the rationale for them. In the process, the participants identified examples of good (and not-so-good) measurement practices and identified directions for further research. It became evident that IS*s can be characterized in terms of purpose or intended use, form, or scope. Two broad classes of uses of IS*s can be identified as follows:

• Decision support. This is the primary use of most IS*s and was the focus of the workshop. Assessments of security properties are used to aid different kinds of decision making, such as risk management, resource allocation, program planning, or selection of specific products or services.

• Mandated reporting of IS status or posture. Organizations also use and define IS*s to respond to external demands. Reporting can be mandated to determine compliance with requirements, serve as a basis for trend analysis, or justify requests for additional resources. Specific metrics can be mandated; however, usually the reporting organization identifies the need for metrics to provide a succinct reporting mechanism.

The mandated reporting of IS status or posture is a relatively structured event, and the items reported tend to be discrete values, such as number of requirements fulfilled and number of intrusions detected. The workshop participants acknowledged the importance of mandated reporting in the determination of an organization's information security posture.

The form of an IS*, that is, how it is reported, can be numeric or non-numeric. The often-attempted distinction between quantitative and qualitative IS*s frequently breaks down in practice. Numeric metrics often represent relative rankings; the numeric difference between ranked values is significant for some metrics, but not for others. The assessment process leading to non-numeric metrics (e.g., red/yellow/green) frequently involves quantitative measurements (e.g., green means zero vulnerabilities found; yellow, one to five; red, more than five). The workshop participants avoided quantitative vs. qualitative discussions.

Conclusions

Surprisingly common themes emerged from this workshop, summarized in the following conclusions:

• No single IS* will successfully quantify the assurance present in a system. Multiple measures will most certainly be needed and they will need to be refreshed frequently.

• Software and systems engineering are very much related to this problem. For example, the quality of software delivered, the architectures and designs chosen, the tools used to build systems, and the requirements specified are related to the assurance to be quantified.

• Penetration testing is in use today as a valid IS*. However, it is imperfect and to some extent nonrepeatable, but it is used in both government and commercial sectors.

• Government and commercial sectors have different agendas: the former is policy driven and the latter is profit driven. Thus, the two sectors may place different values on IS*s.

• Measuring defense in depth and breadth is a critical area that warrants further research.

• In the past, attempts to quantify and obtain a partial ordering of the security attributes of systems have not been successful to a large degree (e.g., the TCSEC and the CC).

• Processes, procedures, tools, and people all interact to produce assurance in systems. IS*s that incorporate these aspects will remain critical to successful IT system operation.

These conclusions indicate that the direct measurement of IS properties is desirable but not always possible. The assessment process should include activities for validating the indicator (e.g., by cross-checking it against other indicators). For example, an indicator of an organization's IS program might be the quality of its documented plans; under the assumption that an organization's commitment to information security will be reflected in its budget, an assessment of organizational plans could be correlated with financial metrics.

IS*s must evolve. A metric that is meaningful and useful today may be less relevant tomorrow, due to changes in technology, practice, or regulations. Organizational processes that use IS*s should include periodic reevaluation of those metrics and redefinition as needed. If metric evolution is not done deliberately, it will occur accidentally: the information that can be gathered will change with technology advances, and assessment that involves expert judgment will change as expertise increases. Care must therefore be exercised in comparing metrics values over time.

Organizational and operational IS*s have more in common with metrics from the social than the physical sciences. IS professionals should apply lessons learned from the social sciences, particularly from public health and safety risk management.

Better models of system behavior are needed to define predictive technical IS*s. In particular, better models are needed of the composition of (and dependencies among) subsystems that provide different security services.