In general, the team felt that the use of criteria and indicators is still in the development phase and there remain considerable difficulties to their use. Some of the indicators are very well developed while others remain weak. There were also difficulties integrating indicators across disciplines. Despite our best efforts to work in an interdisciplinary manner, the team still tended to work on a sectoral basis. There is still a lack of any basic theory of sustainability that integrates across sectoral lines. Because we tended to work by sector, we also arranged our comments by sector. The following section presents specific comments on the strengths and weaknesses of the indicator sets, with comments arranged by type -- ecological, economic, management, and social.

5.1 Ecological C&I -- Strengths and Weaknesses

The ecological component of criteria and indicators has probably received the most attention in the development of indicator sets. Some good ecological indicators have been developed and are being used throughout North America. The main hindrances to further development of ecological indicators appear to be: i) limitations in theoretical understanding in how ecosystems function; and ii) how to practically measure complex ecological variables in the long term.

Monitoring and assessment of the state of an ecosystem can only be as good as the state of ecosystem science. Ecosystem science has many informing concepts that are useful in a general sense but fail to qualify as analytical concepts. An example is the concept of "ecosystem". As an informing concept it forms a foundation of ecosystem science. As an analytical concept, however, an ecosystem is hard to measure according to statistical rules such as means, variances, and replicability. There are also theoretical and practical limitations on ecosystem science because studies have traditionally taken place on very short temporal scales. Brown and Roughgarden (1990) calculated that 60% of all ecological studies have been calculated on a spatial scale of less than one square meter and 70% conducted on a time scale of less than one year. Thus, it is not surprising that ecosystem science tends to better understand small spatial scale, short-term phenomena. Indicators inherently have to deal with these limitations, and it shows in their development.

On the practical side, measuring ecological parameters can be costly and time consuming. Even tracking the changes in population for one species, if done with reasonable confidence limits, is extremely costly. There are very few agencies with the staff, expertise, and financial resources to do this for the long term. Most long-term monitoring programs fail because programs are cut during periods of budget constriction.

Of the C&I sets reviewed, the CIFOR ecological indicators tended to be geared to tropical forests. In practical terms, many of the indicators and even criteria were not applicable to temperate North American forests. An example would be to "minimize gaps in the forest". Such an indicator might be applicable to tropical systems and even mesic or wet forests in North America, but in the dry pine forests of western North America the objective is often to open the forest canopy.

A major problem in testing the indicator sets is that they often had no supporting or explanatory material. The CCFM material was generally more clear and applicable than the CIFOR material, but the theoretical rationale for indicator selection was often very brief or absent. For example, under "Extant Biomass" in the CCFM indicator set, the only indicator provided was mean annual increment. This might be a good measure of the growth rate of selected trees, but it is a very limited measure of biomass because it does not include other biota. Often it seemed that the measurement side of both CCFM and CIFOR relied too heavily on available data, even if the data were only vaguely relevant to the criterion or indicator. While the team understands the needs for practicality in data use, the indicator concept is undermined if available data are stretched to fit.

CIFOR has tried to develop a handbook for ecological indicators. Some of the indicators in this handbook had undergone significant change or redevelopment since the CIFOR phase 1 synthesis was published. This indicator handbook was not available until part way through the Boise test, and thus we were not able to test all the concepts. The team noted, however, that the handbook was not sufficiently well documented to explain the rationale for the indicator. In addition, many of the methods provided to gather data were not statistically valid or in keeping with current developments in ecology.

The GFE and Idaho C&I were at a much different scale than CIFOR and CCFM, so it was hard to consider them in the same light. Neither set was written in the criteria and indicator format. They both appear to be good applications for a specific area under a given set of assumptions.

5.1.1 C&I related to Carbon and Global Climate Change

One subset of ecological C&I are those related to carbon and global climate change. These were difficult to assess at the field management unit level because they were generally written to apply at the national level. The CCFM has a specific criterion (4.0) for forest sustainability called "Forest ecosystem contributions to global ecological cycles". This criterion targets the issue of global warming in response to rising concentrations of CO₂ in Earth’s atmosphere. There is a link between all forests, no matter how small, and the global levels of CO₂ that should be recognized; however, the above criterion does not reflect the critical aspect of forest management that will mitigate the rise in CO₂ levels. Instead, managing forests to maintain structure and function should equate to mitigating climate change. Factors of forest functions relevant to climate change include carbon stocks (above and below ground biomass and soil), sequestration rates (carbon fixation and respiration by plants and soil), and carbon sequestration capacity. Mitigating climate change requires long-term optimization of these factors rather than short-term maximization. In fact, simply maximizing carbon stocks or sequestration in the short-term will frequently conflict with good forest management and with mitigating climate change. For instance, dense stocking of trees increases the carbon stocks and the short-term rate of sequestration, but the denser trees require more water and nutrients, are at a higher risk of burning and are more susceptible to insect infestations. Any of one of these outcomes indicates an unsustainable forest practice and decreases the long-term potential for carbon sequestration in the area, thereby actually contributing to, rather than mitigating, climate change.

Recognition of the link between management of forests and global levels of CO₂ will be increasingly important as we approach deadlines to meet targets agreed to in the Kyoto Protocol and as mechanisms of emission trading are defined. Failure to recognize how management options relate to climate change could result in top-down prioritization that maximizes short-term carbon stocking over the sustainability of the forest ecosystem. It is important to ensure that indicators relevant to climate change be analyzed in context of their contribution to sustainability as a whole. To that end, the linkage between forest health and carbon cycling should be considered when criteria and indicators of forest sustainability are being developed. On the flipside, mitigating the effects of global climate change on forests will be one of the great challenges for future forest managers, and will require a better understanding of the linkage between climate, carbon, and overall sustainability.

5.2 Economic C&I -- Strengths and Weaknesses

The CIFOR and CIFOR-BAG C&I included no overt economic indicators other than some equity considerations in the CIFOR-BAG group. A CIFOR working paper by Ruitenbeek and Cartier (1998) does address C&I from an economic perspective and was used as a source document for the test, although these suggested C&I were not actually part of the test set.

The Ruitenbeek and Cartier discussion offers some strong proposals for C&I, especially from the economic perspective, although weighted to tropical forestry situations. One particularly useful theme is that it may be easier to find the negation of an indicator rather than affirmation (e.g., inequality rather than equality; unfairness rather than fairness). Also, it provides a useful stratification of issues regarding core themes of efficiency, equity, and sustainability. It also offers insights into the importance of policy related to sustainability and possible policy intervention points. Unfortunately, from the perspective of the test, the paper actually provides an alternate set of C&I, reconfigured according to a pervasive economic perspective. This made it difficult in some ways to extricate primarily economic C&I that could be added to the CIFOR set for evaluation. Further, these C&I were largely organized using general sustainability principles rather than those largely focusing on forestry.

The CCFM C&I related to economics were generally limited in scale. Nearly all C&I related to efficiency were focused on national economic parameters (e.g., productivity, capacity, non-market benefits, and contributions to GDP) and as a result were difficult to apply at the FMU level. Further, several of the other economic indicators, while measurable and at an appropriate scale, have non-obvious links to sustainability (e.g., "index of the diversity of the industrial base"). That is, the linkages may be based on second-order hypotheses (e.g., economic diversity promotes economic sustainability) which are not stated and may be untested.

The GFE and Idaho C&I do not include any economic indicators.

The team felt that C&I in the area of economics still required further development. The set tested was quite limited. They were primarily diagnostic and focused on economic structure with few dynamic aspects. The relationships to sustainability are mostly second-order. Further, there was the fundamental difficulty of trying to incorporate the sustainability of economic/social systems into the realm of forest sustainability. Is sustainable forestry a sufficient condition for economic and social sustainability? Human systems are highly dynamic and adaptable. While forest ecosystems are fundamental to meeting human needs, are they the vehicles for assuring economic and social sustainability? Finally, the economic indicators should encompass more of the dynamics of economic processes (changes in system variables over time), since it seems that sustainability is inherently a dynamic concept.

It seems that the principles of sustainability applied in the test -- Ecological integrity is maintained, Yield and quality of forest goods and services are sustainable, Society accepts responsibility for sustainability -- might better have been organized or simply evaluated by using principles from the general literature on sustainability. For example, it might have been more useful to have categorized the C&I according to the following principles:

• Inter-generational equity: Requires that the needs of the present be met without compromising the ability of future generations to meet their own needs. This would encompass concerns about long-term productivity, maintenance of ecological integrity, and making investments in natural capital.

• Intra-generational equity: All individuals of the present generation have an equal right to benefit from the use of resources. This could include many of the C&I dealing with system aspects such as ecological structure and function, income distribution and employment levels, access issues, and participation processes.

• Precautionary management: Where there are threats of serious or irreversible damage and a lack of full scientific certainty about consequences, management should err on the side of caution. This could encompass many of the management C&I as an organizing concept: the need for plans, monitoring and evaluation processes, operating guidelines, economic policies and social fairness.

• Subsidiarity in organization: Generally stated, decision making in society should be located at the lowest appropriate level. In other words, decentralization should prevail so that decisions can be made by and for the communities and individuals most affected, with higher level or organizations being "subsidiary" to lower ones. While this also incorporates many of the management C&I, it also provides the basis for many of the economic and social C&I by focusing on the conditions of local persons and communities, the scale commensurate with the FMU.

5.3 Management C&I -- Strengths and Weaknesses

The management C&I sets that were tested focused on areas of the ecosystem that are actively managed for forestry. This becomes both a strength and weakness. Due to the long history of forest management in the U.S. and other countries, management and concepts are generally based on analysis. Forest resource data are generally readily available to support measurement or assessment of management indicators. Timber inventory data have been collected and organised for decades, dating back to 1920's for portions of the Boise study area. A wealth of timber inventory data exists for the last five decades. The richness of information about timber resources often leads to its use as surrogate measures for other resource values. The management indicators tested are reasonably good measures of "good timber management" or management aimed at the most judicious use of the available resource, providing there is allocation for co-ordinated resource management. This notion is a remnant of the multiple-use philosophy that dominated natural resource management in the 1960's, 70's and 80's.

The downfall of the existing management indicators as a measure of overall sustainability is the fact that the indicators are only aimed at acres of areas included in the forest management program, and tend to be focused only on the balanced management of trees. These measures are relevant to overall sustainability of ecosystem conditions only to the extent that other resource values can be correlated with measures of the forest resource. At the North American test site, the concepts of forest management theory were applied to approximately 25% of the total area. This is because only 25% of the test area was available for forest harvest. This is problematic for a set of indicators intended to provide insight about the whole area.

Suggestion for improvement of the management indicator set would be to develop measures or indicators that can be appropriately applied to the whole study area or an entire ecosystem. This would undoubtedly require a loosening of the definition of management to include activities such as no management, terrestrial and aquatic restoration, non-traditional forest removals, and planning for ecological functions, etc.

Additional suggestion for improvement of the management C&I were developed at the Boise workshop. These include the following:

1. More than half of the nation’s forests are owned by non-industrial private forest owners (NIPF). These lands make a significant contribution to all aspects of sustainable forestry. The Boise test was not able to incorporate evaluation of management of NIPF lands. Efforts need to be made to account for the contributions of these lands to sustainability in the future.

2. C&I need to be developed to measure the sustainable management of non-traditional forest products.

3. The existence of a management plan does not guarantee sustainability. The plan has to be implemented and be effective n achieving desired future conditions. Following the Boise workshop, efforts were made to develop and align existing indicators to meet this need. However, more work is needed to make this C&I set more robust.

5.4 Social C&I -- Strengths and Weaknesses

The social C&I tested came from two sources: CIFOR-BAG and CCFM (neither the Idaho C&I nor the GFE C&I included any social criteria or indicators). CIFOR-BAG indicators have been designed to cover a range of social issues. While these indicators provided a good guide and resource for the North American test, several comments seem pertinent to highlight the challenges with working with this set of indicators, and the differences between the contexts of past tests and the current test.

The developed nation/developing nation contrast between the North American test and previous tests resulted in some incompatibility in applying the indicators. Specifically, the CIFOR-BAG indicators were written from a context where the social systems were more firmly imbedded within the forest. This might be best described as the difference between forest-dwelling or forest dependent people and people whom live in a forested area. The connections in the North American context are generally (although not always in Mexico or in some Aboriginal communities) less tightly related, which means that some indicators must be interpreted in a different way.

In the North American context, particularly in Canada and the U.S., there is an extensive legal and constitutional structure that protects many of the rights (e.g., property and treaty rights) that are more variable in developing countries. The result is the indicators are relatively easy to monitor and assess, but they may not measure people's satisfaction with these legal/constitutional structures.

An extensive set of CIFOR-BAG methods was developed for testing in other locations. These methods were undoubtedly particularly useful where no data sources exist on the social systems. These methods are almost uniformly anthropological in origin, and are not designed to take advantage of the existing data sources in North America. These anthropological techniques may be most useful in relatively small test areas or as initial means of scoping or refining methods. We found these techniques to be less useful in a relatively large area where we wanted to be able to have generalizable results. In a context where data validity and reliability are hotly contested, techniques that acquired perspectives from a wide range of sources in a replicable fashion were more desirable. Part of our discomfort with these methods is the adaptation of the techniques to the North American context, and part is undoubtedly the difference associated with the background of the social scientists involved in the North American test.

Some specific sets of indicators, particularly the public participation indicators, are written from a very different context than exists in North America. In Canada and the U.S., federal and state/provincial planning requirements mandate or legislate public involvement. We adapted the existing CIFOR-BAG indicators specifically to help to differentiate between the style and quality of the public participation here in North America. In spite of this, there were still some substantial challenges. Canada and the U.S. have different legislative requirements for public involvement, and the U.S. Federal Advisory Committee Act (FACA) restricts some forms and levels of public involvement. Consequently, these indicators might have to be further refined for each country.

The CCFM indicators were written from the North American context, and consequently were already adapted to address some of the more pertinent topics. Specific comments regarding this set are largely focused on the extent to which CCFM indicators were developed.

CCFM indicators, while more relevant to the North American context, were poorly detailed. Groups of indicators (what we would term as packages associated with criteria) were discussed generally, but no specific information was given regarding definitions or the context (particularly any potential theoretical or applied context). Known data sources were suggested, but only occasionally were methods discussed. Finally, no means of scaling these indicators were discussed. The CCFM indicators were designed for a national-level scale. As a result, some indicators were not relevant at the study area level, or data sources and methods of data collection were not specific enough to be useful.

One particularly useful CCFM indicator set that varied from the CIFOR-BAG set included issues associated with Aboriginal people. In Canada and the U.S., Aboriginal peoples have legally and constitutionally protected rights. In Canada, the acknowledgement of the need to thoroughly address Aboriginal issues is exceptionally high, particularly as a result of recent court cases (e.g., Delgamukw and Sparrow) and the Royal Commission on Aboriginal Peoples. While neither country has done a particularly good job of addressing Aboriginal rights, concerns and values in the past, this indicator set highlighted the need to specifically focus assessments of sustainability on this issue.

Many individuals and researchers in North America are focusing on the search for the best indicators of social sustainability. The literature associated with the CCFM and CIFOR-BAG sets did not reference this work. Consequently, we expect we missed some well-researched indicators. For example, University of British Columbia researchers, notably Bill Rees, have been working on the concept of an ecological footprint that contains a number of associated indicators. This initiative, however, like many others, is not specifically focused on sustainable forest management.

Other useful sources of indicators may come from the ecosystem management literature and from a social context, particularly the social acceptability of ecosystem management (see the excellent research conducted by the USDA Forest Service (USFS GTR 1995 and 1996).