The Evaluation Framework

Why use the TEEBAgriFood Evaluation Framework?

Most current assessments of agricultural and food systems are partial and ignore a number of important relationships that eco-agri-food systems have with our economy, society, environment, and health. Examples of partial assessments include farm level assessments of productivity on the basis of yield per hectare only or assessments of environmental efficiency that cover the agricultural production chain but focus only on water or energy use. Such assessments, while clear in scope, leave out broader issues of sustainability and equity that are fundamental considerations in assessing food systems.

Thankfully, discussion is growing around new approaches to assessing eco-agri-food systems including the use of sustainability indicator sets, the measurement and valuation of ecosystem services as inputs to food systems, and the assessment of the connections between food and population health. The perspective of the TEEBAgriFood Evaluation Framework is that these types of approaches need to be integrated in order to better inform policy decisions. Assessments that are context specific and which consider a comprehensive set of interactions, as described in the Framework, will ensure that decision making about food systems captures all material interactions between environment, economy, society, and health and covers interactions from the farm to household consumption.

What does the Framework include?

The Framework includes four elements - stocks, flows, outcomes and impacts- which capture the set of interactions (see Figure 1). The stocks of eco-agri-food systems comprise the four different “capitals” – produced capital, natural capital, human capital and social capital. These stocks underpin a variety of flows encompassing production and consumption activity, ecosystem services, purchased inputs and residual flows. The dynamics of an eco-agri-food system lead to outcomes that are reflected in the Framework as changes in the stocks of capitals, both quantitatively and qualitatively. In turn, these outcomes will have impacts on human well-being.

By providing key definitions and associated measurement concepts and boundaries, the TEEBAgriFood Evaluation Framework establishes what aspects of eco-agri-food systems may be included within a holistic evaluation.The chapter does not focus on how assessments should be undertaken, nor does it prescribe methods for assessments. The choice of methods will depend on the focus and purpose of any given assessment, the availability of data, and the scope of analysis.

What is the purpose and role of the Framework?

With these considerations in mind, the Framework identifies and characterizes allrelevant elements of our eco-agri-food systems. Of course, eco-agri-food systems are heterogeneous with significant variation in terms of types of outputs, the nature of production systems and value chains. Further, there will be different purposes and perspectives for each assessment. By way of example, while health impacts at consumption stages for corn produced for corn syrup may be material, this would not be the case for corn produced for ethanol for use in biofuel production. Thus, not every possible combination of elements covered by the Framework will be relevant and material in every assessment.

The Framework has thus been designed to provide broad categories of all interactions that may exist within a given eco-agri-food system. This provides a clear and common starting point for all assessments as they work towards identifying the elements that are most material in their context.

While all assessments will have somewhat different coverage, it is also expected that all TEEBAgriFood based assessments have the following features. They should:

1. be broad and systemic in nature,
2. reflect the contributions of all four capitals and
3. examine connections along the full value chain, including assessing the impacts of food consumption on human health.

If these three features cannot be demonstrated, then the assessment would be considered a partial assessment and not consistent with the spirit of the TEEBAgriFood project.

How can the Framework be used for an evaluation or a study?

To demonstrate how the Framework may be used, the following steps may be followed:

1. Determine the purpose of evaluation. The purpose of the evaluation exercise may differ within and across groups such as researchers, businesses, or consumer groups. A clear articulation of purpose should be used to scope an assessment.
2. Determine the entry point and spatial scale of analysis. The entry point would depend on the research interest or focus of the study. Relatedly, appropriate spatial boundaries would need to be defined – within or across regions, countries etc.
3. Determine the scope of the value chain under analysis. This requires the researchers to understand the system and bring together relevant literature and sources to support their description of the value chain – from production to consumption.
4. Determine the stocks, flows, outcomes and impacts most relevant for the purpose of the study. The relevant aspects that should considered through literature review and research are:
   1. At each and every value chain boundary, identify the flows outlined in Figure 6A.1. It is important to understand that these flows can help identify pathways through which the four capitals contribute to agri-food value chains, and how in turn agri-food value chains may impact the capital stocks. These may include waste or emissions generated along the way. This of course requires certain level of knowledge and research of the given system in question.
   2. At each and every value chain boundary, identify the social, produced, natural, and human capital related outcomes of the system (Table 1 provides some examples). This of course requires certain level of knowledge and research of the given system in question.
   3. Evaluation of these two aspects requires an understanding and mapping of the spatial scales at which these interactions are happening – ecosystem services used at the farm level may be generated beyond the farm, for example. Similarly, health outcomes of a particular food product may happen well beyond the farm, especially if there is international trade.
   4. Given these considerations, the assessment must identify the impacts that it is choosing to address and the ones it is excluding, and provide appropriate reasons.
5. Select evaluation techniques. While the first four steps provide the framing and scope of the evaluation, the next step is to choose the techniques that would help one assess and measure the interactions within a given system. For TEEBAgriFood, the focus is on assessing impacts as contributions to human well-being. Otherevaluation methodologies may include life cycle assessment and value chain analysis, and various modelling tools and techniques including partial and general equilibrium models and system dynamics.
6. Collecting data and undertaking the evaluation. Once the context and methods for evaluation are set, efforts can be made to collect data. While data availability can be an important factor in defining the scope of assessments, by completing steps 1-5 prior, the implications of lack of data can be understood and can provide motivations for identifying and filling information gaps.
7. Reporting and communicating findings. Communicating the results of the evaluation exercise should be seen as an essential part of the process. Particular note should be taken of the need to develop a range of outputs to suit different audiences including politicians and business leaders, technical experts, farmers and local communities and the media.

To support the application and implementation of the Framework and the associated discussions among stakeholders, it may be useful to use the tables and text from section 6.3 of the chapter that explain the various components of the Framework. With this in mind, the table below provides a stylised version of the Framework in the form of a checklist that can be used by researchers and decision makers to consider the relevant interactions and to ensure awareness of those aspects excluded from an assessment.

Table 1 comprises two main sections: i) stocks/outcomes (changes in capital stocks), and ii) flows. Several of these elements may be measured differently – for example, in qualitative, quantitative or monetary terms. Impacts (value addition) elements are excluded from this table since the scope of measured impacts will relate directly to the scope of capital stocks, outcomes, and flows that are included in an assessment. The methodologies for assessing impacts are presented in the TEEBAgriFood ‘Scientific and Economic Foundations’ report, Chapter 7.

It is important to note that several of these elements would require a more detailed description and measurement depending on the scope and context of the assessment being conducted. For example, depending on the assessment, water may include coverage of both surface and ground water resources. Furthermore, quality indicators of water may include several other elements such as habitat quality or nutrient profile. Finally, it is not always the case that all components receive the same type of evaluation and measurement. Thus, in using the table to assess the coverage of an assessment, it will be relevant to distinguish as to whether a component is being assessed descriptively, quantitatively or in monetary terms.

How does the Framework guide researchers, decision-makers (public or private), local communities, farmer groups and other users?

Utilising a comprehensive and universal Framework provides a common basis to compare assessments, a tool for decision-makers to understand what information is missing, and a means to identify areas of further research.

Since it includes all categories of material interactions in a given food system, the Framework can offer entry points to many people – for example, researchers focusing on social impacts of food systems, can use social capital related outcomes as a starting point, and then make linkages to the other three capitals. Similarly, decision-makers can start at the economic elements, but then identify how these may be related to other capital stocks and flows. The Framework can also help decision-makers quickly identify any blind spots in the information base used to support decision-making. In essence, no matter what the starting point or purpose, the Framework can allow researchers to contextualise their assessments within the broader set of interactions that their food system has. This not only brings transparency to their assessments, but also highlights the opportunities to link their work with other research.

The TEEBAgriFood Framework can also be a starting point for identifying the material elements of particular systems, and thus can lead to the development of guidelines on comparable assessments. For example, similar firms, in terms of size and products, in the food and beverage sector could use this Framework to identify the main impacts and dependencies of their sector’s operations. Similarly, organisations such as farmer cooperatives, consumer protection groups and local governments could elaborate the impacts and dependencies most relevant from their perspective. We encourage the adoption and adaptation of the Framework by diverse groups, and hope that over time, sector specific guidelines can emerge from the TEEBAgriFood Framework.

Further, the Framework is intended for use in an interdisciplinary manner, where the questions to be analysed, the options to be compared, and the scale, scope, and relevant variables included are determined in an open and participatory way. This engagement should occur before the appropriate assessment and valuation methods are implemented.

Overall, the Framework also allows for a broadening of our understanding and conversations around agricultural and food systems. Our aim is that international policies and targets increasingly begin to recognize the interlinkages, in terms of impacts and dependencies that food systems have with our economies, societies, health, and environment. In this task, using the Framework and its language can allow for the next generation of agricultural and food research to provide a more comprehensive basis for decision-making.