guide2008_en

2.6Risk assessment

Project appraisal is a forecasting exercise rather than the formulation of an opinion. However, no forecast is without problems. For example, one may know that because of data limitations the forecasts for the demand for drinking water are affected by estimates that are prone to considerable errors. Engineers may explain that the data regarding the performance of the equipment they suggest for use are valid only approximately. One may also have doubts about some parameters crucial to the calculation of the return, such as the shadow wage.

Traditionally, a distinction between risk and uncertainty concepts is made. In some circumstances there is just uncertainty, but in other cases this can be transformed into ‘risk’ with an assessment of probability distributions indicating the likelihood of the realised value of a variable falling within stated limits. Accordingly, it becomes apparent that risk, but not uncertainty, is subject to empirical measurement, and can be analysed and possibly managed.

Against this background, the Funds’ regulations require a risk assessment for major infrastructure and productive investment projects (Article 40 1083/2006 EU Regulations).

A risk assessment consists of studying the probability that a project will achieve a satisfactory performance (in terms of some threshold value of the IRR or the NPV). Probability should here be understood as an index that takes the value 1 under full certainty that a prediction will be confirmed, a zero value for certainty that the prediction will not be confirmed, and intermediate values for anything in between the two extremes.

The recommended steps for assessing the project risk are:

-sensitivity analysis

-probability distributions for critical variables

-risk analysis

-assessment of acceptable levels of risk

-risk prevention.

The rest of the section presents the above mentioned steps and highlights the following additional topics:

-the switching value

-scenario analysis

-precautionary principle

-optimism bias.

2.6.1Sensitivity analysis

Sensitivity analysis allows the determination of the ‘critical’ variables or parameters of the model. Such variables are those whose variations, positive or negative, have the greatest impact on a project’s financial and/or economic performance. The analysis is carried out by varying one element at a time and determining the effect of that change on IRR or NPV.

The criteria to be adopted for the choice of the critical variables vary according to the specific project and must be accurately established on a case-by-case basis. As a general criterion, the recommendation is to consider those variables or parameters for which an absolute variation of 1% around the best estimate gives rise to a corresponding variation of not less than 1% (one percentage point) in the NPV (i.e. elasticity is unity or greater).

60

Table 2.14 Impact analysis of critical variables

An example of a possible result of the sensitivity analysis is shown in Figure 2.5: according to the aforementioned general criterion (a variation of the variable of 1% corresponds to at least one percentage point variation in NPV), the critical variables are demand and productivity, while energy cost and input prices are below the threshold.

Figure 2.5 Sensitivity analysis

FOCUS: THE SWITCHING VALUE

The switching value of a variable is that value that would have to occur in order for the NPV of the project to become zero, or more generally, for the outcome of the project to fall below the minimum level of acceptability.

The use of switching values in sensitivity analysis allows appraisers to make some judgements on the riskiness of the project and the opportunity of undertaking risk-preventing actions. For example if one of the critical variables of a transport project is ‘forecasted demand’ and its switching value is -20%, then the proposer can evaluate if the conditions for such a decrease exist and, in a positive case may consider preventing actions (e.g. tariffs reduction).

The following table provides some examples of switching values for an agricultural project:

Source: adapted from Belli et al. (2001).

62

Figure 2.6 Probability distribution for NPV

The cumulative probability curve (or a table of values) permits an assessment of the project risk, for example by verifying whether the cumulative probability is higher or lower than a reference value that is considered to be critical. One can also assess the probability that the NPV (or FRR) will be lower than a certain value, which is adopted as the benchmark (e.g. zero for NPV and 5% for FRR). In the example, see Figure 2.7, there is a probability of about 30% that the NPV will be negative.

Figure 2.7 Cumulative probability distribution for NPV

2.6.4Assessment of acceptable levels of risk

Often the NPVs and IRRs reported in project appraisal reports refer to best or baseline estimates, perhaps meaning ‘most likely’ values (or mode). However, the criterion for project acceptability should be that of the expected value (or mean) of such indicators, calculated from the underlying probability distributions.

For instance, if a project has a “baseline” ERR of 10% but also the probability risk analysis tells us that the ERR has a value between 4 and 10 with a probability of 70% and a value between 10 and 13 with a probability of 30%, then the expected value of ERR for that project is only 8.35% [average(4,10)*0.7 + average(10,13)*0.3].

In conclusion, the procedure described allows for the selection of projects not only on the basis of the best estimate, but also based on the risk associated with it, simply by weighting the performance with the risk. In principle, the expected performance, and not the modal one, is the value that should be reported in the application form for major projects requiring EU assistance. In order to evaluate the result, one

64

measures for the mitigation of the identified risks, according to international good practice (see Annex H for some examples extracted from the World Bank Project Appraisal Documents).

2.7Other project evaluation approaches

While cost-benefit analysis is the most commonly used technique in appraising public investment and it is the one required by the Funds regulations for major projects, other kinds of project analysis exist and are used. In this section, the main features and fields of application of Cost-Effectiveness Analysis (CEA), Multi-Criteria Analysis (MCA) and Economic Impact Analysis (EIA) are reviewed. These approaches cannot be seen as substitutes for CBA but rather as complements for special reasons, or as a rough approximation when actual CBA is impossible. Moreover, they are difficult to standardise and, under the Structural, Cohesion and IPA Funds, should be used with caution in order to avoid inconsistencies across regions and countries that will make the assessment of projects by the Commission Services more difficult.

2.7.1Cost-effectiveness analysis

Cost-effectiveness analysis (CEA) is a comparison of alternative projects with a unique common effect which may differ in magnitude. It aims to select the project that, for a given output level, minimises the net present value of costs, or, alternatively, for a given cost, maximises the output level. CEA results are useful for those projects whose benefits are very difficult, if not impossible, to evaluate, while costs can be predicted more confidently. This methodology is often used in the economic evaluation of healthcare programmes, but it can also be used to assess some scientific research, education and environmental projects. For these examples, simple CEA ratios are used, such as the cost of research per patent, the cost of education per student, the cost per unit of emission reduction, and so on. CEA is less helpful when a value, even an indicative one, can be given to the benefits and not just to the costs.

Generally, CEA solves a problem of optimization of resources that is usually presented in the following two forms:

-given a fixed budget and n alternative projects, decision-makers aim to maximise the outcomes achievable, measured in terms of effectiveness (E);

-given a fixed level of E that has to be achieved, decision-makers aim to minimise the cost (C).

Although one could compare the simple ratios of costs to outcomes (C/E) for each alternative, the correct comparison is based on ratios of incremental costs to incremental outcomes, since this tells us how much we are paying in adding the extra, more beneficial, measure. In particular, when the alternative projects are competitors and mutually exclusive, an incremental analysis is required in order to rank the projects and single out the one that is most cost-effective.

Generally cost-effectiveness analysis is pursued to test the null hypothesis that the mean cost-effectiveness of one project (a) is different from the mean cost-effectiveness of some competing intervention (b). It is calculated as the ratio:

R = (Ca – Cb) / (Ea – Eb) = C / E defining the incremental cost per unit of additional outcome.

While the measurement of costs is the same as in the financial analysis of CBA, the measurement of the effectiveness depends on the type of outcome chosen. Some examples of measures of effectiveness used in CEA are: number of life-years gained, days of disability avoided (healthcare projects), or test scores (education).

When a strategy is both more effective and less costly than the alternative (Ca – Cb < 0 and Ea – Eb > 0), it is said to ‘dominate’ the alternative: in this situation there is no need to calculate cost-effectiveness ratios, because the decision on the strategy to choose is obvious. However, in most circumstances, the project under examination is contemporaneously both more (or less) costly and more (or less) effective than the

66

There are many ways to design an MCA exercise. One possible approach is as follows:

-objectives should be expressed in measurable variables. They should not be redundant but could be alternative (the achievement of a bit more of one objective could partly preclude the achievement of the other);

-once the ‘objectives vector’ has been determined, a technique should be found to aggregate information and to make a choice; the objectives should have assigned weights reflecting the relative importance given to them by the policy-maker;

-definition of the appraisal criteria; these criteria could refer to the priorities pursued by the different parties involved or they could refer to particular evaluation aspects;

-impact analysis: this activity involves describing, for each of the chosen criteria, the effects it produces. Results could be quantitative or qualitative;

-forecast of the effects of the intervention in terms of the selected criteria; from the results coming from the previous stage (both in qualitative and in quantitative terms) a score, or a normalised value, is assigned (this is the equivalent of ‘money’ in CBA);

-identification of the typology of subjects involved in the intervention and the determination of respective preference functions (weights) accorded to different criteria;

-scores under each criterion are then aggregated (simply with a sum or with a non-linear formula) to give a numerical evaluation of the intervention; the result can then be compared with the result for other similar interventions.

The project examiner should then verify if:

-forecasts for non-monetary aspects have been quantified in a realistic way in the ex-ante evaluation;

-there is in any case a CBA for the standard objectives (financial and economic analysis);

-the additional criteria under the MCA have a reasonable political weight, so as to determine significant changes in the financial and economic results.

EXAMPLE: MCA AS A COMPLEMENTARY TOOL TO APPRAISE ENVIRONMENTAL PROJECTS

MCA is useful when the monetisation of costs and benefits is difficult or even impossible. Let us suppose that a certain project shows, at a discount rate of 5%, a negative economic net present value of one Millions of Euros. This means that the project examiner foresees a net social loss of the project in monetary terms. However, the project proposer could assess that, despite this, the project should be financed by the Funds because it has a ‘very positive’ environmental impact that it is not possible to monetise. For example the project is supposed to cut the polluter Z emissions by 10% per year.

Now one should ask if:

a)the forecast of the emission cut in physical terms is reliable;

b)one Millions of Euros is an acceptable ‘price’ for the reduction of 10% in the emissions;

c)such a ‘price’ is consistent with the weight that the government of the Member State or the Commission attaches to similar projects.

For instance, one may see whether -regularly or even occasionallyMember States have funded similar projects or, if there is no evidence of consistency, one should enquire why this deviation from previous practice is proposed for the project under EU assistance.

When the benefits are not just non-monetary, but also physically not measurable, a qualitative analysis should still be conducted. A set of criteria relevant for the project appraisal (equity, environmental impact, equal opportunity) is collected in a matrix together with the impacts (measured with scores or percentages) of the project on the relevant criteria. Another matrix should then assign weights to each relevant criteria. By multiplying scores and weights, the total impact of the project is obtained: this allows the selection of the best alternative. In the example shown in Table 2.17, project B has greater social impact, given preferences for the chosen social criteria. It should be stressed that these matrices are simple, but highly subjective, and much prudence is needed in the interpretation of the results.

68