Josef
Baumgartner
Evaluation of Macroeconomic Forecasts for Austria in the 1980s and 1990s
The forecasts of the Austrian Institute of Economic Research (WIFO), the Institute for Advanced Studies (IHS) and the OECD for the Austrian economy have been evaluated for three key macroeconomic variables (GDP growth, unemployment rate, and rate of inflation) for the period from 1983 to 1999. In terms of accuracy, no significant differences have emerged between the three institutions, as far as the projections of growth and inflation are concerned. However, the prospects for unemployment are more precisely assessed by the two Austrian institutes. Compared with previous studies, forecasting errors exhibit a slight downward trend. The forecasts by the three institutions are largely unbiased and efficient, and  with the exception of the unemployment rate forecast by the OECD  clearly superior to "naïve" forecasting strategies.
Josef Baumgartner is an economist at WIFO. The author is grateful to Helmut Kramer, Marcus Scheiblecker, Thomas Url and Michael Wüger for useful and constructive comments • The data were processed and analysed with the assistance of Christine Kaufmann and Martha Steiner • Email addresses: Josef.Baumgartner@wifo.ac.at, Christine.Kaufmann@wifo.ac.at
CONTENT
Theil coefficients of inequality
Tests for directional accuracy (sign tests)
Directional
accuracy (sign test)
LIST OF TABLES
Table 1: Descriptive
statistics for autumn forecasts for next year from WIFO, IHS and OECD
Table 2: Forecast comparisons
for autumn forecasts for next year from WIFO, IHS and OECD
Table 3: Descriptive
statistics forspring forecasts for the current year from WIFO, IHS and OECD
Table 4: Forecast comparisons
for spring forecasts for the current year from WIFO, IHS and OECD
Table 5: Descriptive
statistics for autumn forecasts for next year from WIFO, IHS and OECD
Table 6: Forecast comparisons
for autumn forecasts for next year from WIFO, IHS and OECD
Table 7: Descriptive
statistics for spring forecasts for the current year from WIFO, IHS and OECD
Table 8: Forecast comparisons for spring forecasts for the current year from WIFO,
IHS and OECD
[1] Any forwardlooking action under uncertainty by economic policy, firms, consumers, employers or employees is based on expectations and thereby, implicitly or explicitly, upon a forecast of economic developments. The reaction of economic agents to changes in future prospects can be rather different. Thus, the prediction of a business cycle downturn may induce private agents to a more cautious consumption and investment behaviour, and wage settlements may turn out more moderate. The reaction by the government will depend on the goals of economic policy. From the range of options, just a few shall be mentioned here: deliberate countercyclical action or  less proactive  the operation of automatic stabilisers will drive up public expenditure and, with tax revenues falling at the same time, the budget deficit will increase. This will, by tendency, lean against the downward trend. If, on the other hand, fiscal policy wants to keep the government balance unchanged, it will try to compensate for the revenue losses by cutting expenditure. In this way, the downturn will be reinforced, at least in the short run[a]. Provided that the reactions by the different economic agents do not offset each other, the example illustrates the effect a (credible) forecast may have: depending on the reaction of economic agents, the forecast may be either selffulfilling (or selfreinforcing) or selfdestroying.
[2] This renders the evaluation of the accuracy of a forecast difficult. The most straightforward way to proceed would be to compare the forecast with the actual economic outcome (the realisation) and to regard the forecast as good if the degree of congruence is high. In case that the forecast has no effect on the outcome  like the weather not being influenced by the weather forecast  such an approach is perfectly adequate. However, the situation is somewhat different for economic forecasts: in the extreme case, the forecast may have such a strong influence on economic agents that the outcome is based on entirely different conditions and the expost comparison would show little or no congruence between forecast and reality. Still, such a forecast should not be called bad or meaningless, as the alternative course of events could not have been anticipated[b].
[3] Moreover, in judging the quality of a forecast one has to bear in mind that we are usually dealing with conditional forecasts that are valid only on certain assumptions[c]. For a small open economy like Austria such assumptions usually refer to international economic developments (economic trends in the EU and the USA, exchange rates, prices of raw materials, etc.) as well as to policy conditions in the EU and in Austria (such as policycontrolled interest rates, tax rates, etc.). The very assessment of international business conditions is one of the major sources of uncertainty for forecasts in small countries.
[4] The result of a forecast also depends on the (implicit or explicit) structure of the underlying macroeconomic explanatory context (econometric model). If the forecast is validated by actual developments, although the underlying assumptions were false, the forecast ought to be regarded as incorrect even if the forecasting error, defined as realisation minus forecast, is zero. Conversely, an incorrect forecast derived from correct assumptions points to deficiencies in the underlying explanatory model.
[5] Furthermore, the overall purpose of a forecast is not clearly defined, as different economic policymakers and forecasters may be guided by different interests. This problem area is reflected by the selection of a "loss function", i.e., the evaluation criterion for deviations of the forecast from realisations. For "rational" or "pure" forecasters it is important to produce a largely "correct" forecast coming as close as possible to the actual value of the projected variable. Upward and downward errors are given equal weight, implying a symmetric loss function.
[6] However, for economic policymakers and their advisers upward or downward deviations may weigh differently. Thus, it may be that the nonrecognition of an economic downturn or even more a recession carries a higher economic and political cost for a government than the incorrect perception of an upturn. If policy wants to react to a greater extent to a projected slower growth than to an announced higher rate of growth, then cautious economic forecasts will possibly be more appreciated than unbiased ones (Aiginger, 1979). This would point to an asymmetric loss function carrying a higher "sanction" for the overestimation of the actual outcome (i.e., a negative forecasting error)[d].
[7] For monetary policy mainly oriented towards price stability also an asymmetric loss function is plausible. However, with regard to real GDP (or potential output) it would be skewed rather to the opposite side of the one referred to above: the underestimation of real GDP growth (and of inflation) would weigh more heavily in the loss function and carry a higher "sanction".
[8] In the situations described, the asymmetric loss functions of the different users of the forecasts would have to be taken as the measure in assessing the quality of the forecasts. However, as the examples illustrate, policymakers may have (very) different loss functions for each of the key macroeconomic variables; thus, the debate would shift from the accuracy of the forecasts towards the shape of the appropriate loss function.
[9] Moreover, little is known about the statistical properties of forecast errors evaluated on the basis of asymmetric loss functions, and comparisons with previous research on the subject are not straightforward[e]. On the evaluation of forecasts on the basis of symmetric loss functions, however, there is an extensive literature that also allows comparisons over time and across countries.
[10] For these reasons, the measures of accuracy and statistical tests used in the present analysis assume a symmetric loss function. The arguments mentioned above underline that there is no absolute measure of forecast accuracy. It is nevertheless meaningful to investigate the relative properties of forecasts over time, across different variables and between different authors of forecasts.
[11] The following paper compares for the 1980s and 1990s the quality of the forecasts of the two leading Austrian economic research institutes  the Austrian Institute of Economic Research (WIFO) and the Institute for Advanced Studies (IHS)  with the OECD forecasts for Austria. The present analysis covers three key macroeconomic indicators and tries to find out whether the forecast errors exhibit any systematic bias. In addition, the differences in forecasting accuracy between WIFO and IHS on the one hand and the OECD on the other, are tested for their statistical significance[f].
[12] Several investigations of the forecasting performance of WIFO and IHS have been carried out in the past (see Baumgartner, 2002, for a review), with some studies also including the OECD forecasts for Austria (Kramer, 1980, Wörz, 1994, 1995, Öller  Barot, 2000). The quality of the OECD forecasts has been the subject of a great number of studies, particularly for the G7 countries (e.g., Ash  Smyth  Heravi, 1997, 1998, Koutsogeorgopoulou, 2000, Öller  Barot, 2000, Batchelor, 2001, Blix et al., 2001).
[13] With regard to comparisons of its own projections with forecasts from the "private sector", the OECD has repeatedly pointed out that the OECD projections are serving other purposes, namely "to discuss risks and deliver policy messages that are deemed to be useful to policymakers in member countries" (Lenain, 2002). However, this statement is valid to at least an equal extent for the forecasts of the two Austrian economic research institutes: thus, the institutes' forecasts usually do not include projections of policy reactions, but rather formulate recommendations for policy on the basis of the given economic policy framework.
[14] The analysis covers the annual forecasts of WIFO, IHS and the OECD for the Austrian economy from 1983 to 1999 for three widely watched macroeconomic variables:
· the growth rate of real GDP,
· the rate of unemployment (using the national definition for WIFO and IHS, and the OECD definition for the OECD forecast)[g],
· the rate of inflation based on the private consumption deflator.
[15] The two Austrian economic research institutes present macroeconomic forecasts four times a year (in March, June, September and December), each time for the current and the following year[h]. The OECD, in its semiannual "Economic Outlook" issued in June and December, submits economic projections for the current and the next year[i]. The OECD holds the view that for the purpose of comparisons with other forecasts, the "preliminary version" of its "Economic Outlook" should be used that is finalised in May and November of each year (see Lenain, 2002). As these dates come to fall regularly between the forecasting dates of WIFO and IHS, the choice of the forecast issues from the national institutes for the present analysis has the following implications:
· When using the March and September issues (Tables 1 to 4), the OECD, knowing the institutes' forecasts, has an information advantage. If, in this case, the forecasts from WIFO and IHS prove more accurate than those from the OECD, they have to be regarded as clearly better, since they have been made on the basis of a comparatively smaller information set.
· When, alternatively, using the June and December issues (Tables 5 to 8), it is the Austrian institutes that have an information advantage, since their set of information includes the most recent OECD forecast. Should the OECD forecast turn out more accurate, it is then of better quality, as in this case the OECD is in the less favourable starting position, as far as the information set is concerned.
[16] Thus, in judging the quality of the different forecasts, both alternatives have been used: from the autumn forecasts (September and December for WIFO and IHS) of year t1 and from the respective spring forecasts (March and June for WIFO and IHS) of year t the projected values for the current year t have been taken[j].
[17] For growth of GDP and the private consumption deflator, the realisation for year t, against which the forecasts are set, are the preliminary national accounts data calculated by WIFO, disposable in March of the following year[k]. This choice most closely corresponds to the state of information at the disposal of the forecasters at the time of their forecast. Later data revisions carried out by Statistics Austria often go back over several years and sometimes also include methodological changes, such as the changeover of the national accounts standards from SNA 68 to ESA 79 and ESA 95 (see in this regard also Baumgartner, 2002). For the rate of unemployment, the realisations taken for comparison with the WIFO and IHS projections (based on the national definition) are those given by the labour market service in March for the preceding year; for the OECD forecasts, the realisations were obtained from the "Economic Outlook" of the December issue of the following year.
[18] The accuracy of the forecasts is assessed on the basis of different criteria which may be divided into five groups[l]:
1. measures of statistical accuracy with regard to the forecasting error,
2. comparison with "naïve" forecasts (Theil coefficient of inequality),
3. tests for statistical significance of the differences between the two institutes' forecasts,
4. sign tests in order to check the directional forecast accuracy,
5. statistical tests for the unbiasedness and efficiency of the forecasts.
[19] Usually, the 95 percent level of significance is applied for the statistical tests used. This corresponds to a 5 percent probability of error for the underlying null hypothesis being rejected although being correct. In the tables, the error probability is given by the p value: a p value of 0.05 corresponds to an error probability of 5 percent, meaning that the null hypothesis is rejected (at least at the 95 percent level) if p is smaller than 0.05.
[20] The mean forecast error shows the average deviation of the projected from actual values, pointing to a forecast bias. If the forecast errors  defined as actual minus projected values  are distributed equally upwards and downwards, this figure is (close to) zero. A positive sign indicates that the forecast has a tendency of underestimating actual developments. Whether the forecast is actually biased is found out using the test procedures described below. An "optimal" predictor should, moreover, exhibit a lower variance than the realisations, since the forecast, unlike the actual value, includes no irregular component (see Granger  Newbold, 1977). In this regard, Aiginger (1979, p. 173) also points to a revealed "smoothing tendency": if information is scarce, it seems plausible to expect an average development. The more the future trend is uncertain, the more expectations will keep close to the arithmetic mean. The influence of a smoothing tendency is supported by the fact that it is stronger with forecasts of longer horizon than with shortterm ones. The ratio between the standard deviations of the forecasts and the realisation should therefore be smaller than 1 and diminish with increasing forecast horizon.
[21] In calculating the mean forecast error, the positive forecast errors are balanced against the negative ones. Statements going beyond the tendency of the forecast are therefore impossible to make. The mean absolute error (MAE; for all definitions see the methodological annex) and the mean squared error (MSE) can also measure the accuracy of a forecast. Hereby, the absolute values (MAE)  the squares, respectively (MSE)  of the forecast errors are added up and the average is taken. The two measures differ by the weight attributed to the forecast errors entering the calculation: for the MAE, they are weighted linearly, for the MSE by the power of 2; the latter measure therefore gives greater emphasis to large forecast errors. Frequently, instead of the MSE, the root mean square error (RMSE) is shown. This has the advantage that the calculated statistic is of the same dimension as the underlying variable. For all accuracy measures discussed holds: the smaller the value of the calculated statistics, the better the forecast.
[22] Furthermore the mean squared error can furthermore be split in two kinds of inequality proportions, in order to give further evidence on the accuracy of the forecast. The following relations hold:
UM + US + UC = 1,
UM + UR + UD = 1.
[23] A good forecast is characterised by small bias (UM), variance (US), and regression (UR) proportions, while the covariance (UC) or the distribution (UD) proportions should be close to 1 (see Theil, 1966, 1971, or the methodological annex).
[24] No information on the difficulty of predicting a particular variable enters the calculation of the mean absolute or mean squared error. Yet, variables with a low degree of variation are easier to predict than those subject to wide variations. To this end, the RMSE standardised with the standard deviation of the realisations _{} is calculated _{}. This measure allows a better comparison of forecasting errors between different variables.
[25] Theil (1966, 1971) developed several measures that also address the problem referred to above; these statistics are standardised to 1 and are therefore easier to interpret (see below). In general, they compare the RMSEs of different forecasts. In our case, the forecasts of the three research institutions are set against two "naïve" forecasting strategies suggested by Theil.
[26] The statistic Theil W sets as "naïve" forecast the hypothesis of "no change in the rate of change": the last known rate of change of the realisation is used for projecting future changes. For example, the "naïve" forecast made in year t would thus assume that real economic growth in the current as well as in the subsequent year equals the rate of last year (known at the time of the forecast).
[27] The "naïve" forecast according to the Theil U criterion hypothesises a "no change in level" situation. E.g., it would thereby assume a rate of real economic growth of zero for both the current and the following year.
[28] For both measures holds that the mean squared errors of the institutions' forecasts are set against the mean squared errors of the "naïve" forecasts. If the Theil W or Theil U statistics are smaller than 1, the forecasts of the institutions are superior to the "naïve" forecasts. A good forecast therefore exhibits for the Theil inequality coefficients values clearly below unity.
[29] The accuracy of two forecasts A and B is also examined using the modified DieboldMariano test (DM*) and the nonparametric Wilcoxon signed rank test (W*; see Diebold  Mariano, 1995, Hartung, 1991, Harvey  Leybourne  Newbold, 1997, Mariano, 2002, and the methodological annex). Diebold  Mariano (1995) have developed, for different loss functions, a test for the null hypothesis of "equal forecast accuracy" between two forecast providers. The present analysis uses both the absolute and the squared errors as loss functions to examine whether statistically significant differences between the forecasts (or the measures of forecast accuracy) of WIFO, IHS and the OECD exist.
[30] The loss functions used here belong to the family of symmetric loss functions. Particularly the users of a forecast (e.g., economic policymakers) may rate positive or negative forecast errors differently than the forecasters themselves (see the discussion further above). The test developed by Diebold  Mariano (1995) is applicable also to nonsymmetric loss functions. However, a generally accepted asymmetric loss function is not available, at least not at the present stage.
[31] In the tables the values for DM*AE and W*AE, or DM*SE and W*SE, respectively, show the p values of these tests for the absolute (AE) or squared (SE) loss function, respectively. For values below 0.05, the null hypothesis of equal forecasting accuracy can be rejected at the usual levels of significance, i.e., the differences are statistically significant.
[32] For the users of forecasts it is sometimes not the accuracy of a forecast that matters most, but whether at least the direction of future developments has been correctly assessed. Using a nonparametric test (see methodological annex) it is examined whether the direction of change given in a forecast corresponds with the actual course of events. The null hypothesis of the test is stated as follows: the sign of a change in the forecast and the sign of a change of the same variable as actually observed are mutually independent[m]. The ratio of congruence (ER, in the interval [0, 1]) specifies the degree of congruence over the period of analysis. If this ratio is below 0.5, than a forecast by tossing a coin would better predict the direction of actual change. Good forecasts show ratios of congruence close to 1, and the p value should be smaller than 0.05.
[33] Unbiased forecasts have the same mean as the realisations and thus exhibit a mean forecast error of zero. A standard procedure in this regard is the estimation of the "realisationforecast" regression equation by Mincer  Zarnowitz (1969),
_{},
[34] where _{} denominates the realisation, _{} the forecast and _{} the error term. Under the hypothesis of unbiasedness the parameter restrictions _{} = 0 und _{} = 1 must be fulfilled simultaneously, and the residual term _{} corresponds to the forecast error. This hypothesis can be tested by a joint F test. However, Holden  Peel (1990) have shown that this criterion is a sufficient, but not a necessary condition for an unbiased forecast: it is (theoretically) possible to establish unbiased forecasts that with certainty do not withstand that test. They therefore suggest to estimate the equation
_{}
[35] and verify the null hypothesis of unbiasedness through the ordinary t test m = 0. For a p value above 0.05, the null hypothesis cannot be rejected (at the usual level of significance).
[36] The hypothesis of efficiency of a forecast assumes that the forecasters make optimal use of all the information at their disposal. This assumption is verified by testing the residuals _{} from the "realisationforecast" regression according to Holden  Peel (1990) for firstorder autocorrelation through the DurbinWatson statistic (DW). In addition, the orthogonality between forecast errors and available information is checked via the regression
_{}.
[37] If the forecasters learn from past errors _{} and use this information to be derived from the forecast errors, then the joint hypothesis _{} and _{} should not be rejected on the usual level of significance. The p value for the F test should thus be greater than 0.05. The residuals _{} are tested for higherorder autocorrelation using the BoxLjung Q test. The null hypothesis of "no autocorrelation" is rejected for a p value below 0.05.
[38] The results of the forecast comparison, as shown in Tables 1 to 8, refer to the forecasting rounds of September and December (projections for next year) and March and June (for the current year), as far as the WIFO and IHS forecasts are concerned; as for the OECD forecasts, they refer to the autumn (for next year) and the spring forecasts (for the current year), respectively. With the exception of the unemployment rate, the correlation among the forecasts is higher than that between the forecasts and realisations. This observation is consistent with earlier findings and those obtained for other countries. It is explained by the fact that the information set at the disposal of the research institutes and the OECD is highly similar. The advantage of the larger amount of information on international economic developments accessible to the OECD is apparently offset by the higher degree of institutional knowledge at the command of the domestic research institutes. However, in projecting the unemployment variable, the better institutional knowledge may be of decisive importance (see below).
[39] The forecasts exhibit lower standard deviations than the actual outcomes, and the standard deviations diminish with the distance of the forecast horizon. For the variables examined, a systematic deviation towards caution is apparent: prospects generally deemed positive, such as growth of real GDP, tend to be projected too low, while those considered undesirable (e.g., unemployment and inflation) are rather overestimated. This is an indication that forecasters may implicitly face an asymmetric loss function (e.g., Aiginger, 1979). Such a tendency is partly even statistically significant (see the findings on unbiasedness below).
[40] The higher the volatility of a variable, the more difficult it is to predict; nevertheless, for all time series reviewed, the root mean square error (RMSE) is smaller (or only marginally higher) than one standard deviation of the realisations. An exception is the OECD projection of the unemployment rate for which the RMSE is more than twice as high as the standard deviation of the actual outcome. For both forecast horizons, the lowest (standardised) RMSE values have been found for the projections of inflation. Not surprisingly, the RMSE is lower, the shorter the forecast horizon, given the lower degree of uncertainty and thus the risk of error.
[41] A decomposition of the mean square error (MSE) shows that the larger part of the forecast errors is due to the covariance (UC) or regression (UR) proportion, i.e., the components beyond the influence of the forecaster. Exceptions in this regard are the unemployment projections by the OECD and the inflation projections of the IHS, each of which exhibit a (very) large bias proportion.
[42] For some of the variables examined it is possible to compare RMSE values over time, although not all studies refer to the same forecasting dates, forecast horizons and realisations[n]. On this basis it is difficult to assess the evolution of the forecasting errors. It seems, however, that for the RMSE statistics the trend has been slightly declining. The results of Öller  Barot (2000) for the WIFO and OECD projections of growth and inflation for Austria point into the same direction[o]. One should nevertheless bear in mind that as from the mid1980s cyclical variations were less strong than in the ten years before.
[43] The projections of the three forecasting institutions are of better quality than simple "no change" alternatives, with the exception of the OECD unemployment forecast. The TheilW measure of inequality for the OECD forecast of the unemployment rate is clearly above 1 for both forecast horizons. A similar result has been obtained by Wörz (1994, 1995). If the OECD were to take greater account of the figures projected by the domestic institutes in May and September that are already known at the time of its own forecast, it could have improved the quality of the latter. For the unemployment rate, the absolute and the squared forecasting errors are smaller for WIFO and IHS than for the OECD, a difference that is also statistically significant. This comes despite the fact that in the given situation the OECD operates from a better information base. The measures of forecast accuracy show, that the unemployment forecasts by WIFO and IHS are superior to those from the OECD.
[44] The assessment is more ambiguous for the growth rate of real GDP and the rate of inflation: the OECD can benefit from a higher level of information and exhibits (slightly) smaller RMSE values as compared with the forecasts of September and March of the two Austrian institutes. However, this result is turned around when the forecasts of December and June are used, thereby giving the information advantage to the Austrian institutes: in this setting, the forecasts by WIFO and IHS prove more accurate. Thus, from the present analysis no firm conclusion can be drawn as to whether the forecasts from the OECD or those from the domestic institutes are of better quality.
[45] As far as the directional accuracy of forecasts for different economic variables is concerned, the forecasted changes for all variables are correct in more than 50 percent of the cases (ER > 0.5) by all three forecasting institutions; yet, the null hypothesis of independence can be rejected unambiguously on the 95 percent level for the shorter forecast horizon only.
[46] As illustrated above, forecasts should on average not deviate from realisations (i.e., be unbiased) and make best use of the available information (i.e., be efficient). The hypothesis of unbiasedness and efficiency cannot be rejected at the 95 percent level of significance for the forecasts of GDP for all three institutions, of unemployment (of the Austrian institutes) and of the inflation forecast of the OECD.
Table 1: Descriptive statistics for autumn forecasts for next year from WIFO, IHS and OECD 

WIFO, IHS: September forecasts 



Real GDP growth 
Unemployment 
Inflation 



Forecast 
Realisation 
Forecast 
Realisation 
Forecast 
Realisation 



WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 


















Mean 
2.26 
2.27 
2.23 
2.41 
2.41 
5.68 
5.69 
4.20 
5.62 
3.56 
2.84 
3.11 
2.74 
2.55 
2.55 

Variance 
0.62 
0.64 
0.50 
1.48 
1.48 
0.65 
0.62 
0.45 
0.49 
0.09 
1.14 
1.14 
1.09 
1.74 
1.74 

Standard deviation 
0.79 
0.80 
0.71 
1.22 
1.22 
0.81 
0.79 
0.67 
0.70 
0.30 
1.07 
1.07 
1.04 
1.32 
1.32 

Correlation 
















With realisation 
0.14 
0.01 
0.14 


0.71 
0.62 
0.57 


0.89 
0.85 
0.89 



With WIFO 

0.88 
0.90 



0.98 
0.46 



0.97 
0.99 



With IHS 


0.84 




0.55 




0.96 



Realisations: first release by WIFO as of March t+1. 




Table 2: Forecast comparisons for autumn forecasts for next year from WIFO, IHS and OECD 

WIFO, IHS: September forecasts 


Real GDP growth 
Unemployment 
Inflation 


WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
Accuracy measures 









ME 
0.15 
0.14 
0.18 
–0.07 
–0.08 
–0.64 
–0.29 
–0.56 
–0.19 
STDR 
0.65 
0.66 
0.58 
1.15 
1.12 
2.22 
0.81 
0.81 
0.79 
MAE 
1.20 
1.20 
1.10 
0.50 
0.53 
0.65 
0.54 
0.66 
0.54 
MSE 
1.86 
2.12 
1.77 
0.34 
0.43 
0.71 
0.46 
0.79 
0.42 
RMSE 
1.36 
1.45 
1.33 
0.58 
0.66 
0.84 
0.68 
0.89 
0.65 
UM 
0.01 
0.01 
0.02 
0.01 
0.01 
0.57 
0.18 
0.40 
0.09 
US 
0.10 
0.08 
0.14 
0.03 
0.02 
0.19 
0.14 
0.08 
0.18 
UC 
0.89 
0.91 
0.84 
0.95 
0.97 
0.24 
0.68 
0.52 
0.73 
UR 
0.21 
0.29 
0.16 
0.28 
0.29 
0.34 
0.02 
0.00 
0.04 
UD 
0.78 
0.70 
0.82 
0.71 
0.70 
0.09 
0.80 
0.60 
0.87 
_{} 
1.12 
1.20 
1.10 
0.83 
0.94 
2.80 
0.51 
0.68 
0.49 
Theil W 
0.72 
0.76 
0.72 
0.89 
0.98 
1.55 
0.43 
0.56 
0.40 
Theil U 
0.52 
0.55 
0.50 
0.11 
0.12 
0.25 
0.27 
0.36 
0.26 










Test of
significance of forecast differentials 








With WIFO DM*AE 

1.000 
0.259 

0.395 
0.217 

0.102 
1.000 
W*AE 

0.959 
0.420 

0.081 
0.001 

0.161 
0.224 
DM*SE 

0.464 
0.655 

0.029 
0.045 

0.088 
0.534 
W*SE 

0.815 
0.568 

0.086 
0.344 

0.065 
0.136 
With IHS DM*AE 


0.398 


0.312 


0.236 
W*AE 


0.917 


0.002 


0.153 
DM*SE 


0.398 


0.132 


0.141 
W*SE 


0.776 


0.324 


0.082 










Directional forecast accuracy 









ER 
0.69 
0.63 
0.81 
0.50 
0.58 
0.50 
0.81 
0.69 
0.69 
p value 
0.131 
0.280 
0.012 
0.248 
1.000 
1.000 
0.006 
0.091 
0.131 










Unbiasedness 









m 
0.150 
0.138 
0.181 
–0.067 
–0.075 
–0.638 
–0.287 
–0.563 
–0.194 
p value 
0.674 
0.718 
0.602 
0.710 
0.710 
0.003 
0.089 
0.007 
0.246 
DW 
1.700 
1.689 
1.987 
2.295 
2.164 
1.848 
1.437 
1.592 
1.697 










Efficiency 









_{} 
0.026 
–0.004 
0.116 
–0.006 
0.006 
–0.6 
–0.254 
–0.508 
–0.228 
p value 
0.946 
0.991 
0.761 
0.974 
0.975 
0.034 
0.182 
0.055 
0.214 
F test
for all _{} 
0.919 
0.926 
0.953 
0.653 
0.747 
0.021 
0.124 
0.02 
0.343 
BL 
0.606 
0.767 
0.788 
0.087 
0.187 
0.049 
0.445 
0.763 
0.552 
Realisations: first release by WIFO as of March t+1.  ME . . . mean error, STDR . . . standard deviation ratio (forecast/realisation), MAE . . . mean absolute error, MSE . . . mean squared error, RMSE . . . root of MSE, UM . . . bias proportion, US . . . variance proportion, UC . . . covariance proportion, UR . . . regression proportion, UD . . . distribution proportion, UM+US+UC = 1 and UM+UR+UD = 1, _{} . . . ratio between RSME of forecast and standard deviation of realisation, Theil W . . . Theil inequality statistic W, Theil U . . . Theil inequality statistic U; DM* . . . p value for the asymptotically tdistributed modified DieboldMariano test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE), W* . . . p value for the asymptotically standardnormally distributed Wilcoxon signed rank test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE); ER . . . congruence between forecasted and actual sign changes of the variable observed, p value . . probability of error for the _{}distributed test for independence between forecasted and actual sign changes; m . . . constant term of the regression line in the restricted estimation, p value for the t test m=0, DW . . . DurbinWatson statistic; _{} . . . constant term of the regression line, p value for the t test _{}, F Test . . . p value for the joint F test _{} and _{}, BL . . . p value for the BoxLjung Q test for the null hypothesis "no autocorrelation". 
Table 3: Descriptive statistics forspring forecasts for the current year from WIFO, IHS and OECD 

WIFO, IHS: March forecasts 



Real GDP growth 
Unemployment 
Inflation 



Forecast 
Realisation 
Forecast 
Realisation 
Forecast 
Realisation 



WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 


















Mean 
2.09 
2.01 
1.96 
2.38 
2.38 
5.57 
5.61 
4.14 
5.53 
3.60 
3.79 
3.86 
2.76 
2.61 
2.61 

Variance 
0.95 
0.84 
1.02 
1.40 
1.40 
0.67 
0.68 
0.33 
0.54 
0.10 
1.75 
1.86 
1.48 
1.70 
1.70 

Standard deviation 
0.98 
0.92 
1.01 
1.18 
1.18 
0.82 
0.83 
0.57 
0.74 
0.32 
1.32 
1.36 
1.22 
1.30 
1.30 

Correlation 
















With realisation 
0.66 
0.69 
0.71 


0.93 
0.90 
0.43 


–0.34 
–0.41 
0.97 



With WIFO 

0.96 
0.98 



0.99 
0.20 



0.99 
–0.22 



With IHS 


0.97 




0.23 




–0.28 



Realisations: first release by WIFO as of March t+1. 




Table 4: Forecast comparisons for spring forecasts for the current year from WIFO, IHS and OECD 

WIFO, IHS: March forecasts 


Real GDP growth 
Unemployment 
Inflation 


WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
Accuracy measures 









ME 
0.29 
0.37 
0.41 
–0.04 
–0.08 
–0.54 
–0.20 
–0.26 
–0.15 
STDR 
0.82 
0.78 
0.85 
1.11 
1.12 
1.77 
0.88 
0.93 
0.93 
MAE 
0.71 
0.70 
0.71 
0.25 
0.29 
0.54 
0.33 
0.30 
0.28 
MSE 
0.91 
0.89 
0.89 
0.09 
0.13 
0.56 
0.19 
0.16 
0.14 
RMSE 
0.96 
0.95 
0.94 
0.31 
0.36 
0.75 
0.44 
0.40 
0.37 
UM 
0.09 
0.15 
0.19 
0.02 
0.05 
0.52 
0.21 
0.45 
0.16 
US 
0.05 
0.08 
0.03 
0.07 
0.06 
0.11 
0.14 
0.06 
0.06 
UC 
0.86 
0.77 
0.77 
0.91 
0.89 
0.38 
0.65 
0.50 
0.78 
UR 
0.04 
0.01 
0.03 
0.20 
0.20 
0.33 
0.06 
0.02 
0.01 
UD 
0.87 
0.83 
0.78 
0.79 
0.76 
0.15 
0.73 
0.53 
0.83 
_{} 
0.81 
0.80 
0.80 
0.42 
0.49 
2.32 
0.33 
0.30 
0.28 
Theil W 
0.66 
0.64 
0.65 
0.82 
0.96 
2.10 
0.42 
0.37 
0.35 
Theil U 
0.36 
0.36 
0.36 
0.06 
0.06 
0.21 
0.15 
0.14 
0.13 










Test of
significance of forecast differentials 








With WIFO DM*AE 

0.550 
0.922 

0.184 
0.109 

0.176 
0.227 
W*AE 

0.715 
0.714 

0.037 
0.000 

0.331 
0.875 
DM*SE 

0.482 
0.643 

0.067 
0.178 

0.083 
0.204 
W*SE 

0.715 
0.428 

0.034 
1.000 

0.030 
0.235 
With IHS DM*AE 


0.394 


0.189 


0.785 
W*AE 


0.794 


0.000 


0.937 
DM*SE 


0.486 


0.222 


0.682 
W*SE 


0.183 


0.905 


0.937 










Directional forecast accuracy 









ER 
0.81 
0.94 
0.88 
0.67 
0.67 
0.58 
0.88 
0.94 
0.94 
p value 
0.012 
0.000 
0.003 
0.700 
0.700 
0.296 
0.003 
0.000 
0.000 










Unbiasedness 









m 
0.250 
0.319 
0.384 
–0.042 
–0.075 
–0.550 
–0.206 
–0.262 
–0.162 
p value 
0.312 
0.177 
0.105 
0.672 
0.513 
0.006 
0.063 
0.004 
0.086 
DW 
1.647 
2.007 
1.753 
1.919 
2.102 
2.122 
1.230 
1.090 
1.311 










Efficiency 









_{} 
0.243 
0.331 
0.349 
–0.011 
0.049 
–0.574 
–0.147 
–0.143 
–0.104 
p value 
0.381 
0.238 
0.220 
0.919 
0.698 
0.059 
0.236 
0.190 
0.331 
F test
for all _{} 
0.472 
0.446 
0.282 
0.991 
0.879 
0.054 
0.065 
0.007 
0.129 
BL 
0.439 
0.422 
0.290 
0.251 
0.255 
0.708 
0.158 
0.390 
0.129 
Realisations: first release by WIFO as of March t+1.  ME . . . mean error, STDR . . . standard deviation ratio (forecast/realisation), MAE . . . mean absolute error, MSE . . . mean squared error, RMSE . . . root of MSE, UM . . . bias proportion, US . . . variance proportion, UC . . . covariance proportion, UR . . . regression proportion, UD . . . distribution proportion, UM+US+UC = 1 and UM+UR+UD = 1, _{} . . . ratio between RSME of forecast and standard deviation of realisation, Theil W . . . Theil inequality statistic W, Theil U . . . Theil inequality statistic U; DM* . . . p value for the asymptotically tdistributed modified DieboldMariano test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE), W* . . . p value for the asymptotically standardnormally distributed Wilcoxon signed rank test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE); ER . . . congruence between forecasted and actual sign changes of the variable observed, p value . . probability of error for the _{}distributed test for independence between forecasted and actual sign changes; m . . . constant term of the regression line in the restricted estimation, p value for the t test m=0, DW . . . DurbinWatson statistic; _{} . . . constant term of the regression line, p value for the t test _{}, F Test . . . p value for the joint F test _{} and _{}, BL . . . p value for the BoxLjung Q test for the null hypothesis "no autocorrelation". 
Table 5: Descriptive statistics for autumn forecasts for next year from WIFO, IHS and OECD 

WIFO, IHS: December forecasts 



Real GDP growth 
Unemployment 
Inflation 



Forecast 
Realisation 
Forecast 
Realisation 
Forecast 
Realisation 



WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 


















Mean 
2.22 
2.16 
2.23 
2.41 
2.41 
5.69 
5.68 
4.20 
5.62 
3.56 
2.90 
2.92 
2.74 
2.54 
2.55 

Variance 
0.55 
0.51 
0.50 
1.48 
1.48 
0.64 
0.64 
0.45 
0.49 
0.09 
1.18 
1.21 
1.09 
1.60 
1.74 

Standard deviation 
0.74 
0.71 
0.71 
1.22 
1.22 
0.80 
0.80 
0.67 
0.70 
0.30 
1.08 
1.10 
1.04 
1.26 
1.32 

Correlation 
















With realisation 
0.32 
0.37 
0.14 


0.79 
0.78 
0.57 


0.92 
0.94 
0.89 



With WIFO 

0.92 
0.89 



0.99 
0.41 



0.98 
0.98 



With IHS 


0.90 




0.47 




0.97 



Realisations: first release by WIFO as of March t+1. 




Table 6: Forecast comparisons for autumn forecasts for next year from WIFO, IHS and OECD 

WIFO, IHS: December forecasts 


Real GDP growth 
Unemployment 
Inflation 


WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
Accuracy measures 









ME 
0.19 
0.24 
0.18 
–0.08 
–0.06 
–0.64 
–0.36 
–0.38 
–0.19 
STDR 
0.61 
0.59 
0.58 
1.14 
1.14 
2.22 
0.86 
0.87 
0.79 
MAE 
0.99 
0.99 
1.10 
0.41 
0.39 
0.65 
0.51 
0.46 
0.54 
MSE 
1.49 
1.40 
1.77 
0.25 
0.25 
0.71 
0.38 
0.35 
0.42 
RMSE 
1.22 
1.18 
1.33 
0.50 
0.50 
0.84 
0.62 
0.59 
0.65 
UM 
0.02 
0.04 
0.02 
0.02 
0.01 
0.57 
0.34 
0.42 
0.09 
US 
0.15 
0.18 
0.14 
0.04 
0.04 
0.19 
0.08 
0.08 
0.18 
UC 
0.83 
0.78 
0.84 
0.94 
0.95 
0.24 
0.57 
0.51 
0.73 
UR 
0.09 
0.05 
0.16 
0.24 
0.25 
0.34 
0.02 
0.02 
0.01 
UD 
0.89 
0.91 
0.82 
0.74 
0.74 
0.09 
0.64 
0.56 
0.87 
_{} 
1.01 
0.97 
1.10 
0.72 
0.72 
2.80 
0.49 
0.47 
0.49 
Theil W 
0.64 
0.64 
0.72 
0.81 
0.79 
1.55 
0.39 
0.43 
0.40 
Theil U 
0.46 
0.45 
0.50 
0.09 
0.09 
0.25 
0.25 
0.24 
0.26 










Test of
significance of forecast differentials 








With WIFO DM*AE 

0.888 
0.058 

0.536 
0.063 

0.244 
0.425 
W*AE 

0.979 
0.349 

0.048 
0.000 

0.250 
0.917 
DM*SE 

0.001 
0.089 

0.952 
0.030 

0.504 
0.327 
W*SE 

0.856 
0.204 

0.316 
0.782 

0.600 
0.716 
With IHS DM*AE 


0.211 


0.049 


0.232 
W*AE 


0.243 


0.000 


0.364 
DM*SE 


0.057 


0.029 


0.300 
W*SE 


0.055 


0.843 


0.533 










Directional forecast accuracy 









ER 
0.81 
0.75 
0.81 
0.67 
0.58 
0.50 
0.75 
0.69 
0.69 
p value 
0.012 
0.049 
0.012 
0.700 
1.000 
1.000 
0.016 
0.037 
0.131 










Unbiasedness 









m 
0.188 
0.244 
0.181 
–0.075 
–0.058 
–0.638 
–0.363 
–0.381 
–0.194 
p value 
0.557 
0.428 
0.602 
0.626 
0.707 
0.003 
0.014 
0.005 
0.246 
DW 
1.961 
1.873 
1.987 
2.258 
2.040 
1.848 
1.503 
1.433 
1.697 










Efficiency 









_{} 
0.151 
0.169 
0.116 
–0.039 
–0.001 
–0.600 
–0.337 
–0.321 
–0.228 
p value 
0.672 
0.619 
0.761 
0.812 
0.996 
0.034 
0.057 
0.057 
0.214 
F test
for all _{} 
0.905 
0.850 
0.953 
0.791 
0.922 
0.021 
0.024 
0.011 
0.343 
BL 
0.659 
0.583 
0.788 
0.066 
0.064 
0.049 
0.391 
0.181 
0.552 
Realisations: first release by WIFO as of
March t+1.  ME . . . mean error, STDR . . . standard deviation ratio (forecast/realisation), MAE . . . mean absolute error, MSE . . . mean squared
error, RMSE . . . root of MSE, UM . . . bias
proportion, US . . . variance
proportion, UC . . . covariance
proportion, UR . . . regression
proportion, UD . . . distribution
proportion, UM+US+UC = 1 and UM+UR+UD = 1, _{} . . . ratio between RSME of forecast and standard deviation
of realisation, Theil W . . . Theil inequality
statistic W, Theil U . . . Theil
inequality statistic U; DM* . . . p value for the asymptotically tdistributed modified DieboldMariano
test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE), W* . . . p value for the asymptotically standardnormally distributed Wilcoxon
signed rank test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE); ER . . . congruence
between forecasted and actual sign changes of the variable observed, p value . . probability of error for the _{}distributed test for
independence between forecasted and actual sign changes; m . . . constant term of the regression line in the restricted
estimation, p value for the t test m=0, DW . . . DurbinWatson
statistic; _{} . . . constant term of the regression line,
p value for the t test _{}, F Test . . . p value for the joint F test _{} and _{}, BL . . . p value for the BoxLjung Q test for the null hypothesis "no autocorrelation". 
Table 7: Descriptive statistics for spring forecasts for the current year from WIFO, IHS and OECD 

WIFO, IHS: June forecasts 



Real GDP growth 
Unemployment 
Inflation 



Forecast 
Realisation 
Forecast 
Realisation 
Forecast 
Realisation 



WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO, IHS 
OECD 


















Mean 
2.21 
2.09 
1.96 
2.38 
2.38 
5.54 
5.58 
4.14 
5.53 
3.60 
3.75 
3.79 
2.76 
3.70 
2.61 

Variance 
1.45 
1.38 
1.02 
1.40 
1.40 
0.57 
0.58 
0.33 
0.54 
0.10 
2.14 
2.54 
1.48 
2.26 
1.70 

Standard deviation 
1.20 
1.17 
1.01 
1.18 
1.18 
0.76 
0.76 
0.57 
0.74 
0.32 
1.46 
1.59 
1.22 
1.50 
1.30 

Correlation 
















With realisation 
0.85 
0.81 
0.71 


0.97 
0.96 
0.43 


0.99 
0.98 
0.97 



With WIFO 

0.97 
0.95 



0.99 
0.17 



0.99 
–0.18 



With IHS 


0.96 




0.23 




–0.25 



Realisations: first release by WIFO as of March t+1. 




Table 8: Forecast comparisons for spring forecasts for the current year from WIFO, IHS and OECD 

WIFO, IHS: June forecasts 


Real GDP growth 
Unemployment 
Inflation 


WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
WIFO 
IHS 
OECD 
Accuracy measures 









ME 
0.16 
0.29 
0.41 
–0.01 
–0.05 
–0.54 
–0.05 
–0.09 
–0.15 
STDR 
1.02 
0.99 
0.85 
1.03 
1.04 
1.77 
0.97 
1.06 
0.93 
MAE 
0.51 
0.61 
0.71 
0.16 
0.20 
0.54 
0.16 
0.26 
0.28 
MSE 
0.47 
0.62 
0.89 
0.03 
0.05 
0.56 
0.05 
0.11 
0.14 
RMSE 
0.68 
0.79 
0.94 
0.18 
0.23 
0.75 
0.22 
0.33 
0.37 
UM 
0.06 
0.13 
0.19 
0.00 
0.04 
0.52 
0.05 
0.08 
0.16 
US 
0.00 
0.00 
0.03 
0.02 
0.02 
0.11 
0.03 
0.08 
0.06 
UC 
0.94 
0.87 
0.77 
0.98 
0.94 
0.38 
0.92 
0.84 
0.78 
UR 
0.09 
0.08 
0.03 
0.06 
0.07 
0.33 
0.01 
0.13 
0.01 
UD 
0.86 
0.79 
0.78 
0.94 
0.89 
0.15 
0.94 
0.78 
0.83 
_{} 
0.58 
0.67 
0.80 
0.24 
0.31 
2.32 
0.15 
0.22 
0.28 
Theil W 
0.46 
0.52 
0.65 
0.47 
0.59 
2.10 
0.16 
0.24 
0.35 
Theil U 
0.26 
0.30 
0.36 
0.03 
0.04 
0.21 
0.05 
0.08 
0.13 








