WIFO

Josef Baumgartner

Evaluation of Macro-economic Forecasts for Austria in the 1980s and 1990s

 

The forecasts of the Austrian Institute of Economic Research (WIFO), the Institute for Advanced Studies (IHS) and the OECD for the Austrian economy have been evaluated for three key macro-economic variables (GDP growth, unemployment rate, and rate of inflation) for the period from 1983 to 1999. In terms of accuracy, no significant differences have emerged between the three institutions, as far as the projections of growth and inflation are concerned. However, the prospects for unemployment are more precisely assessed by the two Austrian institutes. Compared with previous studies, forecasting errors exhibit a slight downward trend. The forecasts by the three institutions are largely unbiased and efficient, and - with the exception of the unemployment rate forecast by the OECD - clearly superior to "naïve" forecasting strategies.

 

Josef Baumgartner is an economist at WIFO. The author is grateful to Helmut Kramer, Marcus Scheiblecker, Thomas Url and Michael Wüger for useful and constructive comments • The data were processed and analysed with the assistance of Christine Kaufmann and Martha Steiner • E-mail addresses: Josef.Baumgartner@wifo.ac.at, Christine.Kaufmann@wifo.ac.at

 

CONTENT

The data

Methods

Measures of accuracy

Theil coefficients of inequality

Significance tests

Tests for directional accuracy (sign tests)

Unbiasedness and efficiency

Results

Summary and conclusions

Methodological annex

Notation

Measures of accuracy

Theil measure of inequality

Significance tests

Diebold-Mariano test

Wilcoxon's signed rank test

Directional accuracy (sign test)

References

 

LIST OF TABLES

Table 1: Descriptive statistics for autumn forecasts for next year from WIFO, IHS and OECD. 7

Table 2: Forecast comparisons for autumn forecasts for next year from WIFO, IHS and OECD. 8

Table 3: Descriptive statistics forspring forecasts for the current year from WIFO, IHS and OECD. 9

Table 4: Forecast comparisons for spring forecasts for the current year from WIFO, IHS and OECD. 9

Table 5: Descriptive statistics for autumn forecasts for next year from WIFO, IHS and OECD. 10

Table 6: Forecast comparisons for autumn forecasts for next year from WIFO, IHS and OECD. 11

Table 7: Descriptive statistics for spring forecasts for the current year from WIFO, IHS and OECD. 12

Table 8: Forecast comparisons  for spring forecasts for the current year from WIFO, IHS and OECD. 12

Table 9: Contingency table. 17

 

 

[1] Any forward-looking action under uncertainty by economic policy, firms, consumers, employers or employees is based on expectations and thereby, implicitly or explicitly, upon a forecast of economic developments. The reaction of economic agents to changes in future prospects can be rather different. Thus, the prediction of a business cycle downturn may induce private agents to a more cautious consumption and investment behaviour, and wage settlements may turn out more moderate. The reaction by the government will depend on the goals of economic policy. From the range of options, just a few shall be mentioned here: deliberate counter-cyclical action or - less pro-active - the operation of automatic stabilisers will drive up public expenditure and, with tax revenues falling at the same time, the budget deficit will increase. This will, by tendency, lean against the downward trend. If, on the other hand, fiscal policy wants to keep the government balance unchanged, it will try to compensate for the revenue losses by cutting expenditure. In this way, the downturn will be reinforced, at least in the short run[a]. Provided that the reactions by the different economic agents do not offset each other, the example illustrates the effect a (credible) forecast may have: depending on the reaction of economic agents, the forecast may be either self-fulfilling (or self-reinforcing) or self-destroying.

[2] This renders the evaluation of the accuracy of a forecast difficult. The most straightforward way to proceed would be to compare the forecast with the actual economic outcome (the realisation) and to regard the forecast as good if the degree of congruence is high. In case that the forecast has no effect on the outcome - like the weather not being influenced by the weather forecast - such an approach is perfectly adequate. However, the situation is somewhat different for economic forecasts: in the extreme case, the forecast may have such a strong influence on economic agents that the outcome is based on entirely different conditions and the ex-post comparison would show little or no congruence between forecast and reality. Still, such a forecast should not be called bad or meaningless, as the alternative course of events could not have been anticipated[b].

[3] Moreover, in judging the quality of a forecast one has to bear in mind that we are usually dealing with conditional forecasts that are valid only on certain assumptions[c]. For a small open economy like Austria such assumptions usually refer to international economic developments (economic trends in the EU and the USA, exchange rates, prices of raw materials, etc.) as well as to policy conditions in the EU and in Austria (such as policy-controlled interest rates, tax rates, etc.). The very assessment of international business conditions is one of the major sources of uncertainty for forecasts in small countries.

[4] The result of a forecast also depends on the (implicit or explicit) structure of the underlying macro-economic explanatory context (econometric model). If the forecast is validated by actual developments, although the underlying assumptions were false, the forecast ought to be regarded as incorrect even if the forecasting error, defined as realisation minus forecast, is zero. Conversely, an incorrect forecast derived from correct assumptions points to deficiencies in the underlying explanatory model.

[5] Furthermore, the overall purpose of a forecast is not clearly defined, as different economic policymakers and forecasters may be guided by different interests. This problem area is reflected by the selection of a "loss function", i.e., the evaluation criterion for deviations of the forecast from realisations. For "rational" or "pure" forecasters it is important to produce a largely "correct" forecast coming as close as possible to the actual value of the projected variable. Upward and downward errors are given equal weight, implying a symmetric loss function.

[6] However, for economic policymakers and their advisers upward or downward deviations may weigh differently. Thus, it may be that the non-recognition of an economic downturn or even more a recession carries a higher economic and political cost for a government than the incorrect perception of an upturn. If policy wants to react to a greater extent to a projected slower growth than to an announced higher rate of growth, then cautious economic forecasts will possibly be more appreciated than unbiased ones (Aiginger, 1979). This would point to an asymmetric loss function carrying a higher "sanction" for the over-estimation of the actual outcome (i.e., a negative forecasting error)[d].

[7] For monetary policy mainly oriented towards price stability also an asymmetric loss function is plausible. However, with regard to real GDP (or potential output) it would be skewed rather to the opposite side of the one referred to above: the under-estimation of real GDP growth (and of inflation) would weigh more heavily in the loss function and carry a higher "sanction".

[8] In the situations described, the asymmetric loss functions of the different users of the forecasts would have to be taken as the measure in assessing the quality of the forecasts. However, as the examples illustrate, policymakers may have (very) different loss functions for each of the key macro-economic variables; thus, the debate would shift from the accuracy of the forecasts towards the shape of the appropriate loss function.

[9] Moreover, little is known about the statistical properties of forecast errors evaluated on the basis of asymmetric loss functions, and comparisons with previous research on the subject are not straightforward[e]. On the evaluation of forecasts on the basis of symmetric loss functions, however, there is an extensive literature that also allows comparisons over time and across countries.

[10] For these reasons, the measures of accuracy and statistical tests used in the present analysis assume a symmetric loss function. The arguments mentioned above underline that there is no absolute measure of forecast accuracy. It is nevertheless meaningful to investigate the relative properties of forecasts over time, across different variables and between different authors of forecasts.

[11] The following paper compares for the 1980s and 1990s the quality of the forecasts of the two leading Austrian economic research institutes - the Austrian Institute of Economic Research (WIFO) and the Institute for Advanced Studies (IHS) - with the OECD forecasts for Austria. The present analysis covers three key macroeconomic indicators and tries to find out whether the forecast errors exhibit any systematic bias. In addition, the differences in forecasting accuracy between WIFO and IHS on the one hand and the OECD on the other, are tested for their statistical significance[f].

[12] Several investigations of the forecasting performance of WIFO and IHS have been carried out in the past (see Baumgartner, 2002, for a review), with some studies also including the OECD forecasts for Austria (Kramer, 1980, Wörz, 1994, 1995, Öller - Barot, 2000). The quality of the OECD forecasts has been the subject of a great number of studies, particularly for the G-7 countries (e.g., Ash - Smyth - Heravi, 1997, 1998, Koutsogeorgopoulou, 2000, Öller - Barot, 2000, Batchelor, 2001, Blix et al., 2001).

[13] With regard to comparisons of its own projections with forecasts from the "private sector", the OECD has repeatedly pointed out that the OECD projections are serving other purposes, namely "to discuss risks and deliver policy messages that are deemed to be useful to policymakers in member countries" (Lenain, 2002). However, this statement is valid to at least an equal extent for the forecasts of the two Austrian economic research institutes: thus, the institutes' forecasts usually do not include projections of policy reactions, but rather formulate recommendations for policy on the basis of the given economic policy framework.

 

The data

[14] The analysis covers the annual forecasts of WIFO, IHS and the OECD for the Austrian economy from 1983 to 1999 for three widely watched macro-economic variables:

·          the growth rate of real GDP,

·          the rate of unemployment (using the national definition for WIFO and IHS, and the OECD definition for the OECD forecast)[g],

·          the rate of inflation based on the private consumption deflator.

[15] The two Austrian economic research institutes present macro-economic forecasts four times a year (in March, June, September and December), each time for the current and the following year[h]. The OECD, in its semi-annual "Economic Outlook" issued in June and December, submits economic projections for the current and the next year[i]. The OECD holds the view that for the purpose of comparisons with other forecasts, the "preliminary version" of its "Economic Outlook" should be used that is finalised in May and November of each year (see Lenain, 2002). As these dates come to fall regularly between the forecasting dates of WIFO and IHS, the choice of the forecast issues from the national institutes for the present analysis has the following implications:

·          When using the March and September issues (Tables 1 to 4), the OECD, knowing the institutes' forecasts, has an information advantage. If, in this case, the forecasts from WIFO and IHS prove more accurate than those from the OECD, they have to be regarded as clearly better, since they have been made on the basis of a comparatively smaller information set.

·          When, alternatively, using the June and December issues (Tables 5 to 8), it is the Austrian institutes that have an information advantage, since their set of information includes the most recent OECD forecast. Should the OECD forecast turn out more accurate, it is then of better quality, as in this case the OECD is in the less favourable starting position, as far as the information set is concerned.

[16] Thus, in judging the quality of the different forecasts, both alternatives have been used: from the autumn forecasts (September and December for WIFO and IHS) of year t-1 and from the respective spring forecasts (March and June for WIFO and IHS) of year t the projected values for the current year t have been taken[j].

[17] For growth of GDP and the private consumption deflator, the realisation for year t, against which the forecasts are set, are the preliminary national accounts data calculated by WIFO, disposable in March of the following year[k]. This choice most closely corresponds to the state of information at the disposal of the forecasters at the time of their forecast. Later data revisions carried out by Statistics Austria often go back over several years and sometimes also include methodological changes, such as the changeover of the national accounts standards from SNA 68 to ESA 79 and ESA 95 (see in this regard also Baumgartner, 2002). For the rate of unemployment, the realisations taken for comparison with the WIFO and IHS projections (based on the national definition) are those given by the labour market service in March for the preceding year; for the OECD forecasts, the realisations were obtained from the "Economic Outlook" of the December issue of the following year.

 

Methods

[18] The accuracy of the forecasts is assessed on the basis of different criteria which may be divided into five groups[l]:

1.       measures of statistical accuracy with regard to the forecasting error,

2.       comparison with "naïve" forecasts (Theil coefficient of inequality),

3.       tests for statistical significance of the differences between the two institutes' forecasts,

4.       sign tests in order to check the directional forecast accuracy,

5.       statistical tests for the unbiasedness and efficiency of the forecasts.

[19] Usually, the 95 percent level of significance is applied for the statistical tests used. This corresponds to a 5 percent probability of error for the underlying null hypothesis being rejected although being correct. In the tables, the error probability is given by the p value: a p value of 0.05 corresponds to an error probability of 5 percent, meaning that the null hypothesis is rejected (at least at the 95 percent level) if p is smaller than 0.05.

Measures of accuracy

[20] The mean forecast error shows the average deviation of the projected from actual values, pointing to a forecast bias. If the forecast errors - defined as actual minus projected values - are distributed equally upwards and downwards, this figure is (close to) zero. A positive sign indicates that the forecast has a tendency of underestimating actual developments. Whether the forecast is actually biased is found out using the test procedures described below. An "optimal" predictor should, moreover, exhibit a lower variance than the realisations, since the forecast, unlike the actual value, includes no irregular component (see Granger - Newbold, 1977). In this regard, Aiginger (1979, p. 173) also points to a revealed "smoothing tendency": if information is scarce, it seems plausible to expect an average development. The more the future trend is uncertain, the more expectations will keep close to the arithmetic mean. The influence of a smoothing tendency is supported by the fact that it is stronger with forecasts of longer horizon than with short-term ones. The ratio between the standard deviations of the forecasts and the realisation should therefore be smaller than 1 and diminish with increasing forecast horizon.

[21] In calculating the mean forecast error, the positive forecast errors are balanced against the negative ones. Statements going beyond the tendency of the forecast are therefore impossible to make. The mean absolute error (MAE; for all definitions see the methodological annex) and the mean squared error (MSE) can also measure the accuracy of a forecast. Hereby, the absolute values (MAE) - the squares, respectively (MSE) - of the forecast errors are added up and the average is taken. The two measures differ by the weight attributed to the forecast errors entering the calculation: for the MAE, they are weighted linearly, for the MSE by the power of 2; the latter measure therefore gives greater emphasis to large forecast errors. Frequently, instead of the MSE, the root mean square error (RMSE) is shown. This has the advantage that the calculated statistic is of the same dimension as the underlying variable. For all accuracy measures discussed holds: the smaller the value of the calculated statistics, the better the forecast.

[22] Furthermore the mean squared error can furthermore be split in two kinds of inequality proportions, in order to give further evidence on the accuracy of the forecast. The following relations hold:

UM + US + UC = 1,

UM + UR + UD = 1.

[23] A good forecast is characterised by small bias (UM), variance (US), and regression (UR) proportions, while the co-variance (UC) or the distribution (UD) proportions should be close to 1 (see Theil, 1966, 1971, or the methodological annex).

Theil coefficients of inequality

[24] No information on the difficulty of predicting a particular variable enters the calculation of the mean absolute or mean squared error. Yet, variables with a low degree of variation are easier to predict than those subject to wide variations. To this end, the RMSE standardised with the standard deviation of the realisations  is calculated . This measure allows a better comparison of forecasting errors between different variables.

[25] Theil (1966, 1971) developed several measures that also address the problem referred to above; these statistics are standardised to 1 and are therefore easier to interpret (see below). In general, they compare the RMSEs of different forecasts. In our case, the forecasts of the three research institutions are set against two "naïve" forecasting strategies suggested by Theil.

[26] The statistic Theil W sets as "naïve" forecast the hypothesis of "no change in the rate of change": the last known rate of change of the realisation is used for projecting future changes. For example, the "naïve" forecast made in year t would thus assume that real economic growth in the current as well as in the subsequent year equals the rate of last year (known at the time of the forecast).

[27] The "naïve" forecast according to the Theil U criterion hypothesises a "no change in level" situation. E.g., it would thereby assume a rate of real economic growth of zero for both the current and the following year.

[28] For both measures holds that the mean squared errors of the institutions' forecasts are set against the mean squared errors of the "naïve" forecasts. If the Theil W or Theil U statistics are smaller than 1, the forecasts of the institutions are superior to the "naïve" forecasts. A good forecast therefore exhibits for the Theil inequality coefficients values clearly below unity.

Significance tests

[29] The accuracy of two forecasts A and B is also examined using the modified Diebold-Mariano test (DM*) and the non-parametric Wilcoxon signed rank test (W*; see Diebold - Mariano, 1995, Hartung, 1991, Harvey - Leybourne - Newbold, 1997, Mariano, 2002, and the methodological annex). Diebold - Mariano (1995) have developed, for different loss functions, a test for the null hypothesis of "equal forecast accuracy" between two forecast providers. The present analysis uses both the absolute and the squared errors as loss functions to examine whether statistically significant differences between the forecasts (or the measures of forecast accuracy) of WIFO, IHS and the OECD exist.

[30] The loss functions used here belong to the family of symmetric loss functions. Particularly the users of a forecast (e.g., economic policymakers) may rate positive or negative forecast errors differently than the forecasters themselves (see the discussion further above). The test developed by Diebold - Mariano (1995) is applicable also to non-symmetric loss functions. However, a generally accepted asymmetric loss function is not available, at least not at the present stage.

[31] In the tables the values for DM*-AE and W*-AE, or DM*-SE and W*-SE, respectively, show the p values of these tests for the absolute (AE) or squared (SE) loss function, respectively. For values below 0.05, the null hypothesis of equal forecasting accuracy can be rejected at the usual levels of significance, i.e., the differences are statistically significant.

Tests for directional accuracy (sign tests)

[32] For the users of forecasts it is sometimes not the accuracy of a forecast that matters most, but whether at least the direction of future developments has been correctly assessed. Using a non-parametric test (see methodological annex) it is examined whether the direction of change given in a forecast corresponds with the actual course of events. The null hypothesis of the test is stated as follows: the sign of a change in the forecast and the sign of a change of the same variable as actually observed are mutually independent[m]. The ratio of congruence (ER, in the interval [0, 1]) specifies the degree of congruence over the period of analysis. If this ratio is below 0.5, than a forecast by tossing a coin would better predict the direction of actual change. Good forecasts show ratios of congruence close to 1, and the p value should be smaller than 0.05.

Unbiasedness and efficiency

[33] Unbiased forecasts have the same mean as the realisations and thus exhibit a mean forecast error of zero. A standard procedure in this regard is the estimation of the "realisation-forecast" regression equation by Mincer - Zarnowitz (1969),

,

[34] where  denominates the realisation,  the forecast and  the error term. Under the hypothesis of unbiasedness the parameter restrictions  = 0 und  = 1 must be fulfilled simultaneously, and the residual term  corresponds to the forecast error. This hypothesis can be tested by a joint F test. However, Holden - Peel (1990) have shown that this criterion is a sufficient, but not a necessary condition for an unbiased forecast: it is (theoretically) possible to establish unbiased forecasts that with certainty do not withstand that test. They therefore suggest to estimate the equation

[35] and verify the null hypothesis of unbiasedness through the ordinary t test m = 0. For a p value above 0.05, the null hypothesis cannot be rejected (at the usual level of significance).

[36] The hypothesis of efficiency of a forecast assumes that the forecasters make optimal use of all the information at their disposal. This assumption is verified by testing the residuals  from the "realisation-forecast" regression according to Holden - Peel (1990) for first-order autocorrelation through the Durbin-Watson statistic (DW). In addition, the orthogonality between forecast errors and available information is checked via the regression

.

[37] If the forecasters learn from past errors  and use this information to be derived from the forecast errors, then the joint hypothesis  and  should not be rejected on the usual level of significance. The p value for the F test should thus be greater than 0.05. The residuals  are tested for higher-order autocorrelation using the Box-Ljung Q test. The null hypothesis of "no autocorrelation" is rejected for a p value below 0.05.

 

Results

[38] The results of the forecast comparison, as shown in Tables 1 to 8, refer to the forecasting rounds of September and December (projections for next year) and March and June (for the current year), as far as the WIFO and IHS forecasts are concerned; as for the OECD forecasts, they refer to the autumn (for next year) and the spring forecasts (for the current year), respectively. With the exception of the unemployment rate, the correlation among the forecasts is higher than that between the forecasts and realisations. This observation is consistent with earlier findings and those obtained for other countries. It is explained by the fact that the information set at the disposal of the research institutes and the OECD is highly similar. The advantage of the larger amount of information on international economic developments accessible to the OECD is apparently offset by the higher degree of institutional knowledge at the command of the domestic research institutes. However, in projecting the unemployment variable, the better institutional knowledge may be of decisive importance (see below).

[39] The forecasts exhibit lower standard deviations than the actual outcomes, and the standard deviations diminish with the distance of the forecast horizon. For the variables examined, a systematic deviation towards caution is apparent: prospects generally deemed positive, such as growth of real GDP, tend to be projected too low, while those considered undesirable (e.g., unemployment and inflation) are rather over-estimated. This is an indication that forecasters may implicitly face an asymmetric loss function (e.g., Aiginger, 1979). Such a tendency is partly even statistically significant (see the findings on unbiasedness below).

[40] The higher the volatility of a variable, the more difficult it is to predict; nevertheless, for all time series reviewed, the root mean square error (RMSE) is smaller (or only marginally higher) than one standard deviation of the realisations. An exception is the OECD projection of the unemployment rate for which the RMSE is more than twice as high as the standard deviation of the actual outcome. For both forecast horizons, the lowest (standardised) RMSE values have been found for the projections of inflation. Not surprisingly, the RMSE is lower, the shorter the forecast horizon, given the lower degree of uncertainty and thus the risk of error.

[41] A decomposition of the mean square error (MSE) shows that the larger part of the forecast errors is due to the covariance (UC) or regression (UR) proportion, i.e., the components beyond the influence of the forecaster. Exceptions in this regard are the unemployment projections by the OECD and the inflation projections of the IHS, each of which exhibit a (very) large bias proportion.

[42] For some of the variables examined it is possible to compare RMSE values over time, although not all studies refer to the same forecasting dates, forecast horizons and realisations[n]. On this basis it is difficult to assess the evolution of the forecasting errors. It seems, however, that for the RMSE statistics the trend has been slightly declining. The results of Öller - Barot (2000) for the WIFO and OECD projections of growth and inflation for Austria point into the same direction[o]. One should nevertheless bear in mind that as from the mid-1980s cyclical variations were less strong than in the ten years before.

[43] The projections of the three forecasting institutions are of better quality than simple "no change" alternatives, with the exception of the OECD unemployment forecast. The Theil-W measure of inequality for the OECD forecast of the unemployment rate is clearly above 1 for both forecast horizons. A similar result has been obtained by Wörz (1994, 1995). If the OECD were to take greater account of the figures projected by the domestic institutes in May and September that are already known at the time of its own forecast, it could have improved the quality of the latter. For the unemployment rate, the absolute and the squared forecasting errors are smaller for WIFO and IHS than for the OECD, a difference that is also statistically significant. This comes despite the fact that in the given situation the OECD operates from a better information base. The measures of forecast accuracy show, that the unemployment forecasts by WIFO and IHS are superior to those from the OECD.

[44] The assessment is more ambiguous for the growth rate of real GDP and the rate of inflation: the OECD can benefit from a higher level of information and exhibits (slightly) smaller RMSE values as compared with the forecasts of September and March of the two Austrian institutes. However, this result is turned around when the forecasts of December and June are used, thereby giving the information advantage to the Austrian institutes: in this setting, the forecasts by WIFO and IHS prove more accurate. Thus, from the present analysis no firm conclusion can be drawn as to whether the forecasts from the OECD or those from the domestic institutes are of better quality.

[45] As far as the directional accuracy of forecasts for different economic variables is concerned, the forecasted changes for all variables are correct in more than 50 percent of the cases (ER > 0.5) by all three forecasting institutions; yet, the null hypothesis of independence can be rejected unambiguously on the 95 percent level for the shorter forecast horizon only.

[46] As illustrated above, forecasts should on average not deviate from realisations (i.e., be unbiased) and make best use of the available information (i.e., be efficient). The hypothesis of unbiasedness and efficiency cannot be rejected at the 95 percent level of significance for the forecasts of GDP for all three institutions, of unemployment (of the Austrian institutes) and of the inflation forecast of the OECD.

Table 1: Descriptive statistics for autumn forecasts for next year from WIFO, IHS and OECD

WIFO, IHS: September forecasts

 

 

Real GDP growth

Unemployment

Inflation

 

 

Forecast

Realisation

Forecast

Realisation

Forecast

Realisation

 

 

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mean

2.26

2.27

2.23

2.41

2.41

5.68

5.69

4.20

5.62

3.56

2.84

3.11

2.74

2.55

2.55

 

Variance

0.62

0.64

0.50

1.48

1.48

0.65

0.62

0.45

0.49

0.09

1.14

1.14

1.09

1.74

1.74

 

Standard deviation

0.79

0.80

0.71

1.22

1.22

0.81

0.79

0.67

0.70

0.30

1.07

1.07

1.04

1.32

1.32

 

Correlation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

With realisation

0.14

0.01

0.14

 

 

0.71

0.62

0.57

 

 

0.89

0.85

0.89

 

 

 

With WIFO

 

0.88

0.90

 

 

 

0.98

0.46

 

 

 

0.97

0.99

 

 

 

With IHS

 

 

0.84

 

 

 

 

0.55

 

 

 

 

0.96

 

 

 

Realisations: first release by WIFO as of March t+1.

 

 

 

Table 2: Forecast comparisons for autumn forecasts for next year from WIFO, IHS and OECD

WIFO, IHS: September forecasts

 

Real GDP growth

Unemployment

Inflation

 

WIFO

IHS

OECD

WIFO

IHS

OECD

WIFO

IHS

OECD

Accuracy measures

 

 

 

 

 

 

 

 

 

ME

0.15

0.14

0.18

–0.07

–0.08

–0.64

–0.29

–0.56

–0.19

STDR

0.65

0.66

0.58

1.15

1.12

2.22

0.81

0.81

0.79

MAE

1.20

1.20

1.10

0.50

0.53

0.65

0.54

0.66

0.54

MSE

1.86

2.12

1.77

0.34

0.43

0.71

0.46

0.79

0.42

RMSE

1.36

1.45

1.33

0.58

0.66

0.84

0.68

0.89

0.65

UM

0.01

0.01

0.02

0.01

0.01

0.57

0.18

0.40

0.09

US

0.10

0.08

0.14

0.03

0.02

0.19

0.14

0.08

0.18

UC

0.89

0.91

0.84

0.95

0.97

0.24

0.68

0.52

0.73

UR

0.21

0.29

0.16

0.28

0.29

0.34

0.02

0.00

0.04

UD

0.78

0.70

0.82

0.71

0.70

0.09

0.80

0.60

0.87

1.12

1.20

1.10

0.83

0.94

2.80

0.51

0.68

0.49

Theil W

0.72

0.76

0.72

0.89

0.98

1.55

0.43

0.56

0.40

Theil U

0.52

0.55

0.50

0.11

0.12

0.25

0.27

0.36

0.26

 

 

 

 

 

 

 

 

 

 

Test of significance of forecast differentials

 

 

 

 

 

 

 

With WIFO               DM*-AE

 

1.000

0.259

 

0.395

0.217

 

0.102

1.000

                W*-AE

 

0.959

0.420

 

0.081

0.001

 

0.161

0.224

                DM*-SE

 

0.464

0.655

 

0.029

0.045

 

0.088

0.534

                W*-SE

 

0.815

0.568

 

0.086

0.344

 

0.065

0.136

With IHS  DM*-AE

 

 

0.398

 

 

0.312

 

 

0.236

                W*-AE

 

 

0.917

 

 

0.002

 

 

0.153

                DM*-SE

 

 

0.398

 

 

0.132

 

 

0.141

                W*-SE

 

 

0.776

 

 

0.324

 

 

0.082

 

 

 

 

 

 

 

 

 

 

Directional forecast accuracy

 

 

 

 

 

 

 

 

ER

0.69

0.63

0.81

0.50

0.58

0.50

0.81

0.69

0.69

p value

0.131

0.280

0.012

0.248

1.000

1.000

0.006

0.091

0.131

 

 

 

 

 

 

 

 

 

 

Unbiasedness

 

 

 

 

 

 

 

 

 

m

0.150

0.138

0.181

–0.067

0.075

0.638

0.287

0.563

0.194

p value

0.674

0.718

0.602

0.710

0.710

0.003

0.089

0.007

0.246

DW

1.700

1.689

1.987

2.295

2.164

1.848

1.437

1.592

1.697

 

 

 

 

 

 

 

 

 

 

Efficiency

 

 

 

 

 

 

 

 

 

0.026

0.004

0.116

0.006

0.006

0.6

0.254

0.508

0.228

p value

0.946

0.991

0.761

0.974

0.975

0.034

0.182

0.055

0.214

F test for all

0.919

0.926

0.953

0.653

0.747

0.021

0.124

0.02

0.343

BL

0.606

0.767

0.788

0.087

0.187

0.049

0.445

0.763

0.552

Realisations: first release by WIFO as of March t+1. - ME . . . mean error, STDR . . . standard deviation ratio (forecast/realisation), MAE . . . mean absolute error, MSE . . . mean squared error, RMSE . . . root of MSE, UM . . . bias proportion, US . . . variance proportion, UC . . . covariance proportion, UR . . . regression proportion, UD . . . distribution proportion, UM+US+UC = 1 and UM+UR+UD = 1,  . . . ratio between RSME of forecast and standard deviation of realisation, Theil W . . . Theil inequality statistic W, Theil U . . . Theil inequality statistic U; DM* . . . p value for the asymptotically t-distributed modified Diebold-Mariano test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE), W* . . . p value for the asymptotically standard-normally distributed Wilcoxon signed rank test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE); ER . . . congruence between forecasted and actual sign changes of the variable observed, p value . . probability of error for the -distributed test for independence between forecasted and actual sign changes; m . . . constant term of the regression line in the restricted estimation, p value for the t test m=0, DW . . . Durbin-Watson statistic;  . . . constant term of the regression line, p value for the t test , F Test . . . p value for the joint F test  and , BL . . . p value for the Box-Ljung Q test for the null hypothesis "no autocorrelation".

 

Table 3: Descriptive statistics forspring forecasts for the current year from WIFO, IHS and OECD

WIFO, IHS: March forecasts

 

 

Real GDP growth

Unemployment

Inflation

 

 

Forecast

Realisation

Forecast

Realisation

Forecast

Realisation

 

 

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mean

2.09

2.01

1.96

2.38

2.38

5.57

5.61

4.14

5.53

3.60

3.79

3.86

2.76

2.61

2.61

 

Variance

0.95

0.84

1.02

1.40

1.40

0.67

0.68

0.33

0.54

0.10

1.75

1.86

1.48

1.70

1.70

 

Standard deviation

0.98

0.92

1.01

1.18

1.18

0.82

0.83

0.57

0.74

0.32

1.32

1.36

1.22

1.30

1.30

 

Correlation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

With realisation

0.66

0.69

0.71

 

 

0.93

0.90

0.43

 

 

–0.34

–0.41

0.97

 

 

 

With WIFO

 

0.96

0.98

 

 

 

0.99

0.20

 

 

 

0.99

–0.22

 

 

 

With IHS

 

 

0.97

 

 

 

 

0.23

 

 

 

 

–0.28

 

 

 

Realisations: first release by WIFO as of March t+1.

 

 

 

Table 4: Forecast comparisons for spring forecasts for the current year from WIFO, IHS and OECD

WIFO, IHS: March forecasts

 

Real GDP growth

Unemployment

Inflation

 

WIFO

IHS

OECD

WIFO

IHS

OECD

WIFO

IHS

OECD

Accuracy measures

 

 

 

 

 

 

 

 

 

ME

0.29

0.37

0.41

0.04

0.08

0.54

0.20

0.26

0.15

STDR

0.82

0.78

0.85

1.11

1.12

1.77

0.88

0.93

0.93

MAE

0.71

0.70

0.71

0.25

0.29

0.54

0.33

0.30

0.28

MSE

0.91

0.89

0.89

0.09

0.13

0.56

0.19

0.16

0.14

RMSE

0.96

0.95

0.94

0.31

0.36

0.75

0.44

0.40

0.37

UM

0.09

0.15

0.19

0.02

0.05

0.52

0.21

0.45

0.16

US

0.05

0.08

0.03

0.07

0.06

0.11

0.14

0.06

0.06

UC

0.86

0.77

0.77

0.91

0.89

0.38

0.65

0.50

0.78

UR

0.04

0.01

0.03

0.20

0.20

0.33

0.06

0.02

0.01

UD

0.87

0.83

0.78

0.79

0.76

0.15

0.73

0.53

0.83

0.81

0.80

0.80

0.42

0.49

2.32

0.33

0.30

0.28

Theil W

0.66

0.64

0.65

0.82

0.96

2.10

0.42

0.37

0.35

Theil U

0.36

0.36

0.36

0.06

0.06

0.21

0.15

0.14

0.13

 

 

 

 

 

 

 

 

 

 

Test of significance of forecast differentials

 

 

 

 

 

 

 

With WIFO               DM*-AE

 

0.550

0.922

 

0.184

0.109

 

0.176

0.227

                W*-AE

 

0.715

0.714

 

0.037

0.000

 

0.331

0.875

                DM*-SE

 

0.482

0.643

 

0.067

0.178

 

0.083

0.204

                W*-SE

 

0.715

0.428

 

0.034

1.000

 

0.030

0.235

With IHS  DM*-AE

 

 

0.394

 

 

0.189

 

 

0.785

                W*-AE

 

 

0.794

 

 

0.000

 

 

0.937

                DM*-SE

 

 

0.486

 

 

0.222

 

 

0.682

                W*-SE

 

 

0.183

 

 

0.905

 

 

0.937

 

 

 

 

 

 

 

 

 

 

Directional forecast accuracy

 

 

 

 

 

 

 

 

ER

0.81

0.94

0.88

0.67

0.67

0.58

0.88

0.94

0.94

p value

0.012

0.000

0.003

0.700

0.700

0.296

0.003

0.000

0.000

 

 

 

 

 

 

 

 

 

 

Unbiasedness

 

 

 

 

 

 

 

 

 

m

0.250

0.319

0.384

0.042

0.075

0.550

0.206

0.262

0.162

p value

0.312

0.177

0.105

0.672

0.513

0.006

0.063

0.004

0.086

DW

1.647

2.007

1.753

1.919

2.102

2.122

1.230

1.090

1.311

 

 

 

 

 

 

 

 

 

 

Efficiency

 

 

 

 

 

 

 

 

 

0.243

0.331

0.349

0.011

0.049

0.574

0.147

0.143

0.104

p value

0.381

0.238

0.220

0.919

0.698

0.059

0.236

0.190

0.331

F test for all

0.472

0.446

0.282

0.991

0.879

0.054

0.065

0.007

0.129

BL

0.439

0.422

0.290

0.251

0.255

0.708

0.158

0.390

0.129

Realisations: first release by WIFO as of March t+1. - ME . . . mean error, STDR . . . standard deviation ratio (forecast/realisation), MAE . . . mean absolute error, MSE . . . mean squared error, RMSE . . . root of MSE, UM . . . bias proportion, US . . . variance proportion, UC . . . covariance proportion, UR . . . regression proportion, UD . . . distribution proportion, UM+US+UC = 1 and UM+UR+UD = 1,  . . . ratio between RSME of forecast and standard deviation of realisation, Theil W . . . Theil inequality statistic W, Theil U . . . Theil inequality statistic U; DM* . . . p value for the asymptotically t-distributed modified Diebold-Mariano test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE), W* . . . p value for the asymptotically standard-normally distributed Wilcoxon signed rank test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE); ER . . . congruence between forecasted and actual sign changes of the variable observed, p value . . probability of error for the -distributed test for independence between forecasted and actual sign changes; m . . . constant term of the regression line in the restricted estimation, p value for the t test m=0, DW . . . Durbin-Watson statistic;  . . . constant term of the regression line, p value for the t test , F Test . . . p value for the joint F test  and , BL . . . p value for the Box-Ljung Q test for the null hypothesis "no autocorrelation".

 

Table 5: Descriptive statistics for autumn forecasts for next year from WIFO, IHS and OECD

WIFO, IHS: December forecasts

 

 

Real GDP growth

Unemployment

Inflation

 

 

Forecast

Realisation

Forecast

Realisation

Forecast

Realisation

 

 

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mean

2.22

2.16

2.23

2.41

2.41

5.69

5.68

4.20

5.62

3.56

2.90

2.92

2.74

2.54

2.55

 

Variance

0.55

0.51

0.50

1.48

1.48

0.64

0.64

0.45

0.49

0.09

1.18

1.21

1.09

1.60

1.74

 

Standard deviation

0.74

0.71

0.71

1.22

1.22

0.80

0.80

0.67

0.70

0.30

1.08

1.10

1.04

1.26

1.32

 

Correlation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

With realisation

0.32

0.37

0.14

 

 

0.79

0.78

0.57

 

 

0.92

0.94

0.89

 

 

 

With WIFO

 

0.92

0.89

 

 

 

0.99

0.41

 

 

 

0.98

0.98

 

 

 

With IHS

 

 

0.90

 

 

 

 

0.47

 

 

 

 

0.97

 

 

 

Realisations: first release by WIFO as of March t+1.

 

 

 

Table 6: Forecast comparisons for autumn forecasts for next year from WIFO, IHS and OECD

WIFO, IHS: December forecasts

 

Real GDP growth

Unemployment

Inflation

 

WIFO

IHS

OECD

WIFO

IHS

OECD

WIFO

IHS

OECD

Accuracy measures

 

 

 

 

 

 

 

 

 

ME

0.19

0.24

0.18

–0.08

–0.06

–0.64

–0.36

–0.38

–0.19

STDR

0.61

0.59

0.58

1.14

1.14

2.22

0.86

0.87

0.79

MAE

0.99

0.99

1.10

0.41

0.39

0.65

0.51

0.46

0.54

MSE

1.49

1.40

1.77

0.25

0.25

0.71

0.38

0.35

0.42

RMSE

1.22

1.18

1.33

0.50

0.50

0.84

0.62

0.59

0.65

UM

0.02

0.04

0.02

0.02

0.01

0.57

0.34

0.42

0.09

US

0.15

0.18

0.14

0.04

0.04

0.19

0.08

0.08

0.18

UC

0.83

0.78

0.84

0.94

0.95

0.24

0.57

0.51

0.73

UR

0.09

0.05

0.16

0.24

0.25

0.34

0.02

0.02

0.01

UD

0.89

0.91

0.82

0.74

0.74

0.09

0.64

0.56

0.87

1.01

0.97

1.10

0.72

0.72

2.80

0.49

0.47

0.49

Theil W

0.64

0.64

0.72

0.81

0.79

1.55

0.39

0.43

0.40

Theil U

0.46

0.45

0.50

0.09

0.09

0.25

0.25

0.24

0.26

 

 

 

 

 

 

 

 

 

 

Test of significance of forecast differentials

 

 

 

 

 

 

 

With WIFO               DM*-AE

 

0.888

0.058

 

0.536

0.063

 

0.244

0.425

                W*-AE

 

0.979

0.349

 

0.048

0.000

 

0.250

0.917

                DM*-SE

 

0.001

0.089

 

0.952

0.030

 

0.504

0.327

                W*-SE

 

0.856

0.204

 

0.316

0.782

 

0.600

0.716

With IHS  DM*-AE

 

 

0.211

 

 

0.049

 

 

0.232

                W*-AE

 

 

0.243

 

 

0.000

 

 

0.364

                DM*-SE

 

 

0.057

 

 

0.029

 

 

0.300

                W*-SE

 

 

0.055

 

 

0.843

 

 

0.533

 

 

 

 

 

 

 

 

 

 

Directional forecast accuracy

 

 

 

 

 

 

 

 

ER

0.81

0.75

0.81

0.67

0.58

0.50

0.75

0.69

0.69

p value

0.012

0.049

0.012

0.700

1.000

1.000

0.016

0.037

0.131

 

 

 

 

 

 

 

 

 

 

Unbiasedness

 

 

 

 

 

 

 

 

 

m

0.188

0.244

0.181

–0.075

–0.058

–0.638

–0.363

–0.381

–0.194

p value

0.557

0.428

0.602

0.626

0.707

0.003

0.014

0.005

0.246

DW

1.961

1.873

1.987

2.258

2.040

1.848

1.503

1.433

1.697

 

 

 

 

 

 

 

 

 

 

Efficiency

 

 

 

 

 

 

 

 

 

0.151

0.169

0.116

–0.039

–0.001

0.600

–0.337

–0.321

–0.228

p value

0.672

0.619

0.761

0.812

0.996

0.034

0.057

0.057

0.214

F test for all

0.905

0.850

0.953

0.791

0.922

0.021

0.024

0.011

0.343

BL

0.659

0.583

0.788

0.066

0.064

0.049

0.391

0.181

0.552

Realisations: first release by WIFO as of March t+1. - ME . . . mean error, STDR . . . standard deviation ratio (forecast/realisation), MAE . . . mean absolute error, MSE . . . mean squared error, RMSE . . . root of MSE, UM . . . bias proportion, US . . . variance proportion, UC . . . covariance proportion, UR . . . regression proportion, UD . . . distribution proportion, UM+US+UC = 1 and UM+UR+UD = 1,  . . . ratio between RSME of forecast and standard deviation of realisation, Theil W . . . Theil inequality statistic W, Theil U . . . Theil inequality statistic U; DM* . . . p value for the asymptotically t-distributed modified Diebold-Mariano test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE), W* . . . p value for the asymptotically standard-normally distributed Wilcoxon signed rank test for the loss functions on the basis of the absolute errors (AE) and the squared errors (SE); ER . . . congruence between forecasted and actual sign changes of the variable observed, p value . . probability of error for the -distributed test for independence between forecasted and actual sign changes; m . . . constant term of the regression line in the restricted estimation, p value for the t test m=0, DW . . . Durbin-Watson statistic;  . . . constant term of the regression line, p value for the t test , F Test . . . p value for the joint F test  and , BL . . . p value for the Box-Ljung Q test for the null hypothesis "no autocorrelation".

 

Table 7: Descriptive statistics for spring forecasts for the current year from WIFO, IHS and OECD

WIFO, IHS: June forecasts

 

 

Real GDP growth

Unemployment

Inflation

 

 

Forecast

Realisation

Forecast

Realisation

Forecast

Realisation

 

 

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

WIFO

IHS

OECD

WIFO, IHS

OECD

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mean

2.21

2.09

1.96

2.38

2.38

5.54

5.58

4.14

5.53

3.60

3.75

3.79

2.76

3.70

2.61

 

Variance

1.45

1.38

1.02

1.40

1.40

0.57

0.58

0.33

0.54

0.10

2.14

2.54

1.48

2.26

1.70

 

Standard deviation

1.20

1.17

1.01

1.18

1.18

0.76

0.76

0.57

0.74

0.32

1.46

1.59

1.22

1.50

1.30

 

Correlation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

With realisation

0.85

0.81

0.71

 

 

0.97

0.96

0.43

 

 

0.99

0.98

0.97

 

 

 

With WIFO

 

0.97

0.95

 

 

 

0.99

0.17

 

 

 

0.99

0.18

 

 

 

With IHS

 

 

0.96

 

 

 

 

0.23

 

 

 

 

0.25

 

 

 

Realisations: first release by WIFO as of March t+1.

 

 

 

Table 8: Forecast comparisons  for spring forecasts for the current year from WIFO, IHS and OECD

WIFO, IHS: June forecasts

 

Real GDP growth

Unemployment

Inflation

 

WIFO

IHS

OECD

WIFO

IHS

OECD

WIFO

IHS

OECD

Accuracy measures

 

 

 

 

 

 

 

 

 

ME

0.16

0.29

0.41

–0.01

–0.05

–0.54

–0.05

–0.09

–0.15

STDR

1.02

0.99

0.85

1.03

1.04

1.77

0.97

1.06

0.93

MAE

0.51

0.61

0.71

0.16

0.20

0.54

0.16

0.26

0.28

MSE

0.47

0.62

0.89

0.03

0.05

0.56

0.05

0.11

0.14

RMSE

0.68

0.79

0.94

0.18

0.23

0.75

0.22

0.33

0.37

UM

0.06

0.13

0.19

0.00

0.04

0.52

0.05

0.08

0.16

US

0.00

0.00

0.03

0.02

0.02

0.11

0.03

0.08

0.06

UC

0.94

0.87

0.77

0.98

0.94

0.38

0.92

0.84

0.78

UR

0.09

0.08

0.03

0.06

0.07

0.33

0.01

0.13

0.01

UD

0.86

0.79

0.78

0.94

0.89

0.15

0.94

0.78

0.83

0.58

0.67

0.80

0.24

0.31

2.32

0.15

0.22

0.28

Theil W

0.46

0.52

0.65

0.47

0.59

2.10

0.16

0.24

0.35

Theil U

0.26

0.30

0.36

0.03

0.04

0.21

0.05

0.08

0.13