Forecast-Based Model Selection in the Presence of Structural Breaks

By Todd E. Clark and Michael W. McCracken
August 2002 
RWP 02-05
Research Division 
Federal Reserve Bank of Kansas City 

Abstract

       This paper presents analytical, Monte Carlo, and empirical evidence on the effects of structural breaks on tests for equal forecast accuracy and forecast encompassing. The forecasts are generated from two parametric, linear models that are nested under the null. The alternative hypotheses allow a causal relationship that is subject to breaks during the sample. With this framework, we show that in-sample explanatory power is readily found because the usual F-test will indicate causality if it existed for any portion of the sample. Out-of-sample predictive power can be harder to find because the results of out-of-sample tests are highly dependent on the timing of the predictive ability. Moreover, out-of-sample predictive power is harder to find with some tests than with others: the power of F-type tests of equal forecast accuracy and encompassing often dominates that of the more commonly-used t-type alternatives. Overall, out-of-sample tests are effective at revealing whether one variable has predictive power for another at the end of the sample. Based on these results and additional evidence from two empirical applications, we conclude that structural breaks can explain why researchers often find evidence of in-sample, but not out-of-sample, predictive content.

Keywords: power, structural breaks, forecast evaluation, model selection

JEL Codes: C53, C12, C52

Todd E. Clark is is an assistant vice president and economist at the Federal Reserve Bank of Kansas City. Michael W. McCracken is an assistant professor of economics at the University of Missouri. The authors gratefully acknowledge the helpful comments of: Gregory Chow, David Rapach, Norman Swanson, and Jonathan Wright; seminar participants at Emory University, the University of Missouri, and the Federal Reserve Bank of Kansas City; and participants at the 2001 International Symposium on Forecasting, 2001 Midwest Econometrics Group Meetings, Winter 2002 Econometric Society Meetings, Federal Reserve System Committee on Macroeconomics, 2002 WEA Meetings, and 2002 NBER Summer Institute. Jushan Bai kindly provided computer programs for estimating the Bai-Perron break tests. A significant portion of this paper was completed while McCracken was employed at Louisiana State University; financial support is gratefully acknowledged. The views expressed are those of the authors and do not represent those of the Federal Reserve Bank of Kansas City or the Federal Reserve System.

Clark e-mail:  todd.e.clark@kc.frb.org
McCracken e-mail:  mccrackenm@missouri.edu