web analytics

Has Your Smart Beta ETF Gone Fishing?

Posted on: 03.11.2016

As the famous adage states, “hindsight is always 20/20.” This truth forms the basis of a dangerous method to Smart Beta ETF construction: factor fishing. While understanding the past is simple investors should remember that history doesn’t always project the future. In this post, we’ll look at the problem of backtest fitting and how it can lead to a breakdown in the performance of a Smart Beta ETF.

In The Journal of Investment Management, Authors David Bailey, Jonathan Borwein, and Marcos dePrado summarizes the problem with the backtesting methodology. They explain, “Models and strategies suffering from over fitting typically target the specific idiosyncrasies of a limited dataset, rather than any general behavior, and, as a result, often perform erratically when presented with new data.” Trying to replicate the nuances of the past rarely works. Instead, Smart Beta ETF strategies should be kept simple. A backtest strategy looks good on paper but will often fail to deliver the results of the past.

The research above implemented a test using real market data and sophisticated computer algorithms. In doing so, they were able to “produce a stock portfolio that achieves any desired performance profile, based on backtest (in-sample) data.” Their researched showed that as the number of backtesting trials increase so do the instances of false positives. They explain, “any Sharpe ratio is attainable given enough trials.” Imagine spinning a roulette wheel hundreds of times. On each spin, you lose. You keep spinning until finally, on the 300th spin, you win. You then declare that this win can be attributed to the specific, nuanced drop of the ball and flick of the wrist when turning the wheel. From this point forward using these same details will provide identical winning results. Of course, this is nonsense. However, this flawed thinking is not dissimilar from the perils of backtesting.

Other studies have reached conclusions similar to those in The Journal of Investment Management. The European academic think tank, EDHEC Risk Institute published a paper on The Robustness of Smart Beta Strategies. In their review of current methodologies, they determined that “The good idea of factor investing should not be transformed into factor fishing and data mining.” As Smart Beta strategies have increased in popularity, the competition for superior returns has heated up. This competition, in part, explains the unconventional and unsuccessful methods for reverse engineering results. The authors of the paper continue, “lack of relative robustness arises mainly from data mining and non-robust weighting methodologies, while the lack of absolute robustness comes from undiversified factor exposures.” Any investor interested in a Smart Beta ETF must equip themselves with an understanding of the methods used to select the portfolio. Be wary of a prospectus that points to hypothetical backtesting when promising future returns.

Corey Philip

Corey Philip is the founder of RealSmartBeta.com. His focus is on expanding investor knowledge of Smart Beta ETFs and quantitative investing. Learn more about Corey in the 'ABOUT' section of this website.

Discuss with Corey Philip in the comments section below.
Back to Top