Cherry Picking in Data Analysis

There is a type of research that is especially dangerous for individual investors.  The researcher takes current data and makes a statement like one of the following:

  • If you avoided the ten worst trading days over the last five years….
  • If you missed the ten best trading days over the last five years….
  • If you threw out the ten strongest earnings reports….
  • If you threw out the ten worst earnings reports…
  • If you avoided the last five recessions….
  • If you threw out the companies with the strongest stock performance…
  • If you threw out the companies with the weakest stock performance…

This type of analysis is pretty easy to do for anyone with a computer and a data set.

Many trading systems do backfitting, avoiding the recessions or downturns.  It is easy to find an indicator that gave a definitive signal when looking at past data.  The problem is that such systems, lacking rigorous development of hypotheses, failing to use out-of-sample data, and willing to accept an insufficient number of cases, usually produce post-diction rather than robust predictive models.

The Current Example

Barry Ritholtz at The Big Picture highlights an interesting situation posited by Mike Panzer.  Mike reports that a small number of stocks have powered the Nasdaq higher.  Mike  concludes as follows:

Finally, 13 out of 100 stocks — 13% — are responsible for two-thirds of the overall advance.

this heavy lifting by a small number of shares does not mean the index
can’t go higher still, history suggests rallies that lack widespread
participation sometimes lack long-term staying power.

What to Conclude?

We are troubled by this facile conclusion, which was reported without comment by various pundits.   We enthusiastically endorse the more analytical questions  raised by Barry:

What might this mean?

Are Technicals Waning as a Positive
Influence? I’m not exactly sure — What I’d like to see is how past
rallies have moved forward in terms of leadership.

Is it unusual to have 13 stocks in the
NDX’s 100 account for 67% of the aggregate advance? Is this unusually
narrow? When has this occurred, early or late in a run?

I don’t know the answers to these, but I am curious . . . 

In scholarly research one would not start with a conclusion, but with a hypothesis.  It might go something like this…….

When fewer than fifteen stocks in the Nasdaq account for two-thirds of recent gains of X percentage, we define the leadership as "narrow."  Looking back on the Y number of cases fitting this description, we note that stock returns over the following Z days were as follows (table included).

Even with such a statement there are issues about how the definition of narrowness was determined, whether there were enough cases, and whether the researcher really generated the proposition and then tested or just used all of the data to identify key parameters.  This approach would at least provide some comparison.  Ideally, the relevant data would be provided to other researchers to check the selection of parameters.

In the absence of such data, one is left with questions.  The indices are weighted by capitalization.  What is the performance of the rest of the index?  What is the market cap percentage of the top stocks?  Markets often look for leadership.  Is a gain by some of the top stocks indicative of success or of failure?  If other stocks are lagging, is it possible that the market will later show strength in the laggards?

A Final Word

Like Barry, we do not pretend to know the answers to these questions.  In our effort at "A Dash" to raise the standard of Street research, we try to highlight certain problems.  Often these are conclusions that are readily embraced by those who seek support for what they already believe.

Some research is driven by conclusions, not by hypotheses.  It is not scientific.

As usual, the discriminating investor, insisting upon strong research methods,  can gain a contrarian advantage.

You may also like


  • muckdog July 3, 2007  

    Hey, what if you just buy at the bottoms and sell at the tops? How does that work?
    The argument of using a market-cap ETF vs a non-market-cap ETF goes on. Whatever is most popular will probably be the underperformer going forward!
    What we need is a working crystal ball. I keep searching EBAY for one. They’re very rare and somebody keeps putting in a bid $1 higher than mine at the last minute and I lose out…

  • Bill a.k.a. NO DooDahs July 3, 2007  

    Extreme breadth of market participation could imply that traders are chasing any and all stocks, including those of companies with poor fundamentals, transforming an index where all members are rising into a “bull trap.”
    Therefore, limited participation could imply that only the strongest sectors and companies are leading, which is as it should be, hypothetically.
    Of course, one needs a comprehensive analysis to determine whether market breadth means anything in this context.