Evaluating Predictions

Regular readers of "A Dash" know that we have a strong interest in predictions — those based upon models, but also others.

Many investment heroes made a prediction without a time frame.  Our guess is that many readers would be surprised at a comparison of results between these forecasters and those who took the opposite viewpoint.  But that is a subject for another day.

We also like looking at sports predictions.  This takes the investor out of the normal environment.  In sports there are many more instances.  One gets into "the long run" more quickly.

Briefly put, analyzing predictions from sports helps us to think objectively about economic and investment forecasters.

An Interesting Example

In a recent baseball game, meaningless in the overall standings, one of the announcers for Seattle made a very specific prediction.  Take a look at the video.

Sports Videos, News, Blogs

The initial reaction is — Wow!

Evaluation

There are two very distinct ways of looking at this prediction.  The first is inspired by Fooled by Randomness, a book highly recommended, and one that we sent to many clients.

Taking this approach, we ask how many similar predictions are made each season.  Many broadcasting teams (all of them?) have a "pick to click" question.  The skeptical analysis would suggest that there were thousands of predictions, mostly inaccurate and ignored.

If a prediction works, then and only then does it get publicity.  We have no idea how many similar predictions never saw the light of day.

On the other hand, this particular prediction has a very high level of specificity.  The announcer predicted which at bat, what the count would be, that the pitch would be a fastball, that the hitter would connect for a home run, that it would be to left field, and that it would reach the second deck.

This is not just a run-of-the-mill pick to click.

Investment Take

We are interested in comments on this example.  It seems dissimilar to most investing predictions because of the specificity.

Most investment guru's follow the key commandment:

Do not make a prediction that has both a target and a time frame!

You may also like

3 comments

  • Craig Biddle October 7, 2009  

    This prediction is less random than it appears. If Mike Blowers, for example, knew that the opposing pitcher was prone to wildness, and that he liked to throw fastballs when behind in the count, and that the batter in question had good power to left center field and was a good fastball hitter, then all of the seemingly unrelated elements of the specific prediction are, in fact, related.

  • Jeff Miller October 7, 2009  

    Craig — I discussed this with a friend who is both a baseball and probability expert. I suggested that he view it as a conditional probability question, assuming the home run. After all, without the homer, we would never have heard about it. I made arguments similar to yours about the possible correlation in the other elements.
    So I agree with you. He put it all together and told me that it would still take at least ten years of predicting 162 games a year, and probably many more years, to get such a success. Perhaps he will post his analysis.
    I admire the way you are approaching the problem. If only we held the investment prognosticators to such high standards.
    Jeff

  • Danny October 8, 2009  

    Jeff
    A friend of mine and I worked through the numbers and came it out with about 1 in 10000. That is obviously based on several guesses, but some of the ‘factors’ are known, or could be easily known with good data.
    Danny