A Costly Error in Analysis

There is a serious analytical mistake gaining increasing popularity.  Once you are aware of it, you will see it several times every week.

An Easy Example

As we often do at "A Dash," let us step away from the world of investing.  Many important ideas can be more easily understood from a completely different perspective.  Here is the situation.

In our Illinois District for State Representative, there was an open seat.  It was a hotly contested race.  The GOP candidate had a record of service on the School Board and the City Council, but this was her first effort at a bigger office.  Her Democratic opponent was a long-time teacher, and a member of homeowner groups.  This was a hot contest in our community, which is traditionally Republican, but has growing support for Democrats.  The Democratic candidate had plenty of financial help and the Obama coattails.

A key issue in the campaign was negative advertising.  The Dem candidate attacked her opponent on abortion rights.  The expensive campaign material, delivered to our house several times, featured a coat-hanger on the front.  There were other negative ads, but you get the idea.

When the election results were in, the GOP candidate won by fewer than 700 votes out of 50,000 cast — a real squeaker.

Her media reaction was as follows:  "This shows that negative campaigning does not work in our district."

Indeed!  Our own assessment was that the intense, heavily-financed negative campaign is what made it close.  Without the attacks, the GOP candidate, who had better visibility, better credentials, and a better base, would have won in a walk.

What Comparison is Right?

The problem is the difficulty in determining what analysts call "the counterfactual" situation.  Here is how we described it in a past article, analyzing the Greenspan Legacy:

In reviewing public policy decisions historians or analysts often use a concept called the counterfactual.  This approach is designed to thwart the kind of thinking that many know more commonly as "Monday morning quarterbacking."  What would have happened if Truman had not approved dropping the atomic bomb?  What if Bush had not invaded Iraq?  We know what did happen, but not what would have happened.

Current Applications

Here are two very prominent current examples of this error:

  • The stimulus package did not do any good.  The economy still declined.
  • The TARP program has not helped.  AIG and other companies still need more help.  Nothing is having any effect.

There are many other examples, and we invite contributions to the list.

You may also like


  • Juan November 12, 2008  

    What would have happened if, in reaction to the ending of the postwar ‘Golden Age’, developed countries had not turned to ‘the market knows best’ neoliberal policies?

  • Jeff November 12, 2008  

    Juan — An excellent example. It is the right way to think about many problems.

  • Anon November 13, 2008  

    I’m also seeing a lot of post hoc fallacy.

  • Mike O'Neal November 15, 2008  

    Hey Jeff, a funny AP headline today, 11/15, at Yahoo Finance–“Retail Sales Sink as Dems Seek Auto Industry Loans.” Both true, but wholly unrelated events.

  • Jeff November 16, 2008  

    Thanks, Mike.
    Another good example. If you start looking for these, you will see them every day.