Identifying Quackery (and Other Mistakes)

At “A Dash” one of our principal missions is identifying the best sources of information on a given topic.  This is a major source of value, if done accurately.  In the investment blogosphere, however, the ratings mostly reflect the predispositions of the reading public.

The Path to Success in Blogging

It is easy to figure out what works.  One needs only to look at the top-ranked blogs and those most followed on Seeking Alpha.  There is an aggressive cult following (a subject for another day) for specific perspectives on the economy, the role of government, the existence of conspiracies, and the stock market.  If one’s objective is high ratings, this is a blueprint for a strong business model.

The Path to Success in Investing

One of our missions is how to profit from this.  It means reading many perspectives and evaluating arguments and data.  As part of our work (eventually a book) we are analyzing various sources, with an emphasis on factual accuracy, freedom from bias, professionalism, and analytical skill.

Do not start with what you believe.  Start with data.

A Case in Point

We offer the interpretation of various polls as a case in point.  This is partly based upon expertise in developing, conducting, and interpreting poll data.  We have done all of these, and most others have not.  Let us consider some experts in action.

Our first expert is Karl Smith, of the University of North Carolina at Chapel Hill, now added to our list of featured blogs.  He has really stepped up his role in blogging, taking an expert public policy perspective on many issues.  He deserves to be widely read.

Smith takes a current poll and does a careful analysis.  The poll purports to show some terrible answers to some obvious questions.  For example, What is the Supreme Law of the Land?  According to the poll, only 28% of Oklahoma students identified the Constitution and some offered up the Monroe Doctrine.

This might be a true result, in a multiple choice test with attractive alternatives.  We once saw a similar survey of undergrads where most thought that the Electoral College was a small liberal arts school in Minnesota!

But Smith delves deeper into the data, which purported to permit free-form answers.  Look at his which is very strong, but here is an example:

Think about this from a student’s perspective. The first question uses the phrase “Supreme Law.” This means that answer has to be something important. The natural thing to do is to answer the most important thing you can think of. From free response I would not be surprised to get answers like “the bible”, “democracy” or “freedom”.  The student might also simply free associate with things that are like Supreme or like Law. Hence, I would not be surprised to get answers like “Federal government”, “Supreme Court”, “Congress” or even “Roe vs. Wade.”

Since none of these answers were given, but Monroe doctrine was. I find this hard to believe unless the poll was multiple choice and the six answers were the only answers a student could give.

He has accurately identified a false note in the poll.

For our second expert we also take note of the fine work from Nate Silver at, now also added to our list of featured sources.  We have occasionally been accused of credentialism in identifying sources, mostly by those thinking that having a PhD in economics is worthless.  Silver is a fine contrary example.  He developed strong expertise in analyzing baseball statistics.  Anyone with statistical expertise can find many applications.  He has done so in election polling and political analysis, useful to anyone analyzing public policy.  We watch his work carefully.

His take on the dubious polling data is summarized as follows:

For me, some of these results don’t pass the smell test. I agree that public schooling in the United States needs to be improved, particularly in the areas of government and citizenship. But only 23 percent of high school students in Oklahoma knew that George Washington was the first President? Really? I have difficulty accepting that claim at face value. In 2008, 68 percent of Oklahoma fifth graders passed the Oklahoma Core Curriculum Social Studies Test. You can read some of the questions on that test beginning on page 50 of this PDF; they’re generally quite a bit more difficult than the ones that Strategic Vision asks. (For instance, “Which was the most profitable export of the Jamestown settlement?” and “Which group would most likely agree with ideas presented in Common Sense?”). So either those smart fifth graders were really forgetful by the time they got to high school, or there’s something very wrong with this poll.

Investment Implications

There is a special skill involved in interpreting data.  Surveys are the source of much of the information that investors and traders try to interpret.  Few are really expert in how the polls are conducted, the statistical significance of results (often not reported by the source), and the reasons and basis for revisions.

It is foolish to identify the source of the poll, and then dismiss the result if one does not like it.

Our example does not have immediate investment implications, and that is intentional.  It is much easier to step away from the economic polling data, looking instead at an obvious error as a case in point.

Then come back to the news of the day.  CNBC featured today a poll with a response rate of 25% of the CFO’s attending a convention.  Most of them thought we were “still in a recession.”  While we are interested in CFO opinion about current corporate plans, none of them are experts at recession dating.  Mostly they are saying that the economy is still well below trend, something we already know.

There is much more to analyzing polling data than looking at a snapshot of current opinion.  Longitudinal analysis is a starting point, something that most of us do when looking at survey series like the ISM data.

You may also like


  • John Sweda October 8, 2009  

    Dr. Miller:
    Enjoyed your most recent post, as always. Your subject of the Oklahoma phone poll was a question-for-question repeat of an Arizona poll conducted by a conservative think tank (the date on the report is 06/30/09):
    The results were very similar. The objective of conducting these surveys is to demonstrate the poor job done by public schools, thus making a case for homeschooling, schools of choice vouchers, the need for civics courses, etc. I suppose this survey obtains the desired results and is intended to make a point.
    The objective of the survey was never purely scientific.

  • Paul in Kansas City October 9, 2009  

    THe blog Accrued Interest (and your old collegue from real MOney) deserves a shout out for his post today which was along the same lines as this post Jeff. He deserves to be linked! Have a great weekend Jeff.