Evaluating Technical Analysis: Hedge Fund Applicant #2

Like anyone who trades, I use technical analysis.  My funds have strategies based upon technical criteria, and my fundamentally-based trades use technical indicators to help with entries and exits.  Any professional looks at charts of holdings and indices all of the time.  I have read many of the leading works on the subject.  As regular readers know, a former professor’s style is to read these works in some detail with a lot of notes.  I look at various indicators that seem useful.  Perhaps more on that later, but the point is that I am open-minded and receptive to this approach.

If you did not read earlier posts in this series, please take a look here.  I did not hire applicant #2, nor have I hired any pure technical analyst who has applied.  The problem:

I do not know how to evaluate their work.

I read daily the recommendations of five different technical analysts, and have done so for years.  I like Rev. Shark on TheStreet.com’s Real Money site, but part of what I like is his commentary and apparent feel for the current market sentiment.  In a recent "technicals versus fundamentals" debate on the site, something that breaks out in discussions among the participants quite regularly, Rev. Shark insisted that any test had to use a "real"’ technical analyst, not just someone who looks at charts.  That would probably be me!  I could not tell you which of the technical analysts I follow have done the best over time.  That would take a lot of tracking, so I have only impressions about what seems to have helped me.

Here is the assessment of Applicant #2:


He looks at setups, using his criteria, and gives me specific recommendations.


Frequency of trading:  high
Turnover:  high
Ease of use:  excellent
Risk reward:  difficult to evaluate
Drawdown:  difficult to evaluate

Overall Evaluation

I cannot evaulate the candidate’s work because he is showing me a portfolio of past winners.  If he did not have them, he would not be applying.  Even if his record is excellent, I am being "Fooled by Randomness" because the applicants without such a record have moved on to a new occupation.  He might be excellent, but I cannot tell.

I am also bothered by an element of subjectivity.  When a signal has triggered, there is an issue about when to get out of the position.  It makes a big difference.  Partly this is a choice of stops, a topic covered nicely by Brett Steenbarger.  Consistent with his observations, I have found that the choice of stop rules dramatically affects performance, and most stops take you out of good positions.  This seems counter to the prevailing wisdom.

I am also bothered by the applicant reviewing his methods and changing them to find the best 75% of the trades.  This is a constant danger in backtesting.  Has he really learned something, or just improved his fit to past data?

What about the trades he gave me to follow?  What if they work?  Have I learned something meaningful?  Even if I track the his trades for months, I am only seeing one market environment, one part of the cycle, one set of market moods.

How does one determine that a technical analyst adds real value?  I welcome any comments or suggestions on this question.

You may also like


  • Dan Mirkin September 19, 2006  

    You make an interesting point about evaluating the technical analyst. Dr. Brett Steenbarger is using my firm’s newly released Odds Maker technology to do just that, evaluate technical trading models to get a sense of what is and is not working currently. We take a different approach to it than TradeStation and some similar products. Our software tracks an unlimited number of symbols versus just one instrument or a small portfolio. It is set up for intra day trading where the time frame can be up to a whole day or as little as 1 minute. We allow the system to trade up to 1700 times in 15 days to see how this pattern performs intra day. It is a nice way to keep one self honest while trading. No reason to be bearish if a bearish set ups don’t give favorable odds and same for bullish.

  • oldprof September 19, 2006  

    Hi Dan,
    Thanks for your comment. As you probably guessed, the ability to do a systematic test of technical analysis is part of what this series is about. It is a vehicle for reviewing all methods, as well as raising the questions about how one does that. I look forward to doing some tests with your software.