Evaluating Technical Analysis: Hedge Fund Applicant #2

Like anyone who trades, I use technical analysis.  My funds have strategies based upon technical criteria, and my fundamentally-based trades use technical indicators to help with entries and exits.  Any professional looks at charts of holdings and indices all of the time.  I have read many of the leading works on the subject.  As regular readers know, a former professor’s style is to read these works in some detail with a lot of notes.  I look at various indicators that seem useful.  Perhaps more on that later, but the point is that I am open-minded and receptive to this approach.

If you did not read earlier posts in this series, please take a look here.  I did not hire applicant #2, nor have I hired any pure technical analyst who has applied.  The problem:

I do not know how to evaluate their work.

I read daily the recommendations of five different technical analysts, and have done so for years.  I like Rev. Shark on TheStreet.com’s Real Money site, but part of what I like is his commentary and apparent feel for the current market sentiment.  In a recent "technicals versus fundamentals" debate on the site, something that breaks out in discussions among the participants quite regularly, Rev. Shark insisted that any test had to use a "real"’ technical analyst, not just someone who looks at charts.  That would probably be me!  I could not tell you which of the technical analysts I follow have done the best over time.  That would take a lot of tracking, so I have only impressions about what seems to have helped me.

Here is the assessment of Applicant #2:

System

He looks at setups, using his criteria, and gives me specific recommendations.

Characteristics

Frequency of trading:  high
Turnover:  high
Ease of use:  excellent
Risk reward:  difficult to evaluate
Drawdown:  difficult to evaluate

Overall Evaluation

I cannot evaulate the candidate’s work because he is showing me a portfolio of past winners.  If he did not have them, he would not be applying.  Even if his record is excellent, I am being "Fooled by Randomness" because the applicants without such a record have moved on to a new occupation.  He might be excellent, but I cannot tell.

I am also bothered by an element of subjectivity.  When a signal has triggered, there is an issue about when to get out of the position.  It makes a big difference.  Partly this is a choice of stops, a topic covered nicely by Brett Steenbarger.  Consistent with his observations, I have found that the choice of stop rules dramatically affects performance, and most stops take you out of good positions.  This seems counter to the prevailing wisdom.

I am also bothered by the applicant reviewing his methods and changing them to find the best 75% of the trades.  This is a constant danger in backtesting.  Has he really learned something, or just improved his fit to past data?

What about the trades he gave me to follow?  What if they work?  Have I learned something meaningful?  Even if I track the his trades for months, I am only seeing one market environment, one part of the cycle, one set of market moods.

How does one determine that a technical analyst adds real value?  I welcome any comments or suggestions on this question.