Obstacles to comparing different systems

It's tough enough to answer these questions when you are trying to measure the accuracy of a single set of projections. When you open things up and begin to look at multiple prognosticators, then there are even more issues to address.

The number of published projections that appear in print and online has been rising annually, and with them, expectations, questions and unbearable hype. How can there be equivalent credibility with so many different sets of numbers?

I've been asked to prove my prognosticating prowess more often than ever before. There have also been several recent analyses published that compare the Baseball Forecaster and Baseball HQ numbers to those of other touts, but the same thing happens time and time again:

1. We never finish first.
2. The purveyor of the study always does.

Is that a wonder? How can there be so many different “objective analyses” out there, and all of them so allegedly accurate?

Peter “Ask Rotoman” Kreutzer from mlb.com has this take: “Someone who tries to sell you projections that are “much better” than any others is bulls****ing you. The important thing for you as a consumer to understand is what system your prognosticator is using, what biases that introduces, and learn to make the necessary adjustments to incorporate risk evaluation into the process. Only then can you get the players who fit your league's rules best.”

Ah, biases. The truth is, there is an inherent bias that exists in any comparative analysis that includes the author as one of its subjects. It's impossible to avoid. The reason is obvious: A tout is not going to publish such an analysis unless he can present himself in a favorable light. And the only way to do this is to instill some level of bias into the structure of the study.

Here are some of the ways this is done:

Selection of the study group: Some of the analyses I've seen contained perhaps a half dozen or so prognosticators, but I can easily count at least 20 books, magazines and websites that published projections last year. How do we know whether there were other touts not chosen for the study that might have fared better?

I've seen qualifiers such as: “We evaluated only those players who had a forecast provided by each of the seven projections systems.” This means, the addition or omission of any of the seven prognosticators could change the composition of the players studied, and thus the results of the study.

As such, unless the study is exhaustive, it cannot be completely objective.

Selection of the study variables: We've already discussed the limitations inherent in choosing a study variable. However, those who conduct comparative analysis have to select something to compare. Will it be an overall aggregate gauge like OPS or Win Shares? Will it be a fantasy-relevant gauge like dollar values or fantasy points? Will it be a raw, traditional measure like ERA or batting average? And most important, how do we know that the measuring gauge chosen isn't one that just happens to yield the most favorable results?

As such, unless the study uses a viable test variable, it cannot be completely objective.

Selection of the study methodology: Even if a comparative analysis included all relevant test subjects and somehow found a study variable that made sense, there is still a concern about how the study is conducted. Does it use a recognized, statistically valid methodology for validating or discounting variances? Or does it use a faulty system like the ranking methodology used by Elias to determine Type A, B or C free agents? Such a system -- which ironically is the basis for Rotisserie scoring -- distorts the truth because it can magnify tiny differences in the numbers and minimize huge variances.

As such, unless the study uses a proven, accurate methodology, it cannot be completely objective.

And bias immediately enters into the picture. You simply cannot trust the results.

The only legitimate, objective analysis that can filter out the biases is one that is conducted by an independent third party. But the challenge of conducting such a study is finding a level playing field that all participants can agree on. Given that different touts have different goals for their numbers, that playing field might not exist. And even if one should be found, there will undoubtedly be some participants reluctant to run the risk of finishing last, which could skew the results as well.

NEXT PAGE...