And finally, there is the issue of “Marcel the Monkey.” This is the assertion by folks on some of the sabermetric blogs that a “chimp forecasting method” — a simplistic averaging of the last few seasons and making minor adjustments for age — is nearly as good as any other, more comprehensive system.

Well... this is mostly true. If 70% accuracy is the best that we can reasonably expect, Marcel gets us about 65% of the way there. All of our “advanced” systems are fighting for occupation of that last 5%.

Ashley-Perry Statistical Axiom #4: A complex system that works is invariably found to have evolved from a simple system that works.

Occam's Razor: When you have two competing theories which make exactly the same predictions, the one that is simpler is preferred.

Even if it was created by a monkey, I suppose.

Married to the model

It's one thing if the model has a name like Claudia Schiffer, but quite another if a tout is so betrothed to his forecasting model that “it” becomes more important than the projections.

Whenever I hear a tout write, “Well, the model spit out these numbers, but I think it's being overly optimistic,” I cringe. Well then, change the numbers! The mindset is that you have to cling to the model, for better or for worse, in order to legitimize its existence. The only way to change the numbers is to change the model.

On occasion, I will take a look at one of my projections and admit that I think it's wrong. Usually, it's because I see things in the BPIs that I overlooked the first time through. Then I change the numbers.

In the end, is the goal to have the best model or to have the best projections? That should be a no-brainer.

Hedging and the comfort zone

Given the variability in player performance a "real world" forecast should not yield black or white results. Some touts accomplish this by providing forecast ranges, others by providing decile levels. We provide a single statistical projection, for simplicity's sake, and then color it in our player commentaries. In fact, most touts do this, however, many use the commentary as a hedge against the numbers they've committed to. But when does a hedge negatively impact your ability to assess the accuracy of a projection?

One of the best examples was Ben Sheets last year. This was a pitcher coming off a 4.46 ERA, yet had incredible leading indicators. The typical forecast would never venture into uncharted, sub-4.00 ERA territory because straight computer-generated projections would neither find the history nor see a trend that pointed in that direction.

Still, four of us did break rank. But which of these projections, and comments, was the most committed?

Tout 1 — Us (3.94 ERA projection here last year, updated to 3.82 on BaseballHQ.com): “BPIs are developing nicely, but a 5% drop in his strand rate served to hide those gains in a higher ERA. Keep a close eye on this one. He's at the prime spot to post a breakout season. Major sleeper."

Tout 2 (3.97 ERA): “He's got better stuff than his numbers would indicate but his upside is limited pitching for the Brewers. He would warrant fantasy consideration in NL-only leagues but probably not elsewhere.”

Tout 3 (3.83 ERA): “Fewer walks, fewer strikeouts. Hard to say what to make of that. Besides hooray if you're in a 4x4 league, boo if you're in a 5x5 league. Basically the same season.”

Tout 4 (3.79 ERA): “Sheets has been a very good pitcher but lacks the consistency to reach the next level. He relies too much on just his curve and fastball... If he can find a third pitch and stay away from the gopher ball then he has a chance to post a sub 4.00 ERA next season.”

The other three touts provided skittish recommendations, but their “official” published projections all eclipsed the 4.00 ERA barrier. It's a common hedge. Did they truly believe Sheets had the potential to post a sub-4.00 ERA? You wouldn't know it from their comments alone. That makes it difficult to figure out the "official line" on their projection. In the past, some authors used this tactic as a means of playing both sides so that they always had a winning projection to promote the following year. Thankfully, that level of deception is rare these days.

But also notice that none of us four touts came anywhere close to projecting the season that Sheets really did put up, even though the evidence in his BPIs was strong and supported such a breakout performance.

As a group, there is a strong tendency for all pundits to provide numbers that are more palatable than realistic. That's because committing to either far end of a range of expectation poses a high risk. Few touts will put their credibility on the line like that, even though we all know that those outliers are inevitable. The easy road is often just to split the difference.

I handle this phenomenon in the Baseball Forecaster by offering up the possibility of outlying performances in the commentary. Occasionally, I do commit to “official” outlying projections when I feel the data supports it. But on the whole, most projections are going to be within close range of the mean or median expectation of a player's performance.

I like to call this the comfort zone, a range bordered by the outer tolerances of public acceptability of a projection. In most cases, even if the evidence is outstanding, published pundits will not stray from within the zone.

For instance, nearly everyone in 2004 assumed that a healthy Randy Johnson would be a vintage Randy Johnson, yet not one tout had him down for a 20 win, 2.50 ERA season. Most touts doubted Esteban Loaiza's ability to repeat his 2003 numbers, but nobody was willing to risk the possibility that he might revert to his pre-2003 form. In fact, in a survey of 10 touts last April, eight of them projected an ERA between 3.50 and 3.93, even though Loaiza had never posted an ERA in that range in his entire career.

They say that the winners in any fantasy league are those who have the most outliers on their teams. There is an element of truth to this. It is likely that owners who rostered surprises like Johan Santana and Adrian Beltre fared well in the standings this past year. The problem is, these type of performances are the most difficult to project. Still, the prognosticators who fare the best in this exercise should get their props, shouldn't they?

According to analyst John Burnson, the answer is no. He says: “The issue is not the success rate for one player, but the success rate for all players. No system is 100% reliable, and in trying to capture the outliers, you weaken the middle and thereby lose more predictive pull than you gain. At some level, everyone is an exception!”

Peter Kreutzer again: “Those projections that are outside the comfort zone, as Ron calls it, are flashy, but they're of little statistical use. What you want is to follow the predictor who gets the general flow (guys who improve, guys who fall off) more right than anyone else. If someone does that they'll make you money in almost any league.”

Yes! That “general flow” is far more important than any pure accuracy level. And far more attainable. And perhaps, that is the study variable that makes the most sense.

NEXT PAGE...