Prospect Evaluators Search in Vain for a Third Way
This article is from the archive of The New York Sun before the launch of its new website in 2022. The Sun has neither altered nor updated such articles but will seek to correct any errors, mis-categorizations or other problems introduced during transfer.

If there is one thing that baseball fans love this time of year, it’s to talk prospects. Baseball America, Baseball Prospectus, Sports Weekly, and any number of independent analysts whose evaluations are geared toward the fantasy baseball market are getting ready to release their lists of the top 100 prospects and stake their reputations on the futures of ballplayers barely old enough to drink.
Mets and Yankees fans might be disappointed by these lists, as both teams have spent much of the past few years trading away some of their best young talent and graduating the rest to the majors.
But prospect analysis is a worthwhile – if dodgy – enterprise. Of the many broad issues involved in picking out the future stars from among the thousands, there are two really important, and interrelated, ones right now. The first is the debate between those who base their evaluations on scouting information and those who base them on statistical analysis. The second is the debate over whether a prospect should be valued for what he might theoretically do in a major league uniform, and what he’s likely to do.
The first debate isn’t really so antagonistic as people might think. During the last several years, baseball traditionalists have come to value the perspective of those who look at the game objectively quite a lot more than they once did. At the same time, sophisticated analytical tools like Nate Silver’s PECOTA projection system (disclosure: Silver is a friend) and Mitchel Lichtman’s Ultimate Zone Rating, a system that quantifies defense, have shown that many of the old folk wisdoms associated with baseball are quite valid. Among those would be the notion that more athletic players are more valuable than less athletic ones, that bigger players are, on the whole, more valuable than smaller ones, and that players who strike out less develop better than those who strike out more.
All this said, contentious matters remain. As scouting and statistical methods have converged and the space between them has grown smaller, the issues over which they’ve conflicted have become less about evaluating the abilities of players and more about the context into which those evaluations fit.
For instance, which is more valuable: a 25-year-old who, despite never having played at the major league level, has displayed the skills necessary to be a solid big league regular? Or a 23-year-old who has a 10% shot at being a star but will most likely never play regularly in the majors?
To some extent, there are simple philosophical differences at play here. The traditional scouting methods would say that you always want to side with the younger player who has a greater perceived upside. Statistical methods, which have a great deal in common with the actuarial analysis an underwriter would use, would side with the player who’s more certain to have a particular value.
In sum, scouts would side with riskier players; statheads would side with those who are better bets to perform on the biggest stage of all. So where should those of us who feel no ideological fealty to either side come down?
It’s a difficult question. For one thing, even the most sophisticated statistical analysis is based to some extent on the raw judgment of scouts; without the baseball men who go to obscure small towns all over the country in search of talented athletes, there would be no raw data for the analysts to sort through. On other hand, the biases of wizened baseball men who don’t subject their information to the kinds of tests analysts do aren’t, perhaps, to be trusted. It’s a quandary.
The lame answer is to say, “Let’s synthesize the materials. Let’s look at the Baseball America prospect list, and the Baseball Prospectus one, and split the difference. “I don’t think that makes any sense. While both approaches have their merits, and while in an ideal world we’d be able to take the best of both, the fact is that we have to make a choice.
When Baseball America chooses a raw, toolsy player as the best prospect in a team’s system based on subjective analysis and his upside, while Baseball Prospectus rates him as the 20th best based on the fact that a comparison to players who have performed similarly have never fulfilled their potential, there’s almost no way to compromise. One is right, and one is wrong.
As of now, I’d have to side with the scouts. They may get it wrong a great deal of the time, but while they may evaluate based on folk wisdom, they do it on a century’s worth of the stuff, while objective analysts are still refining their analytical tools, and often finding that the more they refine them the more they have in common with those employed by supposedly crude scouts.
The point is worth thinking about right now, though, when any fan or insider looks at the lists that are about to hit the streets; we’re not more than a couple of years from the point when actuarial analysis supercedes its more hunch-based counterpart. Whenever that happens is when it will be time to have a real debate.