Commentary

Much gray area in defensive analysis

Trust the eyes or the numbers? Defensive ratings still highly subjective

Originally Published: November 18, 2009
By Jerry Crasnick | ESPN.com

Poor Matt Holliday. He goes from Oakland to St. Louis in July and slugs .604 down the stretch to help the Cardinals make the playoffs. And just when the lights are shining the brightest -- figuratively -- he takes a James Loney line drive off the anatomical nether regions to help lose a game.

Holliday's reputation as a defender took a bigger hit than the family jewels on that Division Series misplay against the Dodgers. But if this comes as any consolation, one prominent baseball man still vouches for his glove work.

No, it's not Scott Boras. We're talking about A's general manager Billy Beane, the guy who brought Holliday to Oakland by trade last winter.

"I personally think Matt Holliday is a very good left fielder," Beane says. "When people gather opinions with their eyes, certain emotional biases may come into play. Drop a ball in the ninth inning, and they'll remember that more than if you cover more ground than the average major league outfielder does in the fourth inning of a 6-1 game."

So what happens when the eye test and the data conflict? The answer might depend on whether you're talking to the buyer or the seller.

As baseball's free agent season gets into full swing and teams try to distill salesmanship from reality, defense is increasingly being looked upon as a potential separator for players in search of big deals. That's particularly true in the case of Holliday and Jason Bay, the two best hitters on the market this winter.

Executives who want to nitpick can harp on Holliday's mediocre numbers in Oakland or his pronounced splits at Coors Field when he was with the Rockies, or Bay's high strikeout totals and penchant for streakiness. But both players fill the bill as high-impact, middle of the order producers.

Boras, Holliday's agent, and Joe Urbon, Bay's representative, appear to be quibbling over defense when they argue that their guy is the most "complete" player available. And that's where things get a little murky.

The most commonly accepted analytical measures suggest that Holliday is the better defender of the two. He ranked as the fifth-best left fielder in the game for 2009 in FanGraphs' ultimate zone rating, a metric devised by sabermetrician Mitchel Lichtman. Bay, in contrast, had the second worst UZR among left fielders, just ahead of Milwaukee's Ryan Braun.

According to John Dewan's "Fielding Bible" rankings, Holliday saved 14 runs this year with his defense in left field. That placed him third in the big leagues at the position behind Tampa Bay's Carl Crawford and the Angels' Juan Rivera. Bay, in contrast, cost the Red Sox a run with his defense and graded out as the 23rd-best left fielder in the majors.

The eye test, however, tends to complicate matters. Several baseball executives told ESPN.com that they consider both Bay and Holliday below-average defenders, and one American League GM described Holliday as "brutal" with a glove. A National League assistant said his team's internal defensive metrics rank Holliday as superior to Bay, but that the team's scouts actually prefer Bay to Holliday.

"If there's some kind of discrepancy, you need to use your best judgment," the assistant says. "If a scout says, 'This guy stinks,' but the numbers say he's excellent, the truth probably lies somewhere in between."

Phillies GM Ruben Amaro Jr. became an object of sabermetric scorn last year when he expressed disdain for new-age defensive analysis, but even his more open-minded colleagues concede that the metrics tell only part of the story. Seattle general manager Jack Zduriencik, who has a longtime background in scouting, favors a blend of progressive and traditional analysis.

"Another general manager was telling me about his center fielder," Zduriencik says. "He said, 'The metrics say he's off the charts, but I watch him every day, and I don't see it.' That's why I think you have to use all the information available.

"We have guys in house who are always reading and researching and studying, but I don't think you ever replace the eyes of a scout. We're paying them. They work for us. They have experience and they're pretty good. And you have to believe what they tell you."

Zduriencik has made a major commitment to defense in Seattle with the addition of outfielder Franklin Gutierrez and shortstop Jack Wilson, and numerous other clubs have taken a similar tack. In 2008, Tampa Bay rode one of baseball's most efficient defenses to a 31-win improvement in the standings. This year, the Yankees spent $243.5 million on free-agent pitchers CC Sabathia and A.J. Burnett, and GM Brian Cashman considered a defensive upgrade imperative to maximizing the investment.

Throughout the game, teams are channeling resources into more sophisticated defensive analysis -- and taking pains to be secretive about it. Stats Inc., Baseball Info Solutions and other services provide information collected by video scouts who watch games and break down each play. The scouts categorize balls according to how they're hit (on the ground, in the air or on a line) and with how much force (soft, medium or hard). The ultimate goal is to determine how proficient fielders are at making plays inside and beyond their individual "zones."

It's a quantum leap from the stone age, when errors and fielding percentage were the sole barometers of defensive skill. But as Beane points out, the new information is rooted in human judgment, and that means individual biases will inevitably come into play.

"There's no subjectivity in measuring a 95 mile per hour fastball, except for the margin of error on the radar gun," Beane says. "But when you're using 'objective' data, you have to be careful to ferret out how it was collected. There's always going to be some form of subjectivity. The closer you can get to minimizing it, the more accurate you're going to be."

The modern defensive metrics have limitations. They don't account for positioning, so a player might benefit from an astute infield or outfield coach and superior advance scouting. Milwaukee GM Doug Melvin has also observed that it's easier to assess defense for teams that play straight up than for those that employ shifts in the field.

Bay's supporters contend that his range factor stats are hindered by his surroundings. He's not going to get much credit for catching balls outside his designated zone because there's so little room to roam in left field at Fenway Park.

Dewan, the founder of Baseball Info Solutions, has done much to advance the cause of defensive analysis, but some of the Fielding Bible rankings invite skepticism at first glance. This year the Mets' Daniel Murphy, a converted outfielder, tied for first among big league first basemen with a "plus-minus" ranking of +14. The Yankees' Mark Teixeira, who appears to set the standard for the position, ranked 17th in the same system.

[+] EnlargeMatt Holliday
Kirby Lee/Image of Sport/US PresswireMight Matt Holliday's miscue in Game 2 of the NLDS against the Dodgers cost him a few bucks in free agency?
In 2008, Phillies second baseman Chase Utley played several months on a hip so bad it required off-season surgery, yet still ranked as the game's pre-eminent defender with 33 runs saved. This year, Utley returned with a surgically-repaired hip, but his defensive metrics slipped. He's living proof that defensive stats are subject to greater fluctuation from year to year than offensive numbers are.

Baseball executives clamoring for more foolproof analysis might soon get their wish. Sportvision, the company that pioneered the Pitch/FX technology used to monitor the speed and movement of pitches on television and the Internet, is working to apply similar technology to defense. This year, Sportvision installed special cameras at AT&T Park in San Francisco to track everything from fielders' jumps to the velocity of batted balls. A more widespread rollout of the product is still under discussion.

Ryan Zander, Sportvision's general manager for baseball products, said the new application will serve three purposes -- as a coaching tool, a scouting tool and a means for teams to more accurately judge defensive performance. As Christmas gifts go, it's the equivalent of "The Beatles: Rock Band" for the modern-day baseball executive.

"With this system, we'll be able to get things like running speed, throwing speed, reaction times and 'glove to throw' time -- all sorts of information," Zander says. "We're just scratching the surface right now. There's going to be an overwhelming amount of data that will give teams the ability to look at trends over time."

No amount of technology is going to eliminate haggling in contract negotiations. In his Jason Bay sales pitch to clubs, agent Joe Urbon duly notes that Bay handled 325 total chances without an error this season and led the American League with 15 outfield assists. That information, although useful, must seem quaint to baseball executives in the statistical revolution.

Beane listens to his scouts, for sure. But when it comes to analyzing defense, he prefers cold, hard data to knee-jerk conclusions. Even his own.

"You have to be careful when you see something with your eyes, because sometimes your emotions tend to dictate your viewpoint," Beane says. "I can't explain why a magician looks like he pulls a rabbit out of his hat. I just know the rabbit wasn't in the hat."

Jerry Crasnick covers baseball for ESPN.com. His book "License To Deal" was published by Rodale. Click here to order a copy. Jerry can be reached via e-mail.

Jerry Crasnick | email

ESPN.com MLB Sr. Writer