“Re: Nate Silver, most amusing thing about this election is watching political pundits make sports fans look like PhD mathematicians,” tweeted ESPN basketball writer John Hollinger earlier this week.
Hollinger was referring to the mainstream media’s largely ignorant criticisms of Nate Silver, the New York Times writer who uses polls and other information to assign probabilities to political outcomes, including, most prominently, the presidential election. Earlier in the day, Politico’s Dylan Byers argued that because Silver had given Barack Obama nearly three-in-four odds, then if Mitt Romney prevails, Silver will not really be worth paying attention to anymore: “should Mitt Romney win on Nov. 6,” Byers wrote, “it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance.” (Byers is actually wrong on this fact as well: one year ago, Silver gave Romney a 60 percent chance should 2012’s GDP grow two percent, which was last quarter’s annualized rate. But as we’ll see, that’s beside the point.) Byers also cited pundits like Joe Scarborough and David Brooks disparaging Silver after similarly misunderstanding the concept of probabilities.
In addition to this line of criticism, one need only do a quick Google search—or check of John Podhoretz’ Twitter feed, or Matt Lewis at The Daily Caller—to find accusations that Silver, a liberal, is skewing the numbers to help Obama. (Here is Podhoretz, who can be very funny even when he is wrong: “Battleground poll has Romney 52-47. In response, Nate Silver raises Obama probability to 99 44/100% pure.”)
Hollinger, a stats-savvy journalist, is right: the world of sports prognostication is more enlightened than the world of political prognostication. The analogy holds in part because what Silver does as a political statistician is strikingly similar to what he did as a baseball statistician. “When Nate Silver worked at Baseball Prospectus,” notes Columbia political scientist Robert W. Erikson, “he did a lot of statistical analysis of the sort he’s doing here.” His most famous invention was a model called PECOTA that guessed how a player would perform based both on his past performance and the past performances of “comparable” players (much as Silver adjusts for polls’ Democratic or Republican leans). According to Erikson, PECOTA was a slightly more accurate forecaster than any other models that attempted the same thing.
But nobody in baseball today bats an eye when PECOTA is cited. No longer would the basketball equivalent of Scarborough or the football equivalent of Brooks be permitted to make claims that are based on such basic misapprehensions of math without facing near-universal reprimand. A look at the way the sports world has assimilated new, smarter ways of dicing up the numbers contains several lessons for whether and how the political universe’s gatekeepers will admit Silver and his ilk. Chances are that Silver will not earn similar mainstream recognition—even if Obama wins.
IN 2009, UP SIX points late in the game, New England Patriots coach Bill Belichick famously decided to go for a first down on fourth-and-two rather than punt. The moment catalyzed football’s advanced stats community: it was a perfect demonstration that the more unconventional option nonetheless gave the Pats a greater chance of winning. Just as importantly, the gambit didn’t work—the Pats failed to convert, and Peyton Manning’s Indianapolis Colts proceeded to score a go-ahead touchdown to win, 35-34. While this may have appeared to the less-enlightened as a repudiation of the stats, it actually allowed the geeks to make a crucial point: namely, that doing the thing that gives you the best chance of succeeding doesn’t guarantee success. As Bill Barnwell, who was agnostic on whether Belichick was right or wrong, wrote: “you can’t judge Belichick’s decision by the fact that it didn’t work” (bold and italics his).
That probabilities do not ensure outcomes—something every blackjack player who has busted while hitting against a face card has long known—has escaped Silver’s detractors. Brendan Nyhan at CJR and Ben Jacobs at Daily Download have emptied an ample volume of bullets into this barrel of fish. Ezra Klein put it succinctly: “If Mitt Romney wins on election day, it doesn’t mean Silver’s model was wrong. After all, the model has been fluctuating between giving Romney a 25 percent and 40 percent chance of winning the election. That’s a pretty good chance!”
But Silver and his defenders have run aground on the same problem sports statisticians used to face: the failure of laymen to grasp the difference between predictions and probabilities. “The criticism of Nate is that he’s predicting something, when he’s trying to explain that’s not what he’s doing at all,” said Dave Cameron, a baseball statistician at FanGraphs who briefly worked with Silver at BP. “He’s putting the odds on something.” Cameron added, “It is kind of like what we do in baseball. We recognize there are multiple outcomes. A utility infielder can hit a home run off [reigning Cy Young winner Justin] Verlander. It’s just not probable.”
Support thought-provoking, quality journalism. Join The New Republic for $3.99/month.
Aaron Schatz, head of the stats site Football Outsiders, defined three categories of criticisms he typically receives beyond fans’ failure to understand that he is providing probabilities, not predictions. Not surprisingly, Silver has faced political versions of all three.
One is that there is something wrong with the model. “That’s the equivalent of complaining that John Hollinger's measurements of defense aren't accurate enough, or complaining that we don’t give enough importance to red zone [play],” explained Schatz. “Those are technical criticisms. Those are valid.” They are also the ones least likely to be hurled at Silver, in part because his proprietary model (like many sports models, and like Google’s algorithm, for that matter) is secret.
The second is that the expert is biased. Football Outsiders routinely gets accused of jilting fans’ favored teams. Schatz has been particularly accused of this for a particular reason: he is a Pats fan, and, as he put it, “It’s not my fault that I started doing this in 2003 and the Patriots have yet to suck in the decade since.” But he insisted none of his staff are flag-wavers, and neither is Silver: “If this was the 1984 election,” he said, “I don’t think Nate Silver would be afraid to say that Walter Mondale will get his ass kicked.” Schatz further noted, as others have, that in 2010 Silver correctly predicted huge Republican gains in the House.
The third is the attack on the forecaster’s character. The sports version of this is that the statistician is a nerd in his underwear in his mother’s basement. The politics version of this is that, as the Washington Examiner noted, Silver is thin, high-voiced, and maybe gay. It isn’t worth dignyifying this “criticism” with a “rebuttal,” but it is worth noting the parallel. “That kind of ad hominem nonsense—that’s exactly like what we go through,” said an exasperated Schatz. “My God.”
BUT YOU HEAR THESE criticisms leveled at sports geeks less and less frequently. Ever since Michael Lewis published Moneyball, in 2003, and the Boston Red Sox’ sabermetric-driven championship in ’04, you would be hard-pressed to find a Major League front office that doesn’t crunch sabermetric numbers. Where 15 and even ten years ago, broadcasts would list a hitter’s batting average, today they also list his on-base percentage—which sabermetricians pointed out is the more valuable stat. The legendary second-baseman-turned-announcer Joe Morgan, so disdainful of advanced stats that fan Michael Schur (better known as Parks and Recreation’s creator) started a blog called Fire Joe Morgan, has departed the booth.
“I’m part of the Baseball Writers Association of America,” Cameron marveled. “In 10 years, I’ll have a Hall of Fame vote.” Similar revolutions have happened in football, basketball, and even sports like golf, tennis, and soccer. But this happened because the sports statisticians had a bunch of advantages on their road to mainstream acceptance that Silver appears to lack.
A prime leg-up the sports forecasters have over Silver is vastly, vastly more opportunities to see how good their numbers really are. “He only gets to test his model once every four years,” observed Schatz. “I get to test my model 16 times a week.” Of course, and as Schatz noted, there are more ways to test Silver’s numbers—state-by-state, Senate races, primaries—but still no substitute for the NFL’s few hundred games each year, much less MLB’s several thousand.
Another advantage the sports guys have is that while Silver is more or less purely descriptive in his analysis, they tend to be at least partly prescriptive. “We’re not necessarily just trying to forecast the outcome of a forthcoming event as much as making suggestions and helping inform decisions and actions,” explained Cameron, adding, “He’s saying, ‘Obama’s probably going to win.’ We’re saying, ‘That guy’s terrible, trade for someone else.’” (There are ways to divine advice from Silver’s numbers, but the numbers are not really set up that way.) In other words: at this point, sports teams—and therefore the sports media—ignore advanced numbers at their peril; political teams—and therefore the political media—ignore them at their convenience.
But the campaigns themselves are not innumerate. Team Obama has been famous since 2008 for rigorously understanding the numbers, and a longtime business consultant sits at the very top of Team Romney. So the logjam right now may be the media itself. “ESPN in particular decided they were going to grab onto this and ride it,” said Schatz, whose site has had a deal with the ESPN Insider subscription service for several years. By contrast, many members of the political media perhaps perceive the threat Silver poses: they have a horse-race narrative that not incidentally requires interested observers to refresh their websites several times a day to catch wind of the latest episode in the grand unfolding “narrative”; Silver has numbers that suggest a strong likelihood of what the outcome will be. Having perceived Silver’s threat, they have fought it.
Perhaps most dispiritingly, politics’ closest equivalent to ESPN—the ambitious, all-consuming, disruptive media force—is probably Politico, and Politico features authors like Byers, Mike Allen, and Jonathan Martin who in recent days have been among the most vocal opponents of a Silver-style, three-in-four-odds read of the race. (Politico headline yesterday: “Media Stumped by 2012 Outcome.” Silver’s response: “7 polls released in Ohio in past 48 hours: Obama +2, Obama +3, Obama +3, Obama +3, Obama +5, Obama +5, Obama +5 #notthatcomplicated.”)
Then again, maybe the chief distinction between sports and politics is that politics is a decade or so behind. Forward-thinking ESPN was also the place that gave Morgan the most prominent baseball broadcast job for so many years. And Sasha Issenberg’s The Victory Lab or some other book could prove the necessary, Moneyball-esque popularizing tome. Although one thing seems likely to stay constant: as Silver said in a recent New York article, “there weren’t nearly so many assholes in sports coverage.”