Nate Silver: it's the numbers, stupid

Nate Silver is a new kind of political superstar. One who actually knows what he's talking about. In America, punditry has traditionally been about having the right kind of hair or teeth or foaming rightwing views. Silver has none of these. He just has numbers. Lots of them. And, on the night of the US presidential election, they were proved to be right in quite spectacular fashion.

For weeks and months, the election had been "too close to call". Pundit after pundit declared that the election could "go either way". That it was "neck and neck". Only it wasn't. In the end, it turned out not to be neck and neck at all. Or precisely what Nate Silver had been saying for months. On election day, he predicted Obama had a 90.9% chance of winning a majority in the electoral votes and by crunching polling data he successfully predicted the correct result in 50 out of 50 states.

"You know who won the election tonight?" asked the MSNBC TV news anchor, Rachel Maddow. "Nate Silver."

Twitter went into meltdown. The blogosphere went Nate Silvertastic. Sales of his first book, The Signal and the Noise: The Art and Science of Prediction leapt 800% overnight and went to number two in the bestseller charts. And whole portions of the media decided that this wasn't just a personal triumph for Nate Silver – it was the triumph of the nerds. One man and his mathematical model had bested an entire political class of journalists, spin doctors, hacks and commentators.

Silver doesn't look much like America's latest and hottest new television celebrity. Or "the new boyfriend of the chattering classes", as the Washington Post called him. The 34-year-old Silver is a pretty convincing Clark Kent pre the Superman makeover. He's so unassuming, he shuffles, head bowed, into the room, looking almost embarrassed about the idea of being interviewed.

Poor Nate. It's a lot to live up to. He pushes his glasses back on to his nose. "It's been a little crazy," he says. But then, he doesn't really see it being about him. "I've become invested with this symbolic power. It really does transcend what I'm actually doing and what I actually deserve. And I'd be the first to say you want diversity of opinion. You don't want to treat any one person as oracular."

It might be a bit late for that, however. The day after the election, he went on The Daily Show and Jon Stewart saluted him as "Nate Silver! The lord and god of the algorithm."

In other circumstances, if Silver had been a different sort of personality, a more egotistical one, this all could be a bit much. But this is also a story about the underdog coming out on top. In the weeks before the election, Silver's critics (largely on the right, angry that he was predicting an Obama win) attacked not just his methodology, but also him.

Dean Chambers of UnSkewedPolls.com railed against his "voodoo statistics", claimed he'd been "smoking the wacky weed" and finally pronounced him a "thin and effeminate" man "of small stature" with a "soft-sounding voice".

There was more than a touch of homophobia to the criticism (Silver is gay), not to mention an aversion to scientific rationalism that has come to characterise certain segments of the conservative right. (Gawker compared the attack to "something like a jock slapping a math book out of a kid's hands and saying, 'NICE NUMBERS, FAG.'")

But in some ways, it seems like the adulation has been harder to handle. "It does make me nervous. Because I guarantee that we are going to start getting some things wrong," he says. The "we" is his blog, FiveThirtyEight (named after the 538 electoral college votes), which he set up in 2007 to provide a more data-driven analysis of politics and which, in 2010, he moved to the website of the great cathedral of American news and politics, the New York Times.

It's a fact that reinforces his Clark Kent credentials. We're sitting in the Times's super-slick Manhattan skyscraper, with its multimillion-dollar view of the Midtown skyline. And yet, in the days before the election, it was Silver's scrappy little blog that was driving 20% of its traffic. (It's perhaps not a coincidence that DC Comics announced last month that after 70-odd years, Superman would be leaving the Daily Planet to set up a blog.)

Because if Silver is not the Superman of Big Data, he's definitely Supergeek, its pin-up boy. And he's singlehandedly shown that most political punditry is about as effective a method of truth-seeking as the ducking stool.

It's always been about the numbers with Silver. "I've just always been a bit of a dork," he says. Growing up in East Lansing in Michigan, his local baseball team, the Detroit Tigers, won the World Series when he was six "and there were all sorts of stats and that was it really".

After graduating from the University of Chicago with a BA in economics, he spent four years working as a consultant for KPMG. Then he discovered online poker. "It was in the days when there were a lot of people coming on the sites who really had no clue," says Silver. He, on the other hand, was good enough to take their money, jack in his job and "for a while lived the poker dream".

Poker is something of a lodestone in Silver's life. It taught him about chance and the role it plays in life. "And it gave me better training than anything else I can think of about how to weigh new information, what might be important information and what might be less so. Our basic instincts tend to be not very good. We tend to overweigh new information."

Finally, he says, there was "this tenuous relationship between skill and luck and the ambiguity between the two. When I made this money, was I lucky or was I good? You can never know." He estimates he made about $400,000 (£252,000) from online gambling, enough to pursue his other love: baseball. The sport was changing and Silver became one of a few people who saw the potential for it to be more factual and data-driven. He set up a website, Pecota, which modelled a system for forecasting the career prospects of Major League Baseball players and which he later sold to Baseball Prospectus.

It was a battle between old-school scouts and a new wave of statistically minded newcomers, a struggle that came to be told in Michael Lewis's bestselling book, Moneyball, and later a film starring Brad Pitt. By 2007, Silver was casting around for something new.

"I was looking for something like baseball, where there's a lot of data and the competition was pretty low. That's when I discovered politics." He was anonymous initially, calling himself Poblano, until a few months later he outed himself.

Silver thought that by taking the available data and applying Bayesian theory to it, he might have "some small edge". There is a lot of data in American politics and Bayesian theory, a way of calculating conditional probabilities, has been around since an English clergyman, Thomas Bayes, first formulated it sometime at the start of the 18th century. It was not, as his critics have been quick to point out, exactly rocket science.

But then Silver is the first to agree. He's not even the only one doing it. Others had equally good results. At its heart, it's absolutely bog-standard statistics. He aggregates polling data. The twist or the "secret" to his model is what weight he gives to that data. How it's performed historically, what biases it might have, what other information can be brought to bear upon it.

Yet, in the rarefied world of US politics, it's proved spectacularly more accurate than what was around before. Or even in British politics for that matter. We don't have the same abundance of polling data that exists in America, so its use here might be less successful, but it didn't stop the Daily Telegraph's Janet Daley weighing in on the US election. On polling day, she declared a victory for Romney, on the grounds that Obama's campaign didn't "feel" like a winner to her. Others had a "hunch" that Romney would edge it. With competition like this, says Silver, it really wasn't so difficult to do something just slightly less medieval.

"Numbers aren't perfect, but for me, it's numbers with all their imperfections versus bullshit. You had people saying, 'You can't quantify people's feelings through numbers!' But what's the alternative? Me sitting at my Georgetown cocktail party saying that I know how people in Toledo, Ohio, are going to vote better than the actual people of Toledo, Ohio, who answered a survey? It's incredibly presumptuous. And truth is an absolute defence. So if they got it right it would be one thing, but they didn't. They're consistently quite wrong."

Silver doesn't work the Georgetown party scene. He doesn't meet the lobbyists, spin doctors, campaign managers and press officers. He doesn't, in short, play the system, because political reporting, both in the US and the UK, is a system, a system that can at times resemble a cartel. In Britain, the you-scratch-my-back-and-I'll-scratch-yours atmosphere of the lobby came under scrutiny during the expenses scandal, a scandal it took a journalist outside politics to bust apart. In the US, Silver describes it as "transactional".

"Once in a while, you'll get the occasional scoop, if you're well-behaved and play the game. But it's all just a game with a lot of vested interests at work. I try not to talk to the campaigns because it's mostly noise."

What's interesting is that the campaigns, most especially Obama's, understand the importance of data. They hired a "chief scientist" and according to the campaign manager, Jim Messina, set out to "measure everything". Numbers told them who to target and how to target them.

In this context, Silver's skills seem not just relevant but vital. The liberal media don't care, perhaps, when it's their side winning; they may next time around. Because this is military-grade spin, targeted like a drone strike at the level of the individual. The political class has responded by waving the equivalent of a crucifix at it.

Mark Henderson, the British author of The Geek Manifesto, observed on his blog that Silver's recent prominence just goes to highlight the anti-scientific bias at the heart of so much of our media, how, for example, "in the past two years, Melanie Phillips has been on Question Time more than all scientists put together".

Silver's background and methodology mark him out from the rest of what in the US is sometimes called the "gang of 500", the familiar faces with familiar views who are wheeled out on Question Time-like political programmes.

It's a small, self-referencing cohort that the echo-chamber of Twitter has only amplified and distorted, leading to what Silver believes is the worst kind of "group think". He gives as an example the presidential debates "where the conventional wisdom solidifies very quickly. I was 15 minutes late for the first one, and by the time I got home, it had already been decided".

It turns out that what he calls his "dorkiness" is actually the secret to his powers. "I've always felt like something of an outsider. I've always had friends, but I've always come from an outside point of view. I think that's important. If you grow up gay, or in a household that's agnostic, when most people are religious, then from the get-go, you are saying that there are things that the majority of society believes that I don't believe."

What made you more of a misfit, I ask, being gay or a geek? "Probably the numbers stuff since I had that from when I was six."

It's one of quite a lot of things he's started to worry about: that his prominence will mean he won't just forecast future elections, but also influence them and that he'll lose his outsider's edge. "You get these different opportunities, but I don't want to be corrupted and drawn into these scenes."

He's looking for the next low-hanging fruit, an area with lots of data and "not much competition". Economic news, he thinks, is ripe for the approach. And local government, though "not in a predictive way". He did a brilliant analysis a few years back of New York neighbourhoods, which allowed you to weight your biases (green space, say, and good schools) and then used reams of data from mayor Michael Bloomberg's office to provide a personal analysis of where in the city you should live.

Then there's the $700,000 book deal he struck after his success in predicting the 2008 election, on the grounds that, as he says candidly in his introduction, his publishers wanted "a triumph of the nerds" type book. At the time, nobody knew quite how triumphant he would turn out to be. But then, nor did Silver have much of an idea of where the book would lead.

"I thought it was going to be more like, 'Here's how taxi drivers work out how to pick up customers.' And, 'Here's how online dating sites work.' There is some of that, but there's a more philosophical element that emerged from it. "It's the intersection between objective and subjective reality. We are confronted with all this data, but oftentimes we screw it up. Having more information doesn't necessarily make us any better at predicting what will happen in the world."

In the book, Silver evaluates things that we've actually become quite good at predicting (the weather, surprisingly), things we are bad at predicting (share prices), things we are bad at predicting but think we are good at predicting (the economy) and things that could possibly be predicted but we have a track record of not predicting (terrorist attacks).

Before meeting Silver, I had, earlier in the day, met Nassim Nicholas Taleb, author of The Black Swan, the book published in 2007 that claimed that the world we had created was too complex for us to understand and that it was only a matter of time before the global financial system collapsed. What do you think of Nate Silver? I asked him.

"He's the real deal," said Taleb. "What he's doing is absolutely proper stuff."

Silver is delighted when I tell him this. I'm not sure if it's the comparison to The Black Swan or the unsteady feeling I get standing next to the New York Times's vertiginous floor-to-ceiling windows, but later that night, I reread Silver's chapter on terrorism and feel a bit spooked.

He plots terrorist events before 11 September according to something known as a double-logarithmic scale and finds that an atrocity of that type wasn't actually unimaginable; statistically, it was likely to happen within our lifetime. It wasn't a black swan. What's more, there's a mathematical case to be made for an upcoming attack that "might kill tens of thousands or hundreds of thousands".

I hope this isn't Silver's black swan moment and it's just me doing what he says we shouldn't do: cherry-picking information according to our biases. Because if Silver teaches us anything, it's that human judgment is fallible. That Wall Street traders are chancers, pundits are clueless and economic forecasts imaginative works of fiction.

On the other hand, it turns out that elections are easy to predict. You just have to be lucky or good. Or, as in Nate Silver's case, both.

The Signal and the Noise by Nate Silver is published by Allen Lane