How Accurate Are The Polls

Crunch time is right around the corner.  Soon September will give way to October and then it’s only one month until the election.  I don’t think we can call it over, but the polls are certainly pointing to the end of the horse race.

Are The Polls Correct

This, of course, is the $20,000 question.  With all of the polling coming in right now the question that’s burning up the inter-tubes is this:

Are the polls telling the right story or are they just pulling for Obama?

I tend to believe two things:

  1. Journalists are, by nature, liberal.
  2. We all allow bias into our belief systems.
  3. There is no grand conspiracy.

Okay, so that’s three, but still.  I do think that people who conduct polls are human, that as humans they are subject to their tribalism and yet still they try to remain objective.  I simply don’t believe that these pollsters are in cahoots with one another in order to bring about an Obama 2nd term.

Is Rasmussen Right

Of all the major polling outfits, only Rasmussen has Romney ahead or even close.  Every other poll is showing Obama up slightly or even by 7.  So, is Rasmussen overly partisan or do they have a secret?

I thought I’d check out how close they were last year:

A professor at Fordham University went back and checked.  Of the 23 polls conducted only two were wight on the money; Pew and – you guessed it – Rasmussen.  Whatever those folks are doing over there, agree or not, they did a heck of a job last time around.

And even more than that, the 3 polls that over estimated the republican ticket were still in the top 7.

Instead of arguing that Rasmussen is too far off on the Romney side I’d make the argument that the other polls are further off on the Obama side.

19 responses to “How Accurate Are The Polls

  1. Rasmussen was off more in 2010 and in other years, always with a GOP bias. Nate Silver wrote about that in his blog today. Also, note that PEW showed Obama up 8, and it also was very accurate by your own account. Also, pollsters have over time built a methodology that is not supposed to be objective, they stake their reputation on it. The idea that “journalists are liberal” is not something that would show up in the polls. When a preponderance of polls agree on something, the outlier is probably wrong. In this case, the outlier is Rasmussen (on my blog today I speculate on what is causing Rasmussen’s error). Two positives for Romney: a) Obama may be peaking too soon; b) this could be a bump for Obama from Romney’s 47% tape that could recede; and c) team Romney knows it has to make changes in the final month.

    OK, that was three things.

    • Rasmussen was off more in 2010 and in other years, always with a GOP bias.

      I didn’t check the other years, just the last Presidential election. And he hit it right on the screws.

      When a preponderance of polls agree on something, the outlier is probably wrong. In this case, the outlier is Rasmussen

      I suspect most of them are wrong. The question is which is less wrong?

  2. Well, some of the boys were pooh-poohing this poll: http://therionorteline.com/2012/09/20/rand-paul-says-that-the-election-is-already-over-romney-won/

    It’s interesting. I get a different vibe from other sites. Do you know that the lib site Mother Jones censors me? How very Demoflippincratic of her! Uh-huh. Nah, they’re not socialists. I tell you, sweet Pino, so many cannot see the forest for the trees, and quite frankly; I’m sick to death of trying to show them.

    Then again………….I’m Irish. 😉

    • Do you know that the lib site Mother Jones censors me? How very Demoflippincratic of her!

      Yeah, I get hammered at conservative joints too. But yes, I do wonder at the willingness of the libs to ban people so quickly. If you REALLY wanna see some intolerance, go check out some feminist/women’s rights blogs. Holy shit!

  3. The unskewed polls bit is bogus (I dealt with that on my blog today: http://scotterb.wordpress.com/2012/09/27/blame-the-pollsters-too/

    But Pino is probably right that some of these polls are wrong. They all point to a distinct upsurge in support for Obama in the last week — they also show more people are identifying with the Democrats. This could be a short term reaction to the Romney video. If so, then a good performance at the debates can reboot the Romney campaign. But the bottom line is Romney has run a horrid campaign.

    He had a real opportunity to set a course that was different than Obama AND different than the pre-2008 GOP. He could have stated that we need a “sustainable America” and set forth ideas that would creatively tackle the problems we face. Instead, his vague economic plan sounded much like what the GOP touted for the last ten years – less taxes, less regulations, make it seem like the poor are the problem. That gave Obama an opening to define Romney’s economic plan for him – to say he’s going back to the policies that caused the problem (and most Americans blame Bush and the GOP for what happened in 2008). By not giving the voter something new, Romney got defined as more of the old – and the tape did so much damage because it reinforced the plutocratic image that Obama’s summer ad buy created. The Romney camp let Obama define Romney!

    My gut says this is over, Romney isn’t a good enough candidate to fight back. But maybe I’m wrong. Maybe he’ll turn in stellar debate performances, offer a real alternative that people will accept as being something new (none of this ‘I’m a businessman so I can fix this, but I don’t want to talk about my old business’ crap). The ball’s in his court. Also, from reports like the Cook Report, it seems that Romney camp insiders are agreeing that they are behind by about 3-7 points in most swing states.

    • The unskewed polls bit is bogus

      Totally agree. I’ll check ya out later tonight.

      Also, from reports like the Cook Report, it seems that Romney camp insiders are agreeing that they are behind by about 3-7 points in most swing states.

      I hope they believe it; I think Romney IS down.

  4. You silly boys! Everybody knows the opera ain’t over til the fat lady sings.

  5. Two major points:
    1) You’re saying that Rasmussen, despite being an outlier against all other polling joints, is probably more right this time because of a single election that they nailed. The problem there is that Pew also nailed it and, as Scott pointed out above, Pew has Obama +8. So your whole basis for trusting the outlier Rasmussen (that they nailed the last one) goes out the window.
    2) More to the point, despite nailing the last one, Rasmussen is a horrible polling outfit. They are generally wrong and generally biased in favor of GOP candidates. This isn’t slant or wishful thinking, it’s something that has been studied and quantified: http://fivethirtyeight.blogs.nytimes.com/2010/11/04/rasmussen-polls-were-biased-and-inaccurate-quinnipiac-surveyusa-performed-strongly/. It’s pretty hard to miss by 10 points or more on over 10% of your polls, and missing by 40% on anything at all should be a huge red flag.

    So there’s one polling group that miraculously nailed it in 2008 and now they are far outside the margins from everyone else, including another pollster who miraculously nailed it in 2008. It is insane to put credence in Rasmussen at this point. Sure, they MAY be right, but I MAY be right when I guess at numbers, too. Polling is at least partially scientific and needs to be measured (and trusted) on scientific bases, and as a matter of fact Rasmussen is a bad polling joint.

    • Wow, I hadn’t read that Nate Silver analysis, it really makes the argument that Rasmussen can be considered accurate impossible to accept.

    • You’re saying that Rasmussen, despite being an outlier against all other polling joints, is probably more right this time because of a single election that they nailed.

      They are probably more right than people give them credit for.

      From your Nate Silver article:

      Rasmussen’s polls — after a poor debut in 2000 in which they picked the wrong winner in 7 key states in that year’s Presidential race — nevertheless had performed quite strongly in in 2004 and 2006. And they were about average in 2008. But their polls were poor this year.

      So, after coming out of the gate poorly they performed well and, from the article, have had one bad year. Then, of course, they were exactly right in the 2008 presidential race.

      More to the point, despite nailing the last one, Rasmussen is a horrible polling outfit. They are generally wrong and generally biased in favor of GOP candidates.

      The graph I showed above has 23 polling outfits. 21 of them were wrong. I suspect that almost all polls are “generally wrong.”

      What I find interesting is that if bias were evenly distributed, we would see a more equal skewing. Of the 21 polls that didn’t get it right – 18 were bias towards the democrats. And of those that were bias towards republicans, they were, in general, more right.

      I’m Rasmussen is wrong. I’m just saying that I think everyone else is most likely wrong as well.

      • The graph I showed above has 23 polling outfits. 21 of them were wrong.

        1) That’s not a graph.
        2) They aren’t wrong. If I tell you that a model predicts that Obama will get 51%, with a margin of error of 2 points, and he gets 52%, that model is just as right as a model that predicts he’d get 52% with a margin of error of 2. This is a fundamental misunderstanding of statistics and polling.
        3) You’re using numbers that were collected immediately after the election, which are nowhere near final. A week later, Obama’s victory was actually a half point higher, as shown by a far more researched and much more thoroughly explained analysis of accuracy here: http://www.pollster.com/blogs/pollster_accuracy_and_the_nati.php?nr=1

        That link is actually really well written, and hopefully will explain why your whole Rasmussen point is completely wrong. Yes, Rasmussen got it on the nose when compared to the immediate (and incomplete) results of election night, but that doesn’t tell you that Rasmussen was better than a polling group that got it a point off, as long as the right answer is in the margin of error.

        Even if Rasmussen had been exactly right with the final certified result, that doesn’t tell you that they are reliable. You then have to look at their many other predictions, which Nate Silver showed to be horrible. Rasmussen got that result pretty close, and you’d hope that they would learn from that to keep their models improving, but I have little faith in that right now.

        So, the major points here are:
        1) Your chart above is based on incomplete, immediate results.
        2) You are conflating right/wrong with various accuracy metrics and degrees of confidence.
        3) You are confusing getting it right once with having a track record of accuracy.

        • 1) That’s not a graph.

          😉

          They aren’t wrong. If I tell you that a model predicts that Obama will get 51%, with a margin of error of 2 points, and he gets 52%, that model is just as right as a model that predicts he’d get 52% with a margin of error of 2.

          But the data doesn’t show that; the Fordham study just ranks them in order of accuracy.

          So, the major points here are:

          I think another major point is that an overwhelming majority of the polls in the Forham study over estimate the democrat advantage and even the Pollster data shows 14 over and only 5 under estimating the democrat advantage. In fact, some over estimated it so much that given the MoE they aren’t even “right”, [gallup]. In fact, if you look at the only two polls that give a MoE of 2 -the lowest in the bunch- Rasmussen is far closer than Gallup. As I mentioned, Gallup isn’t even close.

          If I have a point its this:

          Obama clearly is ahead, Rasmussen is an outlier but he’s not some crazy wanker who doesn’t get it right an awful lot. After reading Nate Silver, even he admits that 2010 was simply a bad year. And yes, Rasmussen over estimates republicans, but nearly every other poller overestimates democrats. So over estimating isn’t some fatal flaw – it seems to be in the DNA of pollsters.

  6. There’s a faulty premise at the heart of any attempt to “unskew” the polls — most polling outfits do not set out to sample a certain percentage of republicans or democrats or independents. They randomly sample the population and make adjustments for set demographics (such as ethnicity). If they’re getting higher numbers of democrats than republicans in these random samples, it could easily just be that more people are identifying themselves as democrats than as republicans, and that may largely be a function of romney being a crappy candidate.

    Now it may turn out that the party affiliation numbers are off from what we see on election day. But it’s not because a bunch of liberals were doing the polling (many polls use both dem and republican pollsters)

    • There’s a faulty premise at the heart of any attempt to “unskew” the polls — most polling outfits do not set out to sample a certain percentage of republicans or democrats or independents.

      With out a doubt. By the way, perhaps I should be more clear, I do NOT think that unskeweredpoll or whatever it is, is close.

  7. I say, how about these boys?

    “Ken Bickers from CU-Boulder and Michael Berry from CU-Denver, the two political science professors who devised the prediction model, say that it has correctly forecast every winner of the electoral race since 1980.” http://www.huffingtonpost.com/2012/08/22/university-of-colorado-pr_n_1822933.html

    They’ve got an accurate track record since 1980……..oh, and they predict a Romney win.

  8. As a social scientist, I can tell you there are many models out there, right now most are pointing to an Obama victory. That model was done retroactively and there is an N of 7. Not much to go on! But hey, if you’re searching for something to give you hope, go for it! I think Nate Silver’s model is the most persuasive. There is also a model that has been accurate for 150 years (developed in the 80s) called “Keys to the Presidency”: http://scotterb.wordpress.com/2011/08/30/obama-a-sure-thing/

    I do think the race is tightening again, though Romney now has Obama up two. I’d be very surprised if Romney won. It’s very rare for a candidate to come back at this point from being this far down. Yeah, Reagan did, but Romney’s no Reagan.

    • Yeah, Reagan did, but Romney’s no Reagan.

      Yeah, but Obama makes Carter look like Clinton!

      • Obama is so much better and popular a politician than Carter. At this point Carter’s favorability rating was under 35%. Obama’s been at 50% and above. Obama’s personal approval (not job approval) is even higher. Obama is more in a class with Reagan and Clinton than with Carter. If Romney were facing a Carter, he’d have a much better chance.

Leave a Reply to Scott Erb Cancel reply