Looking to the Polls for Answers?

The news was filled with charts and graphics of the latest NBC/Wall Street Journal poll today.

Let’s start by saying I have a little experience with survey research – known popularly as polling. I studied methodology and statistics. I worked as a research officer in mental health for years. I’m not a world renowned expert, but I am an informed consumer.

Some principles are absolutely essential for results that have any value:
* random selection of participants
* large sampling (at least 2000 to have real statistical significance)
* pre-testing the questions to ensure participants understand what you’re asking
* unbiased statement of the questions
* adequate range of responses
* timing – complete the survey efficiently so relevant events don’t intervene (some people get asked before the relevant event, and others after, potentially changing the results – i.e., “Is your home insurance adequate?” asking some before the hurricane hits and some after – the first people might have given you a different answer after!)

The next time you pick up a newspaper or turn on the news and see another poll telling you how the American people are feeling on any particular subject, try to find out whether those principles have been met. It’s hard to do, because the most you usually get is the “+ or – “ levels. If the + or – levels are higher than 2%, ignore the data completely if anything important is riding on a close result.

That said, I was interested to see the latest NBC/Wall Street Journal poll. I searched around a bit until I found the full report. They surveyed only 1005 people, in 350 geographic regions, made sure they got an even number of males and females … not truly random sampling, but they tried to ensure a “representative” sample … and their results had an error rating of +- 3.1% for some questions, and +/- 4.4% for other questions. That pretty much wipes out most of the differences in this poll. The pundits, of course, are finding evidence for just about everybody’s viewpoint in these results.

But there were a few things that are probably accurate reflections of American sentiment. I’ll cherry pick a little.

To start, “What do you think will be an indicator that the economy is improving?” 58% said increased employment – only 9% are looking at the stock market. Can somebody please tell that to politicians and media pundits? Economists have their formulas, but people live in the real world.

51% approve of the job Obama is doing. This is unchanged. 77% approve of him as a person, also pretty steady.

22% approve of the way Republicans in Congress are handling health care reform, 66% disapprove. Even with a small sample and large margin of error, this is a significant finding. But not a surprising one.

Who will be to blame (remember that part about unbiased questions?) if health care reform fails? 10% blame the President; Dems 16%; the Repubs 37%; everyone equally 23%.

39% approval on Obama’s health care plan – 41% disapprove, 20% no opinion. This is an improvement over last month’s poll, where the approval was only 36%, disapproval was 42%, and the no opinion/not sure was 22%. If it were a bigger sample, I’d say this is evidence of a swing in favor of the proposals President Obama has been touting.

45% said it would be better to pass the plan than stick to the status quo; 39% favor keeping the current health system.

This is one of my favorite questions:

“Thinking about efforts to reform the health care system, which would concern you more? Not doing enough to make the health care system better than it is now by lowering costs and covering the uninsured; OR Going too far and making the health care system worse than it is now in terms of quality of care and choice of doctor.”

The way they asked that question was unclear, contained apparent bias, and was pretty restrictive in the responses. They only asked half the respondents this one, so the sample was even smaller. It would be hard to base your vote on this response if you were a member of the Senate Finance Committee, but let’s bet some will.

The responses? Not doing enough 44%; Going too far 48%; Not sure 8%

For what it’s worth, one month ago the responses to that question were: Not doing enough 41%; Going too far 54%; Not sure 5%.

You might interpret this to mean that President Obama’s public sales pitch has been working, because this time more were afraid the reform wouldn’t go far enough, and fewer were afraid it would go too far.

Consider the possibility that of the 41% who disapprove of Obama’s plan, many are progressives who think he hasn’t gone far enough.

Commenting on MSNBC today, White House advisor David Axelrod said they don’t look at the polls (though he was familiar with this one). He said “Washington thinks every day is election day”, and the Obama group doesn’t take that approach. They see a problem and address it without concern for the political fallout.

Fact is, virtually every day IS election day for Congress – with elections every two years for ALL of House and 1/3 of Senate.

That’s why everyone pours over the latest poll results and bases the future of America on them. Here’s one that seems interesting - 49% said they would like to get a new representative, 40% said they would “keep the one I have.” I thought that was encouraging – the higher the number in favor of change, the better – but pundits on both sides claim this one as good news for their side. Truth is, it turns out the numbers have been about the same on nearly every poll since 1992! And incumbents almost always get re-elected.

Cross your fingers for change, but don't bet the kids' college funds on this one.

JM