Statistical Weighting in Online Surveying

Did a quick blurb over on Jason Cherniak's site in response to this entry on polls in Canada showing that the Conservatives are wildly ahead of the Liberals.

Cherniak's problem is that they were online surveys, and the Internet ain't exactly the most demographically representative place in the world. People pointed out (rightly enough) that you can fix that with weighting.

I responded in some detail, but it broke down to two things that bother me about online surveying:

1) The Internet's somewhat abstracted nature means that people may be less likely to either tell the truth or carefully consider their comments. Anybody who's read a blog (or Usenet, or a web forum, or pretty much anything here, really) knows that people often post at length and repent at leisure, because of the prevailing belief that nothing that goes on here really matters.

(That's at the root of good ol' Ted Barlow disease, which is bloggers quitting because they believe that what they write doesn't really matter. I can commiserate. But I digress.)

That could mean that people are less likely to respond carefully and honestly to a poll for the precise opposite reason that they lie on other polls: instead of being too concerned about what it says about them, they'll think it doesn't matter at all. They won't carefully consider the questions or their answer. They'll just vote for the rhinoceros party because it's funny.

2) Demographic problems can be avoided through weighting specific socio-economic groups. The problem, though, is that weighting can massively increase the likelihood that misrepresentation of those specific groups can take place. If you take a poll and there are something like two people from a specific small (but significant) group, no matter HOW you weight them you're going to have problems. If, for example, poor people on the Internet are disproportionately inclined towards conservatism compared to the poverty-stricken population as a whole, and they're overweighted, it's going to REALLY screw with your poll.

And, yes, conservatives really are overrepresented on the Internet. That's the reason this site was created, remember?

Oh, and 3) The respondees are inherently self-selected, even more than in phone and mail polls. You can't weight to correct for that, either.

I don't think phone polls are perfect, of course. Neither are mail polls (the response rate for mail polls is abysmal.) But I really, really don't trust online polling. Not when Zogby was doing it and it showed Kerry way ahead in 2004, and not today with Angus Reid doing it.