A quick follow-up to my discussion yesterday of the poll from the American Research Group (ARG) in South Dakota.
ARG had the winners right in South Dakota and Montana, but that was about it. If we compare the margin between the top two finishers on the poll to the margin in the vote count (with 100% of the precincts reporting according to AP this morning), ARG had errors of 11 points on the margin in Montana and 14 points in South Dakota. Those were not ARG's worst misses of the primary season -- their final polls were off by more in Connecticut, Iowa, Illinois and South Carolina -- but these last two polls were a bit worse than their average (9) for the Democratic primaries.
Looking at all the polls in both Democratic and Republican primary contests, a SurveyUSA report card compiled in early May showed ARG with a median error of 7.0, ranking them 26 on the list of all 41 pollsters, and dead last among the nine organizations that polled in 10 or more contests this year (remember due to "regression to the mean," average errors can be much larger or smaller when a pollster does only a handful of surveys -- as the number of polls averaged goes up, the error should go down).
The most striking aspect of ARG's difficulties during this primary season was the way their surveys tended to err in Hillary Clinton's favor. We first noticed this pattern in their surveys of Iowa last summer. While not totally consistent, the ARG polls tended to show Clinton doing a few points better in Iowa than other pollsters, although at the time the pattern looked like a consistent difference, but not necessarily an error. After the final ARG poll showed Clinton leading Obama by nine percentage points (she finished third), ARG pollster Dick Bennett lashed out at critics and the "deeply flawed" Des Moines Register poll (that had correctly forecast an Obama win).
Bennett might have done better to examine his own methods, because in the primaries that followed, ARG's final poll erred in Clinton's favor in 19 of 27 contests, averaging 6.8 percentage of error on the margin in Clinton's favor. The odds of that happening by chance alone are extremely remote.
Only Bennett is in a position to explain why his surveys were further off the actual result than those of other pollsters, and why his surveys tended to err in Clinton's favor. To be clear: I am not suggesting that Bennett had a pro-Clinton agenda. Rather, I think the answer has something to do with some aspect of ARG's methodology. Unfortunately, since ARG, like all too many public pollsters, tells us so little about their methods, we can only guess.