Survey Methodology and Question Wording

When dealing with survey data, sample selection, question wording and the available answer choices can lead to dramatically different poll results. A poll run by YouGovPolimetrix leads to a similar conclusion about the difficulty of getting to reform, but the enormous amount of uncertainty within Californians could provide some optimism to the reform campaigns. Polimetrix uses an internet sampling matching procedure (PDF) of all Californians and the Field Poll’s methodology restricts its sample to registered voters. This is not a condemnation of one form of polling (internet vs. telephone) or sampling (residents vs. registered voters) compared to another; it is just something to think about when looking at polling results. The method employed and population the poll is sampling from matters when considering what inferences to make about the results.

Registered voters are disproportionally older and whiter than the population writ large. Additionally, California’s demographics are changing as Latino and Asian populations are growing quite rapidly. A policy brief (PDF) released by the Bill Lane Center for the American West shows that “New Californians” (Latinos and Asians) do have some measurable differences from the “Old Californians” (whites and African-Americans). Most notable are the New Californians’ increased levels of uncertainty about proposed constitutional changes, and these Californians tend to be more optimistic about the state’s future as well.

Question wording in the two polls is similar; however, the YouGovPolimetrix poll gave respondents the choice of “don’t know” for many of their questions. Obviously if one poll’s set of answers includes a “don’t know” response and the does not, we would expect differences in the amount of reported uncertainty. The surprising finding is that this ranges between twenty and thirty percentage points on some questions. There are good arguments on both sides of the debate about giving respondents the “don’t know” option, but one thing is clear –opinions about constitutional reform are not set in stone.

Of course the New Californians don’t vote at the same rate as the Old Californians. From the cynical perspective of campaign managers, why bother to educate and mobilize these New Californians when their citizenship and registration rates are drastically lower than the rest of the state and they are difficult to reach (often times exclusively using ethnic media), thereby making the potential electoral benefits expensive and uncertain. As this plays out across the state in the coming months it will be very interesting to see which side of the debate is more effective in educating all of California, but pay special attention to who wins the battle among those growing, hard to reach segments of the population, I have a hunch that side will prevail.