Metro's new Opt In project has been pitched as a relatively inexpensive way to get statistically valid opinions from large and diverse constituencies.
But in its first survey conducted side-by-side with a statistically valid sampling, the Opt In panel came back with results far outside the margin of error.
The survey, conducted in partnership with DHM Research, asked participants about climate change. Its results were released Friday, at Metro's Climate Leadership Summit.
The Opt In panel's results varied by as much as 40 percent from DHM's phone survey. The disparity highlights a lack of diversity on the Opt In panel, which Metro launched in January.
More than 70 percent of Opt In participants are from Multnomah County; the panel is 90 percent white and fewer than 3 percent of Opt In participants didn't finish high school.
Of the 1,626 participants who took the most recent survey, 91 percent were white, 73 percent were from Multnomah County and 83 percent finished college.
So how were the disparities evident in the answers?
Seventy one percent of Opt In participants said they supported government building more compact neighborhoods, compared with 36 percent of phone survey participants. Opt In members were nearly three times as likely to support raising the gas tax to "discourage some behaviors related to transportation." And the online panel was 33 percent more likely to believe that greenhouse gas emissions led to climate change.
In some areas, Opt In's results were close to those from the phone survey. Support for tax incentives, for example, was generally within the phone survey's margin of error. But the survey showed Opt In has a long way to go before it can give policymakers a reliable snapshot of the mood of the Portland region.
"We have the people we often hear from," said Patty Unfred, a communications manager and Metro's Opt In coordinator. "That's not good enough."
Unfred said targeted outreach continues toward underrepresented communities, including minorities, suburbanites and Republicans. Until the demographics start to even out, Metro will continue to "test" the results with statistically valid samples.
Opt In has two key advantages versus phone surveys, Unfred said. One is the cost – a phone survey can cost more than $20,000, where Opt In surveys start at about $4,000 each.
Plus, she said, "Opt In provides an opportunity to engage people in long-term discussions and offers ways they can actually participate."
Results of the Opt In climate change survey are posted on the panel's website; results of the phone survey will be posted later today.