Skip to content

Professional Development

Evaluate a survey before acting on its headline result

A large response count does not tell you whether a survey answers your decision. Check who responded, what was asked, how results were analyzed and what action the evidence supports.

Three teal envelopes sit beside a tray of many cream envelopes.

A survey headline can make a decision seem settled: staff prefer a different schedule, customers want a new service or professionals expect a major industry shift. Before acting, establish what the survey actually measured and whose views its results can reasonably describe.

Begin with the decision you face. Evidence that people express interest in an idea is different from evidence that they will use it, pay for it or benefit from it. A survey can be useful without answering every question needed for a permanent policy or substantial investment.

Find out who could respond

Look for the target population, recruitment method, fieldwork dates and number of respondents. Was a defined group sampled, was everyone in a workforce invited, or could anyone follow an open link? Ask who was unlikely to receive the invitation or able to ignore it.

For example, an employee survey promoted mainly through an office-based communication channel may miss colleagues who work away from a desk. A large response count does not resolve that coverage question. Even an invitation sent to everyone can produce an unrepresentative set of respondents if participation differs systematically.

Pew Research Center’s September 2023 methodological study compared six online US adult samples collected in 2021. The three probability-based panels had lower average error across the selected benchmarks than the three opt-in sources. That result concerns those methods, samples and benchmarks; it is a reason to examine recruitment, not a guarantee that any probability sample is accurate.

Read the question and its denominator

Request the exact wording, response options and any screening questions. Was the question about a preference, an experience or an intention? Could respondents select several answers, choose none or say they did not know? Check whether the headline percentage describes everyone surveyed or only a subgroup.

A question asking whether a new service sounds useful may produce a different answer from one specifying its price, eligibility and practical requirements. Avoid silently transferring the first result to the second decision. If an important condition was omitted, make the uncertainty visible.

Check dates and comparison periods too. An opinion collected during a temporary disruption may still be informative, but it is not automatically a stable preference. Before calling a change a trend, confirm that question wording, recruitment and relevant conditions were comparable across surveys.

Separate precision from accuracy

Sampling uncertainty is only one source of error. Question wording, nonresponse, coverage and processing can also affect results. Ask how weighting was performed and which population information it used; weighting can address some differences without proving that all important biases have disappeared.

A 2021 Nature study by Bradley and colleagues examined large US vaccination surveys against a contemporaneous administrative benchmark. It found that very large samples could still give biased estimates with narrow reported uncertainty. This historical example illustrates why additional responses alone cannot repair an unrepresentative design; it is not current evidence about vaccination rates.

If a report provides confidence intervals or other uncertainty estimates, read the method and assumptions behind them. Do not apply a simple random-sample margin of error to an open-link poll as though the recruitment process were equivalent. Ask an analyst to explain uncertainty that the published method leaves unclear.

Match the finding to a proportionate action

Statistical significance is not the same as practical importance. The American Statistical Association’s 2016 statement says that significance does not measure effect size or a result’s importance, and decisions should not rest on a threshold alone.

Before deciding, write a short evidence record:

  • Supported finding: What can we say about which people, question and period?
  • Missing evidence: Which groups, conditions or behavioral questions remain unresolved?
  • Decision relevance: Is the difference large enough to matter for the choice we face?
  • Next action: Do we need clarification, another source, a limited trial or a wider decision?

An exploratory staff poll might justify discussion and a reversible scheduling trial while remaining insufficient for a permanent change. A customer survey may need to be compared with actual service use. Record both the finding and its limits when briefing decision-makers. The useful outcome is an action whose scale matches the strength and relevance of the evidence.

More to read

Browse insights