BLOG: Coverage or Candour? Survey-Mode Trade-offs for Studying Wartime Ukraine

Aaron Erlich (McGill University)

Russia’s full-scale invasion has created enormous demand for survey data on what Ukrainians think and do. Yet, the choice of survey mode quietly shapes the answers. Drawing on two surveys fielded simultaneously in the summer of 2022, I show that telephone and web surveys each carry a distinct bias. Choosing between them means deciding which kind of error a study can better tolerate.

Two biases, one war

Since February 2022, scholars, journalists, and policymakers have all wanted to know what Ukrainians think and how they behave under full-scale invasion. Meeting that demand means confronting a question that is easy to overlook: how the data are gathered shapes what we learn. Particularly in the full-scale invasion’s early days, face-to-face interviewing was largely infeasible in wartime Ukraine, and it remains difficult and costly. This leaves two realistic options: telephone surveys and online (web) panels. Each of these modes is vulnerable to a different problem.

The first is coverage bias: the survey cannot reach parts of the population it is meant to describe. The second is social desirability bias: respondents shade their answers towards what they believe is expected of them. War sharpens both. Displacement, occupation, and refugee flows scramble the very population a survey is trying to represent, while intense pressure to appear committed to the national cause gives respondents good reason to present themselves as model patriots.

A concurrent two-mode study

To see how much these biases matter — and whether they differ by mode — I ran two near-identical surveys at the same time (25 July to 1 August 2022), each on an independently drawn sample of Ukrainians in government-controlled territory. The telephone survey used random digit dialling and targetted the full adult population (n = 1,206). The web survey drew on an opt-in online panel; crucially, the provider could not supply rural respondents or anyone over 55 at the time of the study, so it covers only the non-rural population aged 18–55 (n = 1,192). The same firm fielded both, holding constant the many other choices that distinguish survey research firms. The anonymised data and replication code are openly archived on the Harvard Dataverse.

Who gets left out: coverage bias

The web panel’s blind spot to rural and older Ukrainians is not a minor footnote. Before the invasion, roughly a third of Ukrainians lived in rural areas and over a third were aged 55 or older; these are precisely the groups online panels miss. And the gap is not random noise. Because the telephone sample also reaches older and rural Ukrainians, it yields meaningfully different estimates on questions central to understanding the war: the telephone survey records fewer displaced respondents (14% against 21% in the web sample), more Soviet-era military veterans, and higher levels of concern about behaviours such as draft evasion.

There is, however, a reassuring result. When I use statistical matching to make the telephone survey’s respondents comparable to the non-rural population under 56, most of these differences disappear. Overall, the web panel is not “wrong”; it simply describes a narrower slice of Ukraine. If a research question genuinely concerns younger, urban Ukrainians, an opt-in panel may serve well. If it concerns the population as a whole, only telephone surveys reach a broad cross-section. Matching to a subpopulation one can directly observe, as above, is feasible; weighting a survey to match the national population is not — Ukraine’s last census was in 2001, and reliable wartime population figures are now a national-security matter. That puts still more weight on the choice of mode.

Telling the truth about volunteering

Coverage is only half the story. To probe social desirability bias, I embedded a pre-registered framing experiment in both surveys, focused on wartime volunteering, which is an activity Ukrainians have taken up on a remarkable scale, and one heavy with patriotic meaning. Respondents reported whether they had taken part in any of fifteen volunteer activities. I randomly assigned each respondent either to a neutral control wording or to one of two “honesty” frames: one gently signals that respondents need not exaggerate, the other notes that people sometimes claim activities they did not perform.

The pattern is striking. The frame that simply lowered the pressure to exaggerate cut the total number of volunteer activities respondents reported by about 16%, but only in the telephone survey. The web panel showed no such drop. This gap holds when I re-run the analysis on the same matched sample used to probe coverage bias: the telephone effect persists at a magnitude consistent with the main model, so it reflects the mode of interview rather than the two samples’ differing composition. This fits a wider literature: the personal, interviewer-mediated nature of a phone call invites respondents to perform civic virtue in a way an anonymous web form does not. The second frame, which hints that respondents may be over-claiming, did not work, and may even have backfired in wartime by provoking an emotional reaction to the question itself — though this last reading is speculative.

What this means for data in the region

The headline is a genuine trade-off. Telephone surveys cover the population well but are more exposed to social desirability bias; web panels systematically miss older and rural citizens, though preliminary evidence suggests they elicit less over-claiming of socially approved behaviour. Neither is the single correct mode, and a mixed-mode design — telephone for population coverage, web for sensitive items — can, in principle, capture some of each mode’s strengths, though it brings its own complications. The right choice depends on which error is more damaging for a given question, and researchers working across conflict-affected parts of Eastern Europe, the South Caucasus, and Central Asia face this decision constantly.

Two practical points follow. First, scholars should ask survey providers pointed questions about who their panels can and cannot reach, especially in wartime, and report those limits plainly. Second, sensitive-question techniques deserve a routine place in wartime survey design, particularly for telephone studies. The broader message, though, is encouraging: with careful attention to mode, high-quality opinion and behavioural data can be collected even under invasion. Making such data openly available, together with honest documentation of its limits, is exactly the work that platforms like Discuss Data exist to support.

This post draws on my article “Two Implications of Survey Research Mode during War: Evidence from Russia’s Full-Scale Invasion of Ukraine”, published in Post-Soviet Affairs (https://doi.org/10.1080/1060586X.2025.2529114).

About the author

Aaron Erlich is an Associate Professor in the Department of Political Science at McGill University and a member of the Centre for the Study of Democratic Citizenship and Centre on Population Dynamics. His research focuses on post-Soviet politics, research methodology, corruption and transparency, and the politics of information provision.

Dicuss Data Blog

Coverage or Candour? Survey-Mode Trade-offs for Studying Wartime Ukraine