General population surveys are currently challenged by several societal developments, such as budget constraints and the respondents’ more active lifestyle, which leads to a lower contact success rate and higher costs in interviewer-based survey designs. At the same time, internet penetration rates are increasing fast and steadily. General population surveys are therefore pushed to innovating the design and several experiments on different designs for fielding a general population survey on the web have been fielded for more than a decade now. Survey methodologists study mode effects between interviewer-based and web/paper self-completion for over a decade. For example, Switzerland has fielded a comprehensive mixed-mode experiment using the European Social Survey (ESS) in 2012, a complex experiment on push-to-web designs using the European Values Study has been fielded in six countries in 2017 and during the pandemic, the ESS has been fielded as a self-completion web/paper survey in several countries in 2021. Given the change towards self-completion of the ESS in 2027, several experiments based on the ESS questionnaire have been fielded or are in the field.
This session welcomes contributions that show the effects of a mode switch on results of general population surveys with a special focus on changes over time. This includes two types of research questions: effects of a mode-switch on time-series data as well as the changes in effects of a mode-switch over time. The first type of research questions includes how to demonstrate a mode effect in a time-series, how to correct for mode effects, how to visualize time-series with a mode change in-between and many more. The second type of research questions includes changes over time in under- or overrepresentation of specific groups in the population, changes in, or persistency of, mode effects regarding some variables or change in the share of paper vs. web participation, mobile participation etc. We welcome contributions based on ESS data but also based on any other general population survey that provides insights into the effect of switching from an interviewer-based to a self-completion survey relevant to the mode-switch of ESS foreseen in 2027 (e.g., including items and concepts used also in the ESS, such as trust, attitudes towards democracy, immigration, family, or welfare state).
|
Switching the mode from face-to-face to web/paper mixed mode: Comparability of real-life analyses across designs and questionnaire lengths nationally and cross-nationally
Michael Ochsner, Michèle Ernst Staehli, Alexandre Pollien
FORS, Switzerland
Several reasons push cross-national survey infrastructures to switch the mode from interviewer-based to self-completion modes. However, it is still not clear how to design a self-completion survey that is comparable to its original interviewer-based version. In this presentation, we investigate the comparability of different designs with a particular focus on questionnaire length of a cross-national survey, i.e., the European Values Study 2017. As there are often concerns about the length of web/paper mixed mode surveys, a complex experimental design was fielded paying special attention to questionnaire length, including a matrix design and splitting the questionnaire into two parts. We present the results of several analyses according to a framework to analyse comparability of results across designs taking into account response bias, representation bias, bias regarding variables of interest as well as order effects (when splitting the questionnaire) for different use-cases of survey data (e.g., descriptive analysis often applied by policy makers or the media as well as more complex models applied in scientific contexts).
We complement the analysis by a comparison across countries. Using data from Germany and Iceland, where the matrix design was also implemented concurrently with a face-to-face survey, we present insights into how to analyse data from a matrix design and into comparability of switching a mode across countries.
The results suggest that differences across modes and designs are generally small even though differences can occur. We also find that applying a matrix design comes with several complications. While an analysis is possible and the results are generally comparable to the other modes, the advantage of shortening the questionnaire considerably through a matrix design does not pay off given the reduction in data usability and increasing the complexity in the analysis of the data. We conclude that switching general population surveys from face-to-face to web/paper mixed mode is possible, at least for countries with a sufficient internet penetration and a high-quality individual register for the population. We discuss how decisions on the design and questionnaire length might influence comparability of results across modes.
The effects of the online and paper survey modes in the ESS 10 self-completion studies
Dragan Stanojević2, Bojan Todosijević1
1Institute of social sciences, Serbia; 2Institute of sociological research, University of Belgrade
ESS10 Self-completion - integrated dataset (ESS10SC, edition 3.1) contains nine studies based on self-completion rather than face-to-face interviews, as has been the rule in previous rounds. In each study, respondents were allowed to choose a preferred mode of questionnaire completion - either to complete an online questionnaire or to request a printed questionnaire and mail it back. Methodological research suggests that the survey completion mode may be consequential for various aspects of the data collected. This ESS dataset represents a unique cross-national data source to study the role of the self-completion mode. In this paper, we compare the online and paper self-completion data components guided by the question of to what extent the two components complement each other, producing a joint dataset of superior quality, or perhaps they augment certain deficiencies or are redundant. We will first compare distributions and point estimates of the standard socio-demographic variables. For instance, the two components differ in gender composition and the average age in most studies. We will examine some bivariate and multivariate associations using theoretically informed substantive models. This will show whether the same conclusions about variable associations would be reached whether one used online or paper mail-back questionnaires or jointly both. The final step will focus on measurement equivalence regarding some of the multi-item operationalizations available in the ESS10 dataset.
The results would provide helpful methodological evidence for the future use of the ESS self-completion data and other studies that adopt this multi-mode approach.
The European Values Study 1981-2026: from face-to-face to self-completion
Ruud Luijkx1,2,3
1European Values Study; 2Tilburg University; 3University of Trento
The European Values Study (EVS) was first conducted in 1981 and then repeated in 1990, 1999, 2008, and 2017, with the aim of providing researchers with data to investigate whether European individual and social values are changing and to what degree. The EVS is traditionally carried out as a probability-based face-to-face survey that takes around 1 hour to complete. In recent years, large-scale population surveys such as the EVS have been challenged by decreasing response rates and increasing survey costs. In the light of these challenges, six countries that participated in the last wave of the EVS tested the application of self-administered mixed-modes (Denmark, Finland, Germany, Iceland, the Netherlands, and Switzerland).
In this contribution, we will present the implemented mode experiments from the last EVS- wave (2017) and sketch the infra-structural challenges to move from face-to-face to self-completion. Special attention will be given to the Dutch situation and the national cooperation between ESS, ISSP, GGS and EVS. It is pivotal for data use in substantive research to make the reasoning behind design changes and country-specific implementations transparent as well as to highlight new research opportunities that will emerge when surveys cooperate and will use probabilistic web panels..
Transitioning from interviewer-based to self-completion modes: Implications for the quality of measures of political behavior
Nursel Alkoç1, Michèle Ernst Stähli2
1University of Lausanne, Switzerland; 2FORS, Switzerland
ESS is one of the most important data sources for political scientists working with survey data, because it is less subject to the self-selection of politically interested respondents and has so far generally provided better estimates of political behavior in Switzerland than the national election survey. As a response to the Covid pandemic, data collection in some countries, though not yet in Switzerland, has shifted from face-to-face interviews to a self-completion design from ESS Round 10 onwards. This paper examines the possible implications of the mode change in the next rounds of ESS in Switzerland with respect to the quality of measures of political behavior. We draw our analyses upon mail, CATI, and web data of the LIVES-FORS mixed mode experiment and CAPI data of ESS 2012. The results show that the paper survey includes less politically interested respondents and provides more accurate estimates of political participation than the other survey modes, albeit insignificantly. To better understand the implications of the mode change in ESS, additional analyses were conducted with the Measurement and Observation of Social Attitudes in Switzerland (MOSAiCH), which has the same quality standards as ESS and switched from a face-to-face survey to a self-survey mode already in 2018. The findings suggest that a shift to self-administered questionnaires in ESS could provide estimates that are of the similar quality as the CAPI data in terms of measures of political behavior, and in some cases even more accurate.
|
