Conference Agenda

Education is one of the most often used variables in social science research based on survey data. It shows net correlations with more attitudinal outcomes than any other background variable commonly controlled for and is a core variable in social stratification research. At the same time, it is the most difficult variable to measure and harmonise in cross-national surveys since educational systems are highly complex today and differ substantially across countries.

Since ESS round 5, the ESS has used a complex education coding scheme, 'edulvlb', extending the International Standard Classification of Education 2011 to harmonise measures of educational attainment across countries. While this led to highly informative and comparable data, it also entailed fairly long lists of response options in many participating countries, with 12 to 26 categories. In light of the switch from face-to-face interviews to self-administered survey modes, there are concerns that such long lists will be too burdensome for respondents and may lead to substantial amounts of item nonresponse or even break-offs.

This paper explores the possibility of simplifying the detailed education coding scheme to alleviate the requirements placed on country-specific survey questions on educational attainment, without compromising validity and comparability. We performed a comprehensive validation analysis using ESS rounds 5 to 9 data to ensure this. We used a data-driven approach to compare the explanatory power of 17 competing specifications for the comparative education coding when predicting almost 400 dependent variables. To assess validity, we looked at the relative losses of partial adjusted R-squares of education compared to the most complex education variable 'edulvlb' across all validation variables. Ideally, they should be as small as possible. To assess comparability, we looked at the distribution of these losses across countries. Ideally, they should be equal.

Regarding validity, results show the highest relative validity for a variable that does not distinguish between general and vocational education but keeps the detailed sub-levels defined by 'edulvlb', which go beyond those defined by ISCED 2011. Regarding comparability, results show the lowest variation in the losses of explanatory power for ISCED 2011 with all three digits, thus keeping the distinction of general and vocational education but dropping the additional sub-levels that 'edulvlb' added to the official ISCED.

Based on these results, a coding scheme was developed for ESS rounds 12 and beyond that corresponds to the three-digit version of ISCED 2011. It only additionally summarises a few internationally rare categories to allow for further simplifications of country-specific measures. It also adds one distinction needed to derive the more aggregated education variable 'ES-ISCED' available in the ESS. The presentation will conclude with an outlook on the (then) ongoing country consultation aiming to optimise country-specific education measures for self-administered survey modes while allowing a high-quality international harmonisation.

In face-to-face surveys, the presence of interviewers can lead to bias during both the recruitment and data collection stages. Focusing on the latter, the primary theoretical rationale for the impact of interviewers on responses is social desirability bias; i.e. when individuals modify their answers to appear more favourable in front of others, rather than sharing their genuine opinion or feelings. Accordingly, a line of studies reported high intra-interviewer correlations when measuring sensitive questions. Interviewer effects are known to be associated with different interviewer characteristics (e.g., gender, age, race), but some studies found that the interviewer’s position on the issue or attitude in question can also explain interviewer effects. Against this background, this study aims to deepen our understanding of the development of interviewer effects in face-to-face surveys. During the 11th round of the Hungarian European Social Survey (fielded between May 2023 and November 2023) we asked the interviewers who worked on the data collection to fill out the same questionnaire as the respondents. Thus, linked responses from interviewers (N=96) and respondents (N=2204) are available. We use multilevel models to assess what interviewer-level factors are associated with the respondents’ substantive answers (socio-demographic characteristics vs. substantive responses, 1) and which types of questions are more prone to interviewer effects (2). The results will be presented at the conference.

The COVID-19 pandemic has prompted a shift towards self-completion protocols to adapt to the challenges posed by social distancing and public health concerns. Nine countries participating in the most recent 10th round of the European Social Survey (2020) could not collect data face-to-face, switching to sequential self-completion modes, with a preferred web-first approach (i.e., an initial invitation to complete the online survey only, with paper questionnaires sent with a later reminder). We investigate whether this transition has compromised the quality of ESS10 results by examining the impact of self-completion approaches on sample bias, response styles, and data missingness. We compare ESS pre-COVID round 9 (2018) and round 10 data collected in Austria, Cyprus, Germany, Latvia, Poland, Serbia, Spain, and Sweden. First, investigating sample bias, we explore whether the shift to self-completion protocols introduces systematic differences in respondent characteristics. By employing internal criteria of representativeness and external benchmarks of population proportions, we aim to discern any disparities in demographic composition and socio-economic status between self-completed and interviewer-administered surveys. In addition, we also contrasted ESS R10 respondents surveyed via self-completion web- and paper-based questionnaires. Secondly, we examine response styles, scrutinising the impact of self-completion on response patterns and investigating whether individuals exhibit varying levels of response styles when completing surveys autonomously. By employing established measures of straightlining response styles, we aim to elucidate whether the absence of an interviewer influences the consistency and accuracy of participant responses. Furthermore, we explore the issue of data missingness. Our analysis investigates whether the switch to self-completion exacerbates the problem of missing data, potentially compromising the overall reliability and validity of the ESS results. Through sensitivity analyses, we aim to discern patterns of missingness and evaluate their potential impact on the interpretation of survey findings.

The eleventh wave of ESS in France provided the opportunity to use CAWI as a new data collection method for the first time in the country, in addition of the usual face-to-face survey.

In this push to web mission, 4,000 individuals were randomly selected, for an expected response rate of 25% (1,000 respondents). These individuals were invited to complete the survey online, receiving an unconditional €5 gift voucher. They were then contacted up to 3 times by postal letters. For the second reminder, a paper questionnaire was included in the letter, so that respondents could complete the survey by paper if they preferred. An additional €10 gift voucher was promised to all respondents. Out of 1000 respondents, 200 paper questionnaires were expected.

Even though the collection phase is not yet complete (collection ends in March 2024), the CAWI and PAPI response rates has proved to be highly satisfactory, even exceeding initial projections (+25% response rate).

An additional source of satisfaction is that the novelty of this mode of collection enabled us to set up a methodological experiment which is already proving highly instructive.

2 distinct protocols were set up:

• In the first protocol (A), individuals were only contacted by postal letter (invitation and reminder letters).

• For the second sample (B), these letters were supplemented by e-mails and SMS messages.

The results of this experiment are already very encouraging, with a +6 points difference of response rate in favor of the second contact protocol. Differences in term of response rate are also observed between the different wave of reminders.

The aim of this presentation will be to review the methodological experiments carried out during this auto-completion survey. Particular attention will be paid to the effect of additional e-mail and telephone contacts on the response rate. This analysis will be complemented by a detailed analysis of the effects of collection methods (CAWI / PAPI vs CAPI) on respondent profiles and on the content of responses obtained (desirability bias in particular).

As the European Social Survey (ESS) transitions from face-to-face interviews to web-based surveys, a critical concern is whether data collected in these different modes are comparable. Responses might vary by completion modes for several reasons. First, responses might be more highly correlated in one completion mode than in another completion mode. For instance, when completing a survey in front of an interviewer, respondents may be more motivated to appear consistent. Second, response profiles may be more extreme in one completion mode more than in another completion mode. For instance, to the extent that response options are processed differently when presented on cards versus on a computer screen, responses might be biased by a primacy effect (selecting the first response option) or a recency effect (selecting the last response option), creating more responses to the endpoints of scales. Thus, aggregating data across completion modes requires establishing whether responses are comparable across completion modes.

We address this question using responses from 144,212 respondents to 49 questions in rounds 8-10 of the ESS. We test whether a machine learning algorithm can predict whether responses came from respondents who completed the survey face-to-face, versus respondents who completed the survey using self-completion modes.

The completion mode in Wave 10 is fully confounded with respondents’ country of origin. In particular, in no country was there more than one type of completion mode. Consequently, any machine learning analysis between different completion modes within Wave 10 might be driven by country of origin rather than by completion mode. To address this confound, the main analysis was between the 9 countries with a self-completion mode in Wave 10, versus those same countries in Wave 8 and Wave 9 (we selected two waves to more closely match the larger sample sizes in Wave 10, as well as to include countries with no data from Wave 9). However, this introduces a confound of time: differences between the two completion modes may not be driven by the completion mode, but by the later time of Wave 10 (compared to Waves 8 and 9). To address this, we quantified the contribution of time to the predictive power of the model by running a separate model comparing the 16 countries which had an interview completion mode in Wave 10, versus those same 16 countries in Waves 8 and 9.

We show that machine-learning algorithms using LightGBM can predict how surveys were completed, demonstrating that data collected in the different modes are not comparable. We demonstrate that this is partially driven by the aforementioned differences in the response profiles of the two different modes, including response consistency across items and extreme responses. In addition to addressing concerns of data comparability with the ESS, these findings reveal that completion modes of surveys have real consequences for response profiles.