/ SAT-PRL-A2-D.10: 1
Submit your abstract
Panels: D.10. Producing and Using Evaluation Evidence to Improve Equity in EducationKeywords: scholarship, social inequality, evaluation, students' attainment
THE EFFECT OF FINANCIAL AID ON THE STUDENTS' UNIVERSITY CAREER: EVIDENCE FROM A NATURAL EXPERIMENT IN PIEMONTE
1University of Trento, Italy; 2FBK-Irvapp
The aim of this paper is to assess the causal effect of the Right to Education scholarship on the performances (marks and credits), persistence (i.e., avoiding drop-out) and graduation of students enrolled in the Polytechnic of Turin in the academic year 2011/2012. The aim of this kind of policies is to foster university achievement of students from low-income families and thus reducing inequalities. We choose this particular year, because in Piedmont up to the academic year 2010/2011 all the eligible students received the grant, while in 2011/2012, due to a cut in the funds, the recipients were only the 30.8% of the eligible students. This state of affairs permits the identification of the causal effect of the programme comparing the group of recipients with the group of those eligible but non-recipients, according to the 2011/2012 rankings for the first and subsequent years of any full-time undergraduate course of the Polytechnic of Turin.
For the evaluation purposes, we use administrative data supplied by the Polytechnic thanks to an agreement with EDISU Piemonte. The identification strategy is based on the Regression discontinuity design (RDD). By applying RDD we (reasonably) assume that students just below and students just above the eligibility threshold are equivalent and thus fully comparable. Due to the presence of non-compliance, we rely on a non-parametric fuzzy RDD. More precisely, this problem is solved using instrumental variables leading to LATE estimates.
The analysis provides two main results. No scholarship effect on performances and persistence when the above and below threshold groups are selected based on their family’s ISEE. No effect when comparing groups with the same prior academic performances and same money constraints to deal with. These results contradict previous studies on the effectiveness of the Right to Education that pointed out a positive effect of this programme on persistence and graduation and seem to agree with the position of Heckman and Carneiro that consider financial aid at his stage as a waste of money. However, we do not believe that this analysis is a challenge for the liquidity constraints theory. Indeed, we have to take into account that in those years in Piedmont (and in Italy) there was a declining trend in the enrolment rates for the negative consequences of the economic crisis. Moreover, we are considering the Polytechnic that comprises usually very demanding faculty. It is realistic to suppose that in those years the students who chose the Polytechnic were very motivated and probably from not disadvantaged socio-economic backgrounds. This sort of self-selection into demanding course of study could be an explanation for the zero-effect result. This is not the only possible explanation. In another possible scenario the budget cuts led to a situation in which only the students with either very low economic conditions or with relatively high previous performances could receive the scholarship. Those students cannot rely on their families if the scholarship does not cover all the expenses. Consequently, students may have to find a job to bear the additional costs.
/ SAT-PRL-A2-D.10: 2
Submit your abstract
Panels: D.10. Producing and Using Evaluation Evidence to Improve Equity in EducationKeywords: Equity, school education, education systems, Eurydice, European Commission
EQUITY IN SCHOOL IN EUROPE: STRUCTURES, POLICIES AND STUDENT PERFORMANCE
Education and Culture Executive Agency of the European Commission, Belgium
In support of evidence-based policy-making, Eurydice (a European network managed by the Executive and Culture Agency of the European Commission) published in October 2021 the report Equity in school education in Europe: Structures, policies and student performance. This report examines a range of key education policies and structures and assesses how they affect the levels of equity in education systems. Equity is examined along two dimensions: inclusion (in terms of achievement gap between high and low achieving pupils/students) and fairness (in terms of the correlation between socio-economic status and student achievement). The report draws on three types of data: original policy information collected from the Eurydice national units, international survey data on student performance and characteristics (PISA, PIRLS and TIMSS), and statistical data collected by Eurostat. Using bivariate and multivariate statistical analysis methods, the report evaluates the impact of these system-level features on educational equity, individually and in combination. It covers 42 education systems across 37 European countries.
The presentation provides an overview of the main findings that emerge from the report. First, it reviews levels of equity and academic segregation in primary and secondary education using data from international student surveys. It then briefly describes data on a number of system level features (public funding, differentiation and school types, school choice and school admission policies, tracking, grade repetition, school autonomy and accountability, opportunities to learn, support to disadvantaged schools and support to low-achieving students) that could potentially influence equity in education. Finally, the brief discusses three multivariate models that identify the policies and structures associated with higher levels of equity.
The quantitative analysis results show that inclusion in primary education is dependent on the level of public funding and the size of the government-dependent private school sector. Inclusion in secondary education is dependent on the level of grade repetition, the size of the vocational sector and the age of first tracking. Equity in terms of fairness depends on the same factors as inclusion in the secondary level with the addition of differences in school admission and choice policies.
/ SAT-PRL-A2-D.10: 3
Submit your abstract
Panels: D.10. Producing and Using Evaluation Evidence to Improve Equity in EducationKeywords: Formative assessment, in-service training program, learning outcomes, quasi-experimental research.
PROFESSIONAL DEVELOPMENT ON FORMATIVE ASSESSMENT: A QUASI-EXPERIMENTAL STUDY IN A MIDDLE SCHOOL
University of Bologna, Italy
Learning data from international and national surveys reveals an urgent need to find strategies to reduce educational attainment disparities. Evidence-based education studies have pointed out the efficacy of formative assessment strategies in raising higher learning results (Hattie, 2009; Black & Wiliam, 1998). The very question, exploring this topic, involves finding ways to change teachers’ practices and beliefs. KLT studies (ETS, 2010) have stressed the relevance of working together with schools and educators to find ways of implementing formative assessment strategies. The present research was aimed to verify a professional development program’s effects on students’ learning outcomes and teaching perceptions. A selected group of teachers of a middle school participated in an investigation whose central question was: “What is the impact of developing an in-service formative assessment training program in terms of students’ learning results and perceptions?”.
Adopting a quantitative research methodology, we designed a quasi-experimental research project. We chose which class and teachers would experiment with formative assessment practices for three years in agreement with the school’s staff, while the remaining classes constituted the control group. During this period, we planned frequent meetings organized in the following pattern: in the initial stage, the trainer explained the transformative function of assessment and the associated strategies; then, he gave teachers opportunities to make peer observations in classrooms and experience teaching video-analysis; in the last phase, he analyzed specific teachers’ classroom experiences. Two learning tests on reading understanding and math abilities were administered every year to control students’ outcomes: almost all of the tests’ items were consistent with PIRLS, INVALSI, and TIMSS’s constructs. To gather data about students' teaching perceptions, a questionnaire was employed and annually delivered; qualitative information was also collected by adding open-ended questions during its last administration. Using SPSS statistic software, tests were run to compare students’ learning quantitative data; open-ended questions were categorized to find recurring themes.
Results show that, after controlling for math pre-test scores, the intervention group’s class significantly outperformed the control group’s classes in the post-test (t 5.99 and p-value 0.02). Text comprehension tests’ results display a substantial increase in experimental pupils’ T-scores, but it is not statistically significant. Findings also reveal an averages growth of students’ teaching perceptions during the study’s development, although no significant difference was observed between classes. Considering open-ended questions’ responses, we can see that most cited themes concern the utility of formative feedback for students’ learning.
Prior studies have demonstrated formative assessment’s impact on learning gains, but little research confirms this evidence in the Italian context. National curriculum guidelines emphasize teachers’ use of formative assessment: thus, the present work could provide relevant suggestions for designing professional development programs. The project’s methodological limits include the COVID-19 emergency that has affected the post-test administration.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7-74.
Hattie J. (2009) «Visible learning: a synthesis of over 800 meta-analyses relating to achievements» Routledge, Taylor & Francis Group.
Educational Testing Service. (2010). About the KLT program. Princeton, NJ: Author.
/ SAT-PRL-A2-D.10: 4
Submit your abstract
Panels: D.10. Producing and Using Evaluation Evidence to Improve Equity in EducationKeywords: Financial aid, Academic performance, Working behaviours, Class attendance, Matching
THE EFFECT OF GRANTS ON DROPOUT AND ACADEMIC PERFORMANCE IN FOUR ITALIAN PROVINCES: EVIDENCE FROM A MATCHING APPROACH
1University of Trento; 2FBK-IRVAPP
Findings from related studies showed that providing economic support to students from more disadvantaged backgrounds allows them to reduce or stop working while studying and have the time to attend classes and be engaged with the social and academic activities related to university, which can lead to increased motivation, performance and attainment (Cabrera et al., 1992; Nora et al., 2006).
The aim of this work is twofold. The first is the evaluation of the impact of need-based financial aid promoted by the Italian government on first-to-second year dropout and academic performance in terms of credits accumulation and average exam marks. The second consists in understanding to what extent the effect of grant is due to the fact that it enhances class attendance or reduces student employment, two factors that are found to be associated to student success in higher education (Nora et al. 2006; Triventi 2014).
Data and method
Data come from the longitudinal survey of the project “Family background, beliefs about education and participation in higher education”, investigating the transition of senior high school students to higher education. The analyses are carried out within a counterfactual approach in order to simulate how grant holders would have performed in the absence of the grant. To identify a proper control group we follow a two-step procedure. First, within a fully non-parametric strategy, we exploit an imperfection in the grant assignment to define the set of eligible but not beneficiary students because of limited funds as the control group (Modena et al., 2018). Second, given possible remaining differences between treated and control groups (partly due to limited sample size), we use inverse probability treatment weighting (IPTW) to further balance the two groups of students on a number of observed characteristics. In the first part, we analyse whether receiving a grant affects credits accumulation and marks (at the mean and different thresholds) in the first year of studies. Second, we inspect whether receiving a grant modifies students’ working behaviours and class attendance, two possible mechanisms explaining the impact of grants on students’ outcomes.
Preliminary results indicate that financial aid has a positive and significant effect on credits accumulation, especially at policy-related thresholds, but not on marks. It seems that part of this effect is due an increased lecture attendance by students with a grant. Working behaviours instead seem not affected by receiving or not financial aid.
Cabrera, F., Nora, A., & Castaneda, B. (1992). The role of finances in the persistence process: A structural model. Research in higher education, 33(5), 571-593.
Modena, F., Rettore, E., & Tanzi, G. (2018). The Effect of Grants on University Drop-Out Rates: Evidence on the Italian Case. Bank of Italy Temi di Discussione (Working Paper) No, 1193.
Nora, A., Barlow, L., & Crisp, G. (2006). Examining the tangible and psychosocial benefits of financial aid with student access, engagement, and degree attainment. American Behavioral Scientist, 49(12), 1636-1651.
Triventi, M. (2014). Does working during higher education affect students’ academic progression?. Economics of education review, 41, 1–13.
/ SAT-PRL-A2-D.10: 5
Submit your abstract
Panels: D.10. Producing and Using Evaluation Evidence to Improve Equity in EducationKeywords: Replication, Randomized controlled trials, Teachers’ effectiveness
“TWO GUSTS IS MEGL CHE ONE”. INACCURATE RCTS’ REPLICATION AS A LEVERAGE TO GAIN EXTERNAL VALIDITY
University of Milan Bicocca, Italy
In two previous national RCTs, we tested whether delivering tips to teachers – aimed at better managing on job relationships – translates for them in higher self-efficacy and in better performances of their students.
We developed a light touch training program, using an innovative methodological approach that combined the potential benefits of a bottom-up developmental framework with those deriving from the reduced training intensity and implementation costs. The intervention took the form of a brief booklet and six short online videos dealing with several relational issues characterizing daily school routine.
The intervention delivery took place in three forms: individual, with only Italian and Maths teachers engaged in the professional development and met one by one (RCT1 – first arm); collegial, with all teachers collectively engaged in the intervention in a school meeting (RCT1 – second arm); individual but through mail boxes sent to school, called to deliver the booklets autonomously (RCT2).
RCT1 proved that the intervention was effective in producing a statistically significant impact on teachers’ self-efficacy, at the end of the first year and, that it had a positive impact also on pupils achievement, in the year following the booklet delivery.
RCT2 showed instead a null effect, suggesting that the intervention required to be properly delivered and promoted in schools, because otherwise self-selection into the treatment may undermine its effectiveness. This result raises questions about the possibility to extend the use of the booklet and these doubts sum up to the ones due to the limited external validity limits of RCT1, in example the fact that it was focused only on middle school teachers and that it did not cover all Italian regions.
In school year 2018/19, thanks to IPRASE, we had the opportunity to replicate the intervention in the province of Trento and to assess again its impact on teachers’ self-efficacy through a new RCT (RCT3). The intervention was targeted to beginning teachers in all school grades (204 randomized cases) and the delivery took the form of very brief collegial meetings. In this RCT, our impact estimate is only on teachers’ self-efficacy, measured through pre-post CAWI interviews. Results show that also in RCT3 the booklet was effective in raising teachers’ self-efficacy.
The replication of previous RCTs, despite inaccurate due to several differences between RCT1, 2 and 3, allowed us not only to confirm the intervention’s effectiveness, but also to extend its external validity. More precisely, RCT3 showed that the intervention’s positive effect may took place: a. in a pretty different school context; b. for teachers from all school grades; c. for beginning teachers; d. through a quite light and cheap form of delivery.
We argue that RCTs’ replication, even if inaccurate due to changing context/targets, in the lucky presence of converging results, may be a powerful leverage to gain external validity and to re-design implementation details of promising interventions.