📣 Help Shape the Future of UKRI's Gateway to Research (GtR)

We're improving UKRI's Gateway to Research and are seeking your input! If you would be interested in being interviewed about the improvements we're making and to have your say about how we can make GtR more user-friendly, impactful, and effective for the Research and Innovation community, please email gateway@ukri.org.

Rethinking Media Literacy and Digital Skills in Europe (REMEDIS)

Lead Research Organisation: London School of Economics and Political Science
Department Name: Media and Communications

Abstract

Over the past decade, we have witnessed major transformations in access to digital media platforms in Europe. However, navigating this digitally mediated world can be challenging as it requires operational, social, content creation, and consumption skills that many citizens lack or insufficiently possess (EC, 2010; European Training Foundation, 2018). To date, researchers and practitioners are unable to make evidence-based claims about the impact of media literacy and digital skills (ML&DS) interventions and their positive outcomes: a rapid evidence assessment by Edwards et al. (2021) of the effectiveness of intervention strategies in the field of tackling misinformation illustrates that robust evaluation of ML interventions remains uncommon. The development of a responsive, forward-looking ML&DS policy in Europe lacks up-to-date knowledge and a deeper understanding of the ML&DS needs of citizens (young and older, specific social target groups). What should a media-savvy citizen be able to do and know? How can we determine what media and digitally literate individuals are like? If we do not have adequate answers to these questions, we will not know which interventions can be used to foster ML&DS or how successful these interventions are. REMEDIS's concept is to develop evidence-based approaches to further develop and evaluate initiatives that foster ML&DS to understand what the impacts of ML&DS interventions in different life domains are in terms of positive outcomes. In accordance with this concept, REMEDIS adopts an innovative research strategy that first aims to identify and quantify the most salient driving factors for ML&DS from a lifelong perspective, and to synthesise the existing evidence concerning the perceived effectiveness of current interventions fostering ML&DS.
 
Description From D3.1 - analysis of impacts interventions from the international dataset - analysis conducted and published by the UK team.

Key findings on Media Literacy and Digital Skills (ML&DS) intervention impacts
A few cross-cutting conclusions about ML&DS interventions more generally can be drawn:
? Participants generally reported modest perceived gains in digital knowledge and skills, although consistent challenges in navigating online environments and recognising misinformation persisted.
? Confidence generally increased, even if literacy and skills did not (confidence is also likely to have collateral benefits in areas other than those that were the explicit aims of the intervention).
? In-service teachers showed an increase in information navigation, wellbeing, social and financial activities post-intervention.
? Elderly individuals demonstrated improved digital skills and knowledge post-intervention, especially in their ability to navigate information and use digital tools, although challenges in social media use and online information evaluation remained.
? Participants who did not have the national language as a first language struggled to participate in these interventions and acquire ML&DS, even though they were often clear on the benefits of being able to engage with ICTs. When translations were offered , participants showed to have higher skill levels than anticipated.

Key insights for intervention evaluation design
The evaluation raised many insights around the process of evaluation and obstacles encountered in evaluating interventions:
• In evaluations involving socially disadvantaged groups, there were considerable complications in applying quantitative methods robustly; observations, interviews and other more personalised forms of evaluation and measurement should be considered since they yielded important insights.
• ML&DS evaluations should be designed in line with the diverse needs of target groups - for example disadvantaged youths and the elderly will approach the interventions and benefit in different ways, and evaluations need to reflect this.
• Success of the evaluation relied heavily on instructor engagement, particularly in overcoming language barriers and low literacy levels among the participants.
• The project highlights the importance of flexible, user-friendly evaluation toolkits to support the effective development and implementation of scalable ML&DS programme evaluations.

Future directions for research in evaluating ML&DS interventions
• In future research, time needs to be scheduled to establish good relationships with intermediaries who work with vulnerable individuals.
• Trust relationships with intermediaries and beneficiaries are key to being able to conduct further research with larger, controlled samples, especially for vulnerable groups that continue to be underrepresented in research and groups less likely to participate in interventions.
• Larger samples and pre- and post-measurement with greater time lags and incorporation of serial interventions are needed to validate the initial findings from REMEDIS.
• Instruments need to be adapted to the needs of both the intermediaries implementing the intervention and the beneficiaries in mind, with clear explanations on what the range of things are that might be considered in evaluation and why.
• While they understood the reasons for including digital skills measures, and observed the impacts on these, stakeholders did not completely grasp the ways in which or why outcomes of various kinds might be meaningfully evaluated, and findings on impacts were more confused. Future research needs to improve and validate these measures and engagement with stakeholders around these.
• Developing robust evaluation instruments and methodologies is crucial for accurately measuring intervention impacts, with a focus on addressing the limitations posed by low participant numbers, short-term or one-off interventions, and ensuring comprehensive pre- and post-test scenarios with the integration of evaluation into the design of longer-term interventions from the start.
• Short instruments measuring the different aspects of ML&DS as well as the outcomes of these interventions in vulnerable populations need to be further developed.
• Considering the low levels of traditional and digital literacy of many of the participants, evaluation instruments need to be developed keeping this in mind, and made available in different languages and formats.
• There is a delicate and complicated balancing act between brevity, understandability and comprehensiveness in measuring impact, which necessitates variations in measurements for different groups and complicates the evaluation of the effectiveness of interventions.
Exploitation Route Other similar organisations working with Media Literacy and Digital Skills interventions can use these findings and to improve their evaluation - the findings report were from only one part of the project, there are further findings from quantitative analysis that have not yet been reported and a toolkit that is about to be published.
Sectors Communities and Social Services/Policy

Digital/Communication/Information Technologies (including Software)

Education
URL https://remedis-chanse.eu/
 
Description Our partners and organisations across Europe have benefited from imput in the design of evaluation tools and a toolkit developed by the project. They improved their evaluation and subsequently alligned the delivery and design of their programmes more with the outcomes they want to achieve. New developments in media literacy and digital skills were incorporated into the programmes and evaluation toolkits, for example, how to think about AI literacy and the outcomes that might be associated with it into the development of media literacy and digital skills interventions. Work in Spain that the UK team is very heavily involved in is still ongoing with schools across the region of catalunya.
First Year Of Impact 2024
Sector Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Education
Impact Types Societal
 
Title REMEDIS Media Literacy and Digital Skills Intervention Questionnaire 
Description This questionnaire is aimed at evaluating changes after Media Literacy and Digital Skills interventions in participant groups. It comprises four key parts: A. Socio-demographic data B. Digital skills and knowledge and media literacy C. Actual Uses and Outcomes D. Future Uses and Expected Outcomes These parts can be combined according to the intervention's particularities to apply as pre-test, immediate post-test or a delayed post-test. 
Type Of Material Improvements to research infrastructure 
Year Produced 2024 
Provided To Others? Yes  
Impact The application and analysis of the results by the use of the questionnaire has provided valuable insights to our partner organisations in terms of understanding what their contribution for improving participant's digital skills level, and how this can be translated to better usage opportunities and outcomes in different domains. This has been used by such organisations to communicate their impact to stakeholders like funders and volunteers. 
 
Description Event "Be Media Smart: Measuring What Matters" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The event, jointly hosted by Making Sense of Media and Media Literacy Ireland and supported by Ofcom and Coimisiún na Meán, explored the process, practicalities and potential of evaluating media literacy initiatives
Year(s) Of Engagement Activity 2023
URL https://us02web.zoom.us/rec/share/PMlTMwGt_ltHqXH4Un1tOykoMke9pHKYVbW_YC7-_0UjyskqcXg_31zJcs6IR85x.d...
 
Description • ESRC Festival of Social Science. "The Digital Literacy Provision in the UK: Challenges and Possible Solutions" 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact It was an event to exchange ideas and experiences with policymakers, local authorities, academics and non-profit organisations on current challenges on media literacy and exploring opportunities for improvement.
Year(s) Of Engagement Activity 2024
URL https://www.eventbrite.co.uk/e/digital-literacy-provision-in-the-uk-challenges-and-possible-solution...
 
Description • Webinar "Enhancing Media and Digital Literacy Interventions" 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The REMEDIS project was introduced to organisations related to media and digital literacy interventions in Europe, highlighting the key findings from the literature review and the work in progress on the co-development of improvements and evaluation activities-
Year(s) Of Engagement Activity 2024
URL https://youtu.be/nmEf78S9MLI
 
Description • Webinar (In Spanish) "Mejora de las intervenciones de alfabetización mediática y digital" 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact This webinar was aimed at Spanish-speaking organisations working on media and digital literacy interventions in Europe. The webinar comprised of an overview of the REMEDIS project, a summary of the findings of the literature review, and how the co-development and evaluation of our partners' interventions will take place.
Year(s) Of Engagement Activity 2024
URL https://youtu.be/EPQDTXT21mI