The publication was prepared as a part of the American Spaces Digital Literacy and Training Program. This is a program of the U.S. Department of State, administered by PH International and its partners the Georgian Centre for Strategy and Development and Debunk.org. Read more about the project here.
Evaluation of digital literacy initiatives is crucial as it gives a proper understanding of the true performance of a programme, its shortcomings and successes, thus providing great insights on ways to improve current programmes and on the development of future curriculums for all the interested ML (media literacy) actors. However, it is observed that such programmes often do not undergo these necessary evaluations. European Digital Media Observatory (EDMO) notes that “of the 68 media and information literacy projects analysed by the Council of Europe’s Committee of Experts on quality journalism in the digital age, one-third did not undergo any form of evaluation or assessment.”
Measuring the impact of digital literacy initiatives is often quite complicated, there is no one-size-fits-all approach which would allow for an easy and accurate account. Nonetheless, it would be misleading to say that there is a complete lack of guidelines. Individual scholars and governmental and non-governmental organisations all provide valuable information on what are some available routes to choose from when assessing the impact of media literacy interventions.
What Can Be Evaluated?
The evaluation of media literacy programmes can consist of measuring the outputs and the outcomes. The outputs would include physical results-e.g. the number of enrolled students, while the outcomes are the impacts of the intervention/program - e.g. an increased ability of participants to detect misinformation on social media. Of the two, it is of course the second which should be of greater importance. However, measuring the outputs is comparatively easy, can provide an important general understanding of programme success and should be included.
Suggested Good Practices
Media Literacy Design Manual (U.S. Department of State)
The Media Literacy Design Manual provided by the U.S. Department of State confirms that evaluating ML interventions is a complex and problematic process, most notably, due to an absence of a universal definition of ML preventing a proper cross-programme comparison and ML outcomes are linked to behaviour changes, which may take a long time to develop.
Most importantly, developing and then later being able to measure an ML intervention successfully requires the development of a theory of change and logic model. This includes defining the overall objective, sub-objectives, inputs, activities, outputs, short-term and long-term outcomes, key assumptions, and (iii) the causal logic underlying the pathways between programmatic inputs, outcomes, and the overall objective.
Another key consideration for monitoring media literacy projects is to consider what data will be needed and how it will be used. ML includes a very broad range of skills and knowledge (and thus outcomes which might be sought). This means that measuring every single aspect of ML holistically is quite unfeasible. Thus, the decision on what to measure, which aspects of ML to include, and what methods to use are highly intervention-dependent.
A list of the three most common techniques of outcome measurements is provided in the Manual:
Self-reported measures: These involve asking beneficiaries about changes in their knowledge, attitudes, and practices regarding specific outcomes, usually using Likert scales ranging from strongly disagree to strongly agree. They provide insights based on individuals' perceptions of their own development or changes resulting from a project or intervention.
Competency-based measures: These assess beneficiaries' knowledge and abilities directly, often through tasks such as identifying misinformation in news articles or applying critical thinking skills. Unlike self-reported measures, they offer a more objective evaluation of ML skills. While basic measures might focus on knowledge of ML-related definitions, more robust assessments involve exercises that require critical thinking and real-world application. Developing such measures may be complex, requiring piloting or expert consultation, but they provide valuable insights into individuals' actual proficiency in ML-related tasks.
Observational measures: Unlike competency measures, which may occur under controlled or artificial conditions, observational measures evaluate ML competencies in real-life contexts. This approach examines behaviours such as the sources individuals choose to read, or their interactions on social media platforms. While directly monitoring individuals' online behaviour may have practical and ethical limitations, analysing aggregate behaviours, such as social media metrics before and after a campaign, can yield valuable insights into changes in behaviour patterns, such as shifts in the frequency of sharing misinformation or the usage of relevant hashtags.
Some other considerations to take into account would be the way measurement is administered, how long it takes, etc. It is unlikely that participants of online courses would be willing to answer a lengthy and complex survey before starting the programme. On the other hand, if the programme already has integrated quizzes for participants’ self-evaluation, these might provide viable insights on the outcomes of the programme without involving substantial costs and difficulties in implementation.
The frequency of measurements should also be considered very carefully. While it can also be programme-dependent (e.g. already mentioned quizzes might mean that some measurement is taking place after each course video), it is stressed that “ML measurement should be administered at the start and end of a given individual’s engagement with a project[1].” In an ideal situation, a survey would also be administered after some time since the programme has ended, as this allows for an analysis of the longevity of acquired skills and knowledge.
However, measuring before and after of a particular ML intervention is not the ideal way of capturing the impact. A proper measurement of the intervention’s impact would involve the strict following of the experimental scientific method when a randomly selected sample of individuals would be discerned into experimental (those who participate in the ML programme) and control (those who do not) groups[2]. This is a complicated and time-consuming undertaking. Nonetheless, similar alternatives like giving out expert-prepared final surveys for both the participants and non-participants (with similar demographic characteristics) or usage of statistical methods should be considered as this can provide invaluable evidence of the success of the programme.
Evaluation Toolkit (Ofcom, 2023)
Another useful resource on the evaluation of ML programmes is the Evaluation Toolkit developed by Ofcom (UK's communications regulator). To start with, it is worth considering between internal and external evaluation. The former is less costly, more adaptive and allows the development of the competencies of internal members of an organisation. The latter is more useful if there is a lack of internal expertise. Moreover, it helps ensure the impartiality of the evaluation.
Regarding the outcomes of the programme, it is viable to make a distinction between immediate and medium-term outcomes. Moreover, while difficult to assess, the contribution of these outcomes to a wider societal change could also be analysed. Ofcom also highlights the importance of establishing key evaluation questions. They are essentially questions stemming from the hypotheses made in the designed theory of change of the intervention[3]. Moreover, they might include not only the questions regarding the impact of the programme but also the ones related to the evaluation of the process like “Did project staff successfully build workshop resources that were effective?”
It is stated that many sources recommend around five to seven key evaluation questions and this final list should be the product of a discussion involving major stakeholders on a broader list of questions. Finally, the identification of whether outcomes were reached is done through outcome indicators: “measurable pieces of evidence that allow you to track the change that has taken place as a result of your intervention.” These can involve both objective (score of a test on dis/misinformation detection) and subjective (a person's self-assessment on how confident they feel regarding their skills) measures.
In addition to already discussed types of measurements (self-reported measures, competency-based measures, observational measures), the possibility of carrying out interviews or focus groups is also discussed[4], as this would capture in-depth qualitative information which otherwise would not be shown by the more popular quantitative measures.
DigComp 2.2 (EU)
The Digital Competence Framework for Citizens (DigComp 2.2) developed by the European Commission’s Joint Research Centre “provides more than 250 new examples of knowledge, skills and attitudes that help citizens engage confidently, critically and safely with digital technologies, and new and emerging ones such as systems driven by artificial intelligence (AI)[5].” This framework might also be useful as a source for identifying what is to be evaluated. For all the outlined competencies there is a list of related knowledge, skills and attitudes. This raises the potential of building a programme following already existing digital literacy frameworks.
Another potentially useful document from EU institutions is “Guidelines for Teachers and Educators on Tackling Disinformation and promoting digital literacy through education and Training.” It also provides key competencies regarding digital literacy and examples of ways to assess them.
UNESCO’s Media & Information Literacy Curriculum for Educators & Learners is another very comprehensive framework on media literacy. A 403 pages-long document[6] covers many relevant areas. Similarly to DigiComp 2.2 and depending on the curriculum of the programme, this framework could be used to map out the competencies which are to be monitored.
How do Peer Programmes Measure Success and Impact?
Bad News Game
“The Bad News Game is a multiple award-winning fake news intervention aimed at building psychological resistance against online misinformation. The intervention is a theory-driven social impact game developed in collaboration with the Dutch media collective DROG and graphic design agency Gusmanson[7].” Its gamified nature offers an appealing way of developing resilience towards the dis/misinformation. The impact of the game was evaluated via surveys administered before and after gameplay. The first method: “Survey participants were asked to rate the reliability of a series of nine social media posts on a 1–7 Likert scale, with 1 being ‘unreliable’ and 7 being ‘reliable’.” Part of the posts were credible. The second method: “Participants were [...] shown four real and four false headlines before gameplay, and a different set of four real and four false headlines after gameplay, for a total of eight real and eight false headlines.” Moreover, the participants were divided into two groups: the first group received an “A” set of headlines at the start and a “B” set at the end, and the second group vice versa. In addition, a set of demographic questions were asked.
Harmony Square
Harmony Square is another similar gamified ML intervention based on inoculation theory. A 2 (treatment vs control) by 2 (pre vs post) mixed design randomised controlled trial was used to determine the impact of the intervention. There were a total of 16 fake social media posts, they were divided into two sets, each containing 4 “real” fakes and “fictional” fakes and each made use of one of 4 manipulation techniques. For each of the posts, 3 Likert scale-based questions were asked: i) How reliable do you find this post? ii) How confident are you in your judgement? iii) How likely are you to forward this post to others[8]?
MIST
While not a ML intervention programme per se, the Misinformation Susceptibility Test (MIST) is a psychometrically validated measure of news veracity discernment. “The MIST was developed to be the first truly balanced misinformation susceptibility measure with an equal emphasis on discernment, real news detection, fake news detection, and judgment bias[9].” In essence, participants are asked to categorise different headlines into two groups: true and false. There are an equal number of true and false headlines. An online version of the MIST20 and MIST16 surveys can be found here: https://yourmist.streamlit.app/.
The strengths of this method for measuring the impact of an ML intervention would be its simplicity, the scientific rigour with which it was developed and the universality which allows for cross programme and cross-population comparison. However, the headlines are difficult to incorporate in a course intended for a diverse audience from different countries.
Other Examples
Of course there are other, not experiment-based measurement techniques. For example, Grzegorz Ptaszek outlines some of the used methods and among other case studies describes that of Tom Hallaq (Hallaq, 2016, p. 66) who “designed, in turn, Digital Media Literacy Assessment (DOMLA), which enables a holistic measurement of media literacy. Based on the literature, the researcher distinguished five dimensions of media literacy: media awareness (MAw), media access (MAc), ethical awareness (EA), media evaluation (ME), and media production (MP). [...] A paper-and-pencil questionnaire has a self-assessment character and includes 50 statements that are assessed by a respondent using the six-degree Likert scale, where 1 stands for “strongly disagree” and 6 for “strongly agree[10].”
Outcomes of the ML Programme
The desired outcomes would include the increase of knowledge on the subjects and the acquisition of related skills.
However, it is also important not to forget the broader outcomes outlined in Table 1. For example, in regards to the ability of discerning misinformation from real news (an obviously seeked outcome), it would be worthwhile not only to test the retention of course material (e.g. through a quiz after course video or a task to categorise fake and real headlines) but also to carry out more universal test on ability to discern misinformation using either pre-post surveys or experiment/control groups or comparisons with already existing statistics (e.g. peer programmes). Some other of these more general parameters could be the levels of digital citizenship (e.g. how often does a person report instances of hate speech on social media), broader behavioural changes like critical thinking, etc.
The proper evaluation of a programme's impact would require a baseline against which the intervention’s effect are being measured. Hence, quizzes after each course video are useful for participants to test their immediate knowledge but they might not contribute much to a more in-depth understanding of the programme's impact.
Measuring Engagement
The measurement of engagement might be quite a bit easier in comparison to the measurement of impact. The KPIs - Completion Rate, Usage Rate, Time Spent, and Website Statistics (frequency of forum discussions, etc.) are easily quantifiable and not hard to capture, most of this data is already recorded by various platforms which can be used for the course. Thus, participants’ feedback (through either qualitative or quantitative responses) could also be used to draw insights into the engagement of the programme. This should potentially be done continuously as it would allow for adjusting course material (if possible) to meet participants' expectations.
TO-DO list
The development of media literacy programs is a demanding process that requires besides the development of a fitting methodology several other steps:
Clear, precisely defined topics to be taught have to be set out and objectives, target audiences and how the programme will be administered (online, offline, hybrid) have to be defined. In essence, a theory of change model needs to be developed.
The corresponding, already existing courses and pieces of training have to be reviewed to avoid overlaps.
After clear objectives are set out, short-term and medium-term outcomes are to be sought.
With the list of objectives, outcomes, and evaluation questions to be answered, the best techniques for measuring the impact of the program can be chosen.
Self-reported, competency-based, and observational measures can be used. The use of them needs to be determined for the specific outcomes to be measured, considering the pros and cons of each of the techniques.
Establish the baselines against which the effects of the program are being compared.
The use of universal measures (like MIST) would allow a comparison with peer programmes and an assessment of the broader societal impact of the programme.
Simpler KPIs like engagement, enrolment and completion rates are also important. If demographic data can be collected, this might provide further insight into what communities are targeted.
A pilot research by administering the programme on smaller focus groups before the full launch would help make needed relevant adjustments
References
[1] Media Literacy Design Manual.
[2] Ibid.
[3] Evaluation Toolkit, 13.
[4] Ibid, 21
[8] Roozenbeek, Jon, and Sander van der Linden. "Breaking Harmony Square: A game that “inoculates” against political misinformation." The Harvard Kennedy School Misinformation Review (2020).
[9] Maertens, R., Götz, F.M., Golino, H.F. et al. The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment. Behav Res (2023). https://doi.org/10.3758/s13428-023-02124-2
[10] Ptaszek, G. (2024). Media Literacy Outcomes, Measurement. In The International Encyclopedia of Media Literacy (eds R. Hobbs and P. Mihailidis). https://doi.org/10.1002/9781118978238.ieml0103