Are High Impact Approaches Helpful in Boosting News Media Literacy? Lessons from an Online Experiment Conducted in the United States During the 2024 Election Season

Abstract

This project evaluates whether high impact pedagogies are effective in boosting media literacy among American college students. In September of 2024 (less than two months before U.S. national elections), we integrated a controlled experiment in politics courses that we teach at our respective Minnesota colleges. We created learning modules that represent two different pedagogical approaches, one using a blend of high impact and traditional strategies and one using a traditional approach only. Students were randomly assigned to one of these two conditions. Before and afterwards, they took a Misinformation Susceptibility Test, in order to examine whether and how much their scores improved after engaging with the curriculum. Our results reveal that students in the high impact condition exhibited more media literacy improvement compared to students who engaged only with a traditional approach. Implications of these findings for democratic health, civic dialogue, and the public commons are discussed.

Keywords

High Impact Learning, Media Literacy, Misinformation Susceptibility, American Elections

Introduction

At the September 10, 2024 presidential debate, former President Donald Trump shared viral online misinformation that Haitian immigrants were eating the pets of native-born Americans. He was corrected immediately by fact-checking moderators, and it all happened in real time before an audience of 67.1 million viewers (Coster, 2024; Garsd, 2024). Today’s gaping divisions in the U.S. electorate are ideological in part but exacerbated by the inability of many American voters to accurately identify misinformation in an increasingly bewildering political and media landscape (Lewsey, 2023).

This project evaluates whether high impact pedagogical strategies (as opposed to traditional approaches) are more effective in enhancing news media literacy among American college students. In this project, we evaluate “news media literacy” specifically, which is students’ ability to critically analyze stories presented in news media. News media literacy is essential to democratic health because citizens need timely, accurate information to be effective decision-makers in elections and engage meaningfully in civic discourse and the public commons. Recent survey research from YouGov suggests that young Americans (under 45) are less likely than their older counterparts to classify real and fake examples of news headlines accurately (Sanders, 2023). With media becoming increasingly complex, it is essential to integrate more training on media literacy in schools. We need to develop pedagogical strategies that are effective in teaching about how to combat misinformation and improve media literacy among young Americans. 

High impact pedagogies refer to active, hands-on learning practices that promote stronger student connectedness with learning material. They often involve active, team-based, and experiential learning approaches and opportunities for students to discuss and engage with applied examples and problems. These pedagogies offer numerous benefits, including increased student motivation, deeper understanding of course material, improved critical thinking skills, and enhanced career preparation. They can also foster a more inclusive and collaborative learning environment. In contrast, traditional approaches tend to be more passive and less intentional about cultivating meaningful connections with the course material (Greenman, Chepp & Burton, 2021; Kuh & O’Donnell, 2013; Kuh, O’Donnell & Schneider, 2017). 

During September of 2024, we evaluated a group of students in our respective Minnesota institutions of higher education, Inver Hills Community College (a two-year public college located in Inver Grove Heights) and Hamline University (a comprehensive, four-year private university located in St. Paul). The courses included an introductory level class American Government and an upper level class Political Psychology. Both classes are taught with online components and require that students read articles and complete an online quiz prior to attending a synchronous virtual or in person class discussion. 

In our sample for this study (n=45), 53% of the participants attended Hamline University (and were in the Political Psychology course) and 47% attended Inver Hills Community College (and were in the American Government course). On average, students had completed about two and a half years of college and were more likely to identify as female (71%) rather than male (24%). Students’ ages ranged from 17 to 29 and averaged 20 years old. 

Before engaging with materials on news media literacy related to democratic elections, all students took the MIST-20 Misinformation Susceptibility Test as the “pre-test” that evaluates the extent to which they can identify trustworthy information (Maertens et al., 2023). Scores on the MIST-20 test range from 0 to 20 depending on the number of news headlines that are categorized correctly as either fake or real. Our student sample averaged 15.64 on the pretest. According to survey research, nationally representative samples of Americans tend to score lower than that at 13, which is about 65% accuracy (Lewsey, 2023). It makes sense that college students might have higher scores, given that they have self-selected into higher education, are taking politics classes, and average over two years of college-level coursework. 

For the experimental design, we created learning modules that represent two different pedagogical approaches, one using a blend of high impact and traditional strategies and one using a traditional design only. Our expectation was that the blended approach would be more effective in boosting media literacy compared to the traditional approach used alone. Students at both schools were randomly assigned to one of these two conditions and completed the online module, as usual, prior to class discussion. All students were assigned two short media literacy articles to read as part of the week’s module. The articles define fake news, provide specific examples, and offer clear lists of criteria to determine whether a news source is trustworthy or lacks credibility (Enoch Pratt Library, 2024; Kiely & Robertson, 2016). In a module quiz, all students answered two multiple choice questions that tested their understanding of the assigned articles. These strategies, reading about a topic and testing comprehension, represent traditional pedagogies. 

The main difference between the two groups is that, in addition to reading about fundamentals of media literacy, students in the experimental condition had opportunities for “real world” application (which is a cornerstone of high impact learning). Students in the experimental condition were challenged to locate, analyze, and discuss examples of high/low quality media from actual media sources. They wrote responses to the following essay question (about 2-3 paragraphs in length), and they discussed their answers in small groups during class discussion: “What are the best ways to identify high quality information about the upcoming U.S. elections and distinguish them from fake or misleading news? What is an example of a reputable news article about the upcoming elections, and how can you tell that it’s legit? In contrast, what is an example of fake or misleading news about the upcoming elections, and how can you tell that it’s suspicious? Does learning about cognitive biases help you to discern between fake and reputable news and, if so, how? Make sure to cite assigned materials in your answer, and share links for your article examples, and come to class ready to share and discuss!” Students in the control condition also wrote about and discussed an essay question of similar difficulty, but it was unrelated to media literacy. For instance, in the Political Psychology course, students responded to these questions: “Your assignment for this module was to find and read an article published in Political Psychology. Which article did you read? How does your article relate to Chapter 1 of the Introduction to Political Psychology book that you read? What is the article’s main question and thesis argument? What kind of evidence do they use to support their argument(s), and are you convinced by their findings? Why or why not?”     

After students completed the module materials and engaged in small group discussions, they all re-took the Misinformation Susceptibility Test (the “post-test”) in order to examine the extent to which their scores changed after engaging with the curricula. The classical experimental design allows us to compare the two groups of students and evaluate whether the blended strategies affect student learning and beliefs above and beyond a traditional approach used alone. In the experimental group, the average change in media literacy (from pretest to posttest) was +1.15. In the control group, the average change in students’ scores was -0.40, so scores declined slightly. An independent-samples t test comparing the mean scores of the experimental and control groups found a significant difference between the two averages (t(43) = 1.841, p < .05). This finding is similar to differences we uncovered earlier this year in a preliminary, pilot study designed the same way, which included a smaller group of 21 students at Hamline University (Oxendine & Sullivan, 2024). In the pilot control group, the average change in media literacy was -0.27, and in the pilot experimental group, scores increased by +2.1. Similarly, an independent-samples t test found a significant difference between the two averages (t(19) = -1.82, p < .05). All of these results support our initial hypothesis that high impact strategies enhance student learning about and understanding of media literacy. 

American democracy faces intense headwinds from disinformation campaigns conducted by hostile foreign actors like Russia (Tucker, Kepler & Lee, 2024), as well as grassroots forms of viral misinformation that reinforce popular resentments in the electorate. Even more challenging is the widespread distrust of traditional, professional news media. Political leaders intensify these trends by labeling reporting critical of them as “fake news.” Research indicates that politicians often face minimal political backlash for this misconduct but, in the process, jeopardize the influence of journalistic ethics and expertise (Egelhofer et al., 2022).

Educators need effective tools for capacity building, which support students to developing the skills necessary to discern between high/low quality information. In this process, Open Educational Resources (OERs) are particularly helpful in providing teachers and students the tools to be better informed participants in democratic processes. Given that these resources are in the public commons, they have flexible intellectual property licenses that are versatile, adaptive tools for educators (https://oercommons.org/). Also relevant is the News Literacy Project (https://newslit.org/), which is an open educational resource based in the United States devoted to helping young Americans share their voice and participate effectively and responsibly in the public sphere. It is our objective to expand this high impact news media literacy curricular activity into a standalone curricular unit and provide open access in the spirit of the OER movement. As news media literacy is an increasingly global challenge for higher education, it is useful to have tools that facilitate interdisciplinary and international partnerships with interested scholars and institutions.

Lastly, it is essential that educators embrace high impact pedagogical strategies to help students to think proactively about “red flags” that may identify mis- and disinformation. We believe that our study could be useful for both researchers interested in examining news media literacy and civic dialogue, as well as higher education teachers interested in learning new pedagogical approaches and improving classroom practices.

References

Coster, H. (2024, September 11). ABC News US presidential debate attracted 67.1 million TV viewers. Reuters. Retrieved from https://www.reuters.com/world/us/abc-us-presidential-debate-attracted-575-million-tv-viewers-preliminary-nielsen-2024-09-11/ 

Egelhofer, J. L., Boyer, M., Lecheler, S., & Aaldering, L. (2022). Populist attitudes and politicians’ disinformation accusations: Effects on perceptions of media and politicians. Journal of Communication, 72(6), 619–632. https://doi.org/10.1093/joc/jqac031

Enoch Pratt Library. (2024, April 14). Spotting fake news. https://www.prattlibrary.org/research/guides/spotting-fake-news 

Garsd, J. (2024, September 11). The stereotype of immigrants eating dogs and cats is storied – and vitriolic as ever. NPR. Retrieved from https://www.npr.org/2024/09/11/nx-s1-5108401/donald-trump-debate-eating-dogs-cats-immigrants-false-stereotype 

Greenman, S. J., Chepp, V., & Burton, S. (2021). High-impact educational practices: leveling the playing field or perpetuating inequity? Teaching in Higher Education, 27(2), 267–279. https://doi.org/10.1080/13562517.2021.2000384 

Kiely, E. & Robertson, L. (2016). How to spot fake news. https://www.factcheck.org/2016/11/how-to-spot-fake-news/ 

Kuh, G. D., & O’Donnell, K. (2013). Ensuring quality and taking high-impact practices to scale. Washington, DC: Association of American Colleges and Universities.

Kuh, G., O’Donnell, K., & Schneider, C. G. (2017). HIPs at Ten. Change: The Magazine of Higher Learning, 49(5), 8–16. https://doi.org/10.1080/00091383.2017.1366805 

Lewsey, F. (2023). The Misinformation susceptibility test. https://www.cam.ac.uk/stories/misinformation-susceptibility-test 

Maertens, R., Götz, F. M., Golino, H. F., et al. (2024). The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment. Behavior Research Methods, 56(6), 1863–1899. https://doi.org/10.3758/s13428-023-02124-2

News Literacy Project. (2024). Retrieved from https://newslit.org/

OER Commons. (2024). Retrieved from https://oercommons.org/

Oxendine, A. & Sullivan, Z. (2024, July). Are high impact approaches helpful in boosting news media literacy? Presented at the 5th International Media Literacy Research Symposium, Ponta Delgada, Portugal. 

Sanders, L. (2023, June 29). How well can Americans distinguish real news headlines from fake ones? https://today.yougov.com/politics/articles/45855-americans-distinguish-real-fake-news-headline-poll  

Tucker, E., Kepler, D. & Lee, M. (2024, September 4). U.S. accuses Russia of spreading disinformation before November election. Time Magazine. https://time.com/7017608/us-russia-disinformation-2024-election/