It Doesn’t Always Add Up: The Ugly Truth of AI in Education

Book Review of Cathy O’Neil’s “ Weapons of Math Destruction”

Abstract

Technology and artificial intelligence (AI) has become a major part of our lives, from the virtual assistant software, like Alexa and Siri, to recommended videos on YouTube. As artificial intelligence begins to infiltrate every aspect of society, investigating the impact of AI in education integration is paramount. AI is the foundation of major models that influence the trajectory of our educational system. Weapons of Math Destruction (WMD) are the result of  models implemented by industries to reinforce algorithmic inequity, human bias, prejudices, and exploitation. In Weapons of Math Destruction, Cathy O’Neil (2016) exposes the role of WMDs in various industries. In this review, we will highlight how WMDs affect the educational system, specifically their relationship with student learning, teacher assessment and employment, college recruitment, and college & university ranking systems. Finally, we will examine how WMDs can be improved to enhance education.

Keywords

Artificial Intelligence; Data Algorithms; Biases; Education Research; Educational Opportunities; Equity; Socioeconomics

I. Introduction

While technology and AI have made momentous and beneficial contributions to the many facets of the human experience, there is much to assess regarding its implementation. Cathy O’Neil (2016) highlights the dark side of algorithmic models and the complex systems and social cycles it perpetuates. The global educational system is not exempt from the influence and exploitative nature of the Weapons of Math Destruction created by these models.

II. What is a Weapon of Math Destruction?

As defined by O’Neil (2016) a Weapon of Math Destruction (WMD) is an algorithmic model that continuously sifts, collects, and computes data to identify patterns. The algorithmic models manipulate data about “human habits, hopes, fears and desires” (O’Neil, 2016, p. 75) to make action-guiding conclusions for the models’ creators. Through studying human behaviors these models manipulate the masses to advance their objectives and benefit financially. Knowingly and unknowingly, WMD models reinforce algorithmic inequity by embodying the implicit biases of their creators. The darkness of WMDs can be illuminated by examining three areas: scale, opacity and damage.  By its nature, a WMD is obscured, vast in scope and elicits damage unnoticeable to the naked eye.

The Ugly Truth

WMDs thrive by identifying societal patterns and shackling consumers in detrimental  cycles sustained by past behavior. The sordid truth is that these models rely on data regarding past human behaviors and do not consider human capacity for change and improvement. When this data is unavailable, WMDs utilize “stand-in data or proxies” (O’Neil, 2016, p.17) to make conclusions about future human behaviors. This data produces generalized conclusions that disadvantage the individuals who exemplify the bias and prejudice of the model creators. Larry Summers, a past president of Harvard University, suggested that the low numbers of women in math and the hard sciences might be due to genetic inferiority – what he called the unequal distribution of ‘intrinsic aptitude” (O’Neil, 2016, p. 38). Contemporary society does not support the mental inferiority of females as more females attend a higher education institution than men. Biases like Summer’s,  are responsible for sustaining WMD models where less females enroll into programs of math or hard science. In response to this systemic bias, many institutions actively recruit females for enrollment into STEM-related fields of study.

This poses the question: Are females micro-targeted as a means to meeting recruitment quotas created by algorithms or based on merit alone? Regardless of an academic institution’s intentions toward female applicants,“models despite their reputation for impartiality, reflect goals and ideology” (O’Neil, 2016, p.21). “A model’s blind spots reflect the judgements and priorities of its creators” (O’Neil, 2016, p. 21).  The attempt to remove bias toward a specific group of applicants in itself has reinforced the original bias. This exemplifies the opacity and ambiguity female applicants encounter when engaging with the WMDs of academia.

III. The Faceless Victims

Weapons of Math Destruction rely on the pretense that math is difficult and therefore not understandable by most lay people. These models’ algorithms are deemed irrefutable even as they punish the poor and oppressed of society. “WMDs are opaque, unquestioned and unaccountable, and they operate at a scale to sort, target or optimize millions of people” (O’Neil, 2016, p.12). When there is an error in the model, it is minute to the model creators and devastating for the masses. One small glitch in the algorithm can “turn someone’s life upside down” (O’Neil, 2016, p.10) and create a victim.  Overlooking the “glitches’’ does not phase the creators of WMDs because from their point of view, the victims are insignificant and faceless. The impact of WMDs in the educational system is no different. Like many other industries, it is a business and is driven by economic success and academic rankings. This drive impacts students, parents and educators of all economic classes.

Students

Students by far are most damaged by the impact of Weapons of Math Destruction. From the moment high school students begin their college search and create a College Board account, they are bombarded by advertisements curtailed by algorithms. “Impoverished students who provide their contact information are subsequently stalked” (O’Neil, 2016, p. 79) by for-profit institutions. These institutions do not target wealthier students. The assumption is that students of wealthier families have more knowledge and guidance to discern that for-profit institutions are not cost effective. Therefore, strategically released for-profit ads lure students to apply to the schools and the damage is not felt until students acquire hefty tuition costs and unhelpful less advantageous diplomas. Vicious ads “pinpoint people in need, selling false promises” (O’Neil, 2016, p. 70) that higher education will improve the lives of students by increasing employment opportunities and socioeconomic status. Educational WMDs feed on the limited access to resources of veterans students and students with childcare concerns, history of incarceration, emotional/mental challenges and those struggling to make ends meet. Diplomas from for-profit colleges were “worth less in the workplace than those from community colleges and about the same as a high school diploma. And yet, these colleges cost 20% more than “flagship” public university” (O’Neil, 2016, p. 80). Unbeknown to students, WMDs are a form of assessment in which students’ backgrounds and characteristics are being evaluated and used against them.

Parents

In hopes of providing their children with every advantage in today’s society, parents of all socioeconomic levels subscribe to the belief that education is the key to success. Taking this into account, these WMDs models assign economic classes into silos where there is little opportunity for change. Often wealthy parents are targeted as money makers. An example of this trend is the concept of legacy students, students whose parents or family members attended the same school and are more likely to donate to it. “Legacy students receive a major boost at elite schools. At Stanford, legacy students are three times more likely to gain admission” (What is a legacy student?: BestColleges 2020). Wealthy parents also hire college admissions coaching firms that use AI to prepare and guarantee students will be admitted to their desired institutions. These firms  target the wealthy, formulate polished interviews, resumes, letters and personal statements for their clients, and can cost tens of thousands of dollars. Some affluent parents go as far as hiring companies to falsify application documents and get their students accepted to desired schools. Actress Lori Loughlin, known for her role on Full House, pleaded guilty to paying a “fixer” or coaching firm around $500K to get her child accepted into college (NBCUniversal News Group, 2021). Students of impoverished families may not have access to legacy programs or coaching firms and must try other means of getting ahead including applying to for-profit institutions. WMD models create flawed formulas that encourage these kinds of manipulation.

Educators

For educators, WMD models play a controversial role in assessment, specifically teacher evaluations. The purpose of these value-added models were to “evaluate the teachers. Get rid of the worst ones, and place the best ones where they can do the most good” (O’Neil, 2016, p. 4). Good intentions to improve the educational system were thwarted by un-checked and human-encoded bias which produced a Weapons of Math Destruction model. In an attempt to remove human bias and rely on score driven data, a Washington school district implemented a value-added model to base their teacher evaluations on student achievement scores (O’Neil, 2016, p. 5). Fifth-grade teacher Sarah Wysocki fell victim to this WMD model, despite her reputation as “one of the best teachers… ever come into contact with” (O’Neil, 2016, p. 4). Wysocki scored poorly on a teacher evaluation and was fired. The score generated by the model’s algorithm out-weighed the outpour of accolades from school administration and parents. When Wysocki questioned the evaluations outcome, the model’s procedures were camouflaged by sophisticated mathematical algorithms and kept opaque.  This model directed the Washington school district to forfeit impactful teachers such as Wysocki and 205 others making them voiceless victims. O’ Neil highlights the deficiencies of WMDs stating “without feedback, however, a statistical engine can continue spinning out faulty and damaging analysis while never learning from its mistakes” (p. 7). This pernicious feedback loop is detrimental to numerous lives. Instead of correcting the model’s flawed algorithms, the faceless victims of these WMDs are often blamed and held accountable for their insufficiency.

IIII. The Vicious Cycle

The impact of Weapons of Math Destruction on education stakeholders is not only paramount but also cyclical. In the educational system, a key catalyst of the vicious cycle of WMDs is the establishment of the data-driven U.S. News Best Colleges ranking. This ranking questionnaire was subjectively created by a small group of journalists to drive readership with hopes of qualifying educational success. Knowingly and unknowingly, this created a nocuous feedback loop reinforcing human-encoded bias. The prejudice of the creators led college administrators to focus their energy and funding on increasing their ranking rather than student learning. Schools with poor rankings are at risk of losing prospective students and prime faculty; “alumni would howl and cut back contributions (O’Neil, 2016, p.53) causing the college’s ranking to plummet. The subjective and incomprehensive qualifiers of this ranking system led administrators to focus on non-academics such as state-of-the-art student activity centers, popular sporting events, and even hot tubs. With the goal of increasing school pride and applications, administrators partner with the professional sporting industry to grow their reputation, ranking, and profit. This leads us to question whether adding college degree requirements within professional sports was instigated by this suspected partnership?

This cyclical scheme impacts the educational experience of young prospective students and educators alike. The entire ecosystem of education has adapted to the “de facto law of the US. News Best Colleges” (O’Neil, 2016, p. 61) algorithm. A slight change in the algorithm used for college ranking dictates what schools and students focus on during preparation. For example, society as a whole is now focused on producing more STEM related professionals, therefore colleges have redirected funds from the humanities and social sciences to STEM programs. In K-12 classrooms, educators are integrating STEM activities in order to build their students’ resumes. Social media often partners with higher education institutions to influence K-12 students, by advertising desirable schools and targeting specific student demographics with specific college ads.

With the wide use of algorithms in all aspects of our lives, the educational systems have begun to capitalize on the Big Data collected by social media giants. This data is collected from many different sources, so it is impossible to avoid being involved. Particularly, Facebook can harvest information from tens of millions of people and configure advertisements specific to their digital footprint. In education, schools pay to advertise on Facebook, but the algorithm is responsible for choosing which Facebook users see a particular school’s ad. Education, like any other company, revolves closely around finances, which means stirring up interest and  maintaining student enrollment and retention rates. One problem with the educational ecosystem prioritizing and advertising specific institutions is that it overlooks other programs and trade schools. This labels these programs as less lucrative career opportunities and pushes students to apply to more expensive schools.

V. The Undoing of WMDs

The problem with Weapons of Math Destruction is not the collection of Big Data, but how the information is used to inhibit the progress of the educational system and restrict its stakeholders into predetermined silos. Like the North American Industrial Revolution, algorithmic models can prompt tremendous progress and perpetuate grave suffering. It isn’t until you shed light on the vicious cycle that you see the faces of the sufferers. The intentions of education administrators when creating these algorithmic models are often centered around improving student learning and the educational process. This being said, the models are not doomed to failure. These algorithmic models need to be dynamic and ever evolving. This requires systematic and periodic checks and balances. For these models to be more beneficial than harmful, continuous incorporation of new data, and stakeholder (administrators, educators, students, etcs.) feedback is quintessential.

O’Neil (2016) emphasizes that “mathematical models should be our tools, not our masters” (p. 207). Society needs to first evaluate its dependence on tech and the role of tech in solving social problems. While technology is essential, it will not create a “techno-utopia” (O’Neil, 2016, p. 207), human intelligence is imperative. Human beings are biased and algorithmic systems are created to avoid implicit bias in our systems. Those systems became WMDs because they embodied the biases of their creators.  They don’t incorporate the reality that people change and evolve. Patterns and hypothetical data alone should not predict the future, “while often flawed, [human decision making] has one chief virtue. It can evolve” (O’Neil, 2016, p. 203). WMDs require human involvement to routinely assess and improve their algorithmic models. For instance, in 2007, O’Neil (2016) experienced the inception of the 2008 housing market crash. She recalled that WMD models were not able to clean up the mess they created within the housing market crash; mathematicians, not machines, had to filter through the mess and fix the problems. “Math could multiply the horseshit but not decipher it. This was a job for human beings” (p.43).

Because humans are multi-faceted, the algorithmic models that organize our society, especially education, need to be dynamic and comprehensive. Bernhardt’s (1998) Multiple Measures calls attention to the need for assessments to be routinely evaluated from numerous perspectives and multiple criterion. Routine evaluations will bring “blind spots” (O’Neil, 2016, p. 21) to the forefront and guide us in eliminating the implicit biases of its creators. Including needs assessments in the evaluations will provide a holistic blueprint of what needs to be addressed and improved. WMDs are often a  form of assessment in which education stakeholders’ backgrounds and characteristics are being evaluated and used against them. Routine evaluation of this model will provide qualitative analysis and “ditch the unfair system” (O’Neil, 2016, p. 208). O’Neil (2016) suggests “an ecosystem with positive feedback loops, can we expect to improve teaching using data (p. 209).

As O’Neil (2016) states, “the heart of the problem is almost always the objective” (p. 197). If we use algorithmic models to improve the educational ecosystem instead of ignoring the hidden exploitation, these models will no longer be Weapons of Math Destruction. O’Neil (2016) suggests Derman and Wilmott’s (2009) Hippocratic Oath as an example of illuminating the “misuse and misinterpretation” (p. 205) done by WMD models and  encouraging mathematicians to create transparent, equitous, dynamic and efficient algorithmic models. In order for algorithmic models to be successfully and transparently implemented, the operations need to be regulated. Government policy can help to ensure ethical use of Big Data and give consumers the choice to opt-in to participating in the models, rather than the current practice of having to opt-out.

Conclusion

The use of Big Data in education requires “subtlety and context” (O’Neil, 2016, p. 209). We cannot rely on WMDs to measure success while the models are riddled with bias and misinterpretations. Consumer ignorance is a “crucial piece of the puzzle” (O’Neil, 2016, p. 72) and sustains WMD models. Correcting the process is not going to be easy or smooth; it’s going to be disruptive since these negative patterns are ingrained in our society and we are ignorant of its power. WMD models are missing the unique element of human moral and subjection. Humans can qualify and quantify data that mathematical models can only quantify. This includes social, emotional and circumstantial data. Reliance on algorithmic models within the educational system will continue to increase. “The moral of the story is, if we want to get different results, we have to change the processes that create the results” (Bernhardt, 1998, p. 5). Humanity is the nexus to shattering the pernicious influence of Weapons of Math Destruction.

References

Bernhardt, V. L., (1998, March). Invited Monograph No. 4.California Association for Supervision and Curriculum Development (CASCD). Retrieved April 23, 2022, from https://nces.ed.gov/pubs2007/curriculum/pdf/multiple_measures.pdf

Derman, E. & Wilmott, P. (2009). The financial modeler’s manifesto. Retrieved April 23, 2022, from https://www.uio.no/studier/emner/sv/oekonomi/ECON4135/h09/undervisningsmateriale/FinancialModelersManifesto.pdf

NBCUniversal News Group. (2021, October 28). Lori Loughlin pays $500K for college tuition of 2 students after admissions scandal. NBCNews.com. Retrieved April 22, 2022, from https://www.nbcnews.com/pop-culture/pop-culture-news/lori-loughlin-pays-500k-college-tuition-2-students-after-admissions-n1282644

O’Neil (2016), C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Penguin Books.

What is a legacy student?: BestColleges. BestColleges.com. (2020, September 10). Retrieved April 22, 2022, from https://www.bestcolleges.com/blog/legacy-student/