feedback-loop

Chapter 9 of Weapons of Math Destruction examines the use of mathematical models in evaluating teacher performance, specifically focusing on the Washington, D.C. school district's value-added model (VAM) implemented through the IMPACT evaluation system. The chapter highlights the complexity and opacity of these models, their reliance on limited data, and their failure to incorporate meaningful feedback, resulting in unfair and destructive outcomes such as the dismissal of effective teachers based on flawed algorithmic scores. This example illustrates the broader dangers of WMDs, which perpetuate bias and punish individuals without accountability or transparency.

Chapter 3 explores how the U.S. News & World Report college rankings evolved from subjective opinion surveys to a data-driven algorithm that became a national standard, shaping higher education in the United States. The chapter details how the rankings, relying on proxies like SAT scores and graduation rates, created a self-reinforcing feedback loop that pressured colleges to game or optimize for the ranking metrics, often at the expense of broader educational values. This ranking system exemplifies a Weapon of Math Destruction (WMD) due to its scale, opacity, and destructive impact on institutional behavior and educational diversity.

Chapter 5 examines the use of predictive policing software like PredPol in economically struggling cities such as Reading, Pennsylvania, highlighting how these models rely on historical crime data to allocate police resources. While these models aim to reduce serious crimes by focusing on geographic hotspots, the inclusion of nuisance crimes disproportionately targets impoverished and minority neighborhoods, creating a feedback loop that perpetuates racial and economic disparities. The chapter critiques the zero-tolerance policing approach and contrasts the under-policing of financial crimes with the over-policing of minor offenses in poor communities.

This chapter examines how advanced scheduling software, a form of Weapons of Math Destruction (WMD), creates highly irregular and unpredictable work schedules for low-wage workers, particularly in retail and food service industries. These algorithms optimize labor costs by minimizing staffing and maximizing efficiency, often at the expense of workers' well-being, family life, and health. The chapter highlights the systemic nature of these practices, their roots in operations research and just-in-time logistics, and their broader social consequences including worker precarity and negative impacts on children.

Chapter 8 explores the evolution of credit evaluation from subjective, community-based judgments to algorithmic scoring systems like FICO, highlighting how early credit scoring improved fairness by focusing on financial data rather than proxies such as race or social connections. However, it critiques modern e-score models that rely on numerous proxies, such as zip codes and browsing behavior, which perpetuate systemic biases and create unfair feedback loops, disproportionately harming marginalized individuals. The chapter also discusses the unregulated nature of these e-scores compared to regulated credit scores and the broader misuse of credit reports as proxies for trustworthiness in various societal domains.