STEPHAN SCHMITZ/FOLIO ART
A high-profile Science paper that suggested Facebook’s algorithm plays little role in driving political polarization has drawn a pointed critique of its methods and conclusions. The work examined how the platform presented information to users around the 2020 U.S. elections. But in a letter published today in Science, researchers argue the paper is flawed because, during the study, Facebook instituted a series of emergency changes to its algorithm to reduce the spread of misinformation. They say the paper failed to properly alert readers to the potential effect of those changes on the study’s findings.
The authors of the study, published in July 2023, reject that argument and are standing by their findings, which they say are more limited and nuanced than often perceived. And in an editorial, Science Editor-in-Chief Holden Thorp says the journal will not ask for corrections to the paper but will make readers aware of the critiques.
The study in question tackled a hot question in social science: How powerfully do the algorithms structuring what we see on social media platforms such as Facebook, Instagram, YouTube, and X (formerly Twitter) impact our minds?
SIGN UP FOR OUR CAREERS NEWSLETTER
Get great career content biweekly!
In a bid to find out, a group of 17 academics joined with researchers at Meta, the parent company of Facebook and Instagram, to manipulate the feeds of some 20,000 Facebook and Instagram users from late September to late December 2020. They found that replacing the platforms’ usual feeds, which rank posts according to criteria Facebook does not publicly disclose, with one that was chronological—showing the newest posts first—increased the amount of misinformation people saw. And the chronological feed did not reduce political polarization, which the researchers measured by surveying users before, during, and after the experiment on issues such as immigration, health care, or COVID-19 restrictions. The results, the authors wrote, tempered expectations that social media algorithms “directly cause affective or issue polarization in individuals or otherwise affect knowledge about political campaigns or offline political participation.”
The critics, however, note Meta changed its algorithm during the experiment, undermining the usefulness of any comparisons. As a result, “The main conclusions of [the paper] may be incorrect,” says Przemyslaw Grabowicz, a researcher at the University of Massachusetts Amherst and a co-author of the letter.
Meta has acknowledged that it instituted 63 emergency measures—known as “break glass” measures—around the 2020 elections to reduce the flow of inflammatory content and misinformation on its platforms, says Filippo Menczer, a computer scientist at Indiana University and a co-author of the letter. When he and colleagues noticed “that there was this overlap between the period of the study and the period of the break glass measures,” they wondered how Meta’s changes might have influenced the study’s outcome. Because they did not have easy access to the data used by the Science study’s authors, they used a different data set from Meta to look at how much misinformation users were exposed to in late 2020 and early 2021. The level dropped during that period, suggesting Meta’s measures worked as intended, Menczer says. But that meant the chronological algorithm may have looked worse by comparison than it would have if Meta had not made those changes.
The paper should have called attention to the break glass measures, the letter’s authors say. Some researchers who weren’t involved in the letter agree. “If you have knowledge about something substantial being done to the algorithms while you’re running an experiment on those algorithms, then I think there is an ethical imperative to disclose that,” says Stephan Lewandowsky, a University of Bristol psychologist.
Northeastern University’s David Lazer, an author of the study, says it was always clear that Meta would be making some changes to their algorithm during the study period. “So, we put lots of cautionary language in the paper,” he says. “We didn’t use the phrase ‘break the glass,’” he notes. “But we did talk about the fact that this was an exceptional period where there might be exceptional measures being taken.”
That explanation doesn’t satisfy Menczer, who compares the situation to being invited for dinner and asking the host how long it will take to drive to their house. There is a big difference, he says, between being told, “It usually takes about 20 minutes, but the time will vary based on traffic” and “It usually takes about 20 minutes, but this week they shut down the main bridge” so it could take much longer. “Answer one is technically accurate and truthful, but I think that anyone would find it misleading compared to answer two.”
It is unclear who knew about Meta’s changes. The company said in a statement it communicated the measures publicly through newsroom posts and press briefings. “We shared this information with the academic researchers on this project as well,” says Chad Kiewit de Jonge, research director at Meta. But Michael Wagner, a social scientist at the University of Wisconsin–Madison who observed the work, says he does not remember any discussion. “I don’t recall specific instances that I observed where Facebook researchers revealed that break glass measures were taking place.” The company should have been more forthcoming, he says. “Look, the pace of change on platforms is mind-boggling, but that doesn’t mean that the solution is to not engage researchers, especially researchers they are collaborating with, about them in an upfront, transparent way.”
In his editorial, Thorp takes the position that “the authors made clear in the original paper that these kinds of changes could have affected the interpretation.” As a result, “Science is not correcting the paper.” But he says he is linking the paper to his editorial “to alert readers who may cite the work that the default algorithm was in flux during the experiment.”
The larger question is what Meta’s changes to the algorithm mean for the significance of the paper. Meta published a blog post at the time saying the study added “to a growing body of research showing there is little evidence that key features of Meta’s platforms alone cause harmful ‘affective’ polarization, or have meaningful effects on key political attitudes, beliefs or behaviors.”
But Andy Guess, first author on the Science paper, acknowledges the finding might not apply to other platforms or conditions. “We tried to be very clear in the original paper that we were studying a specific algorithmic change on a specific platform during a specific time in a single country.”
The only way to gain a more general picture of the role of algorithms in shaping political attitudes is to sew together a “quilt” of findings from various platforms over time, Lazer says. “The problem is we are quite limited in getting other squares [of fabric],” he says. “We’re not doing the same study in 2024. We’re not doing this with TikTok. We’re not doing this with X.”
In the end, Lewandowsky says the disagreement highlights the need for social media companies to provide researchers with access to their algorithms, so they can be independently studied. But in recent years, Meta and other social media firms have retreated from an array of data-sharing arrangements. Many researchers are looking to Europe, where the Digital Services Act requires social media companies to provide some data access to researchers.
In the meantime, the algorithmic changes Meta made during the experiment are themselves worth studying, Grabowicz says. He thinks Meta’s break glass measures could even have affected the aftermath of the 2020 presidential election. “Perhaps because of [Meta’s measures] the U.S. Capitol attack on January 6 was smaller than it would have been,” says Grabowicz, who is now trying to tease out such impacts. “It’s so important to understand the effects of these break glass interventions on society.”