The End of Trust and Safety?: Examining the Future of Content Moderation and Upheavals in Professional Online Safety Efforts | Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

1 Introduction

"Now, there are fewer people checking in on the ‘dark, scary places’ where offenders hide and abusive activity gets groomed…More importantly, nobody knows how bad it can get.”

Tech layoffs ravage the teams that fight online misinformation and hate speech - CNBC News, May 26th 2023 [16]

Trust & Safety (T&S) teams have become central departments within tech platforms, working in tandem with product developers and engineers to consider how novel products and technologies may be used as vectors for abuse and misuse. T&S teams also ensure compliance with global laws against the sharing of Child Sexual Abuse Materials (CSAM), identify fraud and financial scams, and have increasingly reckoned with the spread and impact of misleading information around core societal topics such as elections and public health. Despite their societal value and role in risk compliance, T&S teams have faced significant downsizing during recent (2021-present) widespread layoffs across the tech industry. Media coverage of tech layoffs has focused on the deep cuts made to T&S teams at large platforms, including Meta, Amazon, X (formerly Twitter), and Google, and highlighted concerns that reduced workforces may not be able to handle bad actors using platforms to abuse vulnerable populations and financially extort users.

CHI research highlights the centrality of social media platforms within civil and political discourse and the role of interventions made by T&S teams in curtailing the spread and impact of related misinformation through the enforcement of community guidelines [29, 60]. Accordingly, media commentators fear a reduction of T&S teams may exacerbate issues of online misinformation, particularly pertinent ahead of the 2024 U.S. Presidential election. Such concerns co-exist with a rising public awareness of a core practice within T&S work—content moderation. An increase in partisan discourse around content moderation actions taken on COVID-19 and election-related content has brought the practice to public attention, often with a negative framing. Partisan actors argue that content moderation represents an attack on freedom of speech and is a form of censorship [30]. T&S teams, therefore, face the difficult task of justifying their work—work that is already difficult to quantify the impact of—during a time of heightened economic crisis and political controversy. Moreover, there exists crucial questions over the future of content moderation and T&S in the digital information environment, questions central to the work of CHI researchers interested in intervening in online harms through design, policy and user research.

This paper examines how T&S professionals are navigating the professional upheaval of recent years, including how they weather economic downturns and questions around the future viability of content moderation, given public distrust and political pressures. Through 20 in-depth interviews with T&S professionals, we explore the impact of widespread layoffs within the T&S industry and examine current professional perspectives of content moderation and broader strategies for maintaining safe digital environments. Our analysis highlights a duality between economic contractions within the industry and an expansion of the types of roles and actors working on T&S-related activities inside and outside of platforms. Further, we highlight the driving narratives undermining public trust in content moderation and how, despite pressures and practical hurdles, T&S professionals see the practice as a central pillar for online safety. Finally, we discuss routes to shore up public trust and platform investment in content moderation and broader T&S activities and the role of CHI research in supporting online safety through empirical research and the development of novel tooling to reduce the expense and bias associated with current practices.

2 Extant Literature

2.1 Defining Trust & Safety as a professional endeavor

While T&S teams have become a mainstay for technology firms, the range of roles within the industry and the nature of activities performed by such teams differs across the sector [2]—often dependent on the nature of the technology and its user base, and each company’s conceptualization and prioritization of online safety. In their overview of T&S’s key functions and roles, the Trust & Safety Professional Association (TSPA) identifies a broad range of roles, including content policy design and enforcement, product design, research and data analysis, legal & law enforcement, and threat discovery, among others [2]. In these roles, T&S professionals undertake a wide range of activities including; content moderation (through automated and human-driven means); the development, maintenance, and enforcement of content and platform community guidelines; user support; risk assessment and response; education and literacy around product use; serving as government and law enforcement liaisons; product development and tests; and UI/UX research and design. Across role types and activities, T&S teams work to reduce platform risks and improve user experiences by identifying potential vectors of harm within products and safeguarding users against bad actors and abuse. In doing so, T&S teams aim to shore up public trust in online platforms and meet legal requirements for data privacy and digital safety set by a patchwork of global legal frameworks.

The expansion of the profession and its concretization into a known industry can be best understood within a broader context of platform history and, specifically, the development of content moderation tools and approaches. Extant literature documents the trajectory of moderation alongside the progression of the internet from web forums to social platforms and beyond into Web 3.0 [21, 56]. Accordingly, the following literature review first attends to the history of content moderation online. We then turn to professional documentation and media coverage of T&S to understand how T&S activities developed into a specific profession within the technology field. Finally, we examine recent literature on the political economy of social media platforms to understand the current destabilization of T&S and the wider tech layoffs that have occurred since the 2020 COVID-19 pandemic.

2.2 The progression of online content moderation efforts

Sarah T. Roberts defines online commercial content moderation as “the organized practice of screening UGC (user-generated content) to posted Internet sites, social media and other online outlets, in order to determine the appropriateness of the content for a given site, locality, or jurisdiction” [53]. In the main, content moderation involves the removal of content, though there exists a growing number of moderation practices such as flagging—applying a warning label of sorts [44] or adding interstitials [25]— and algorithmic deprioritization [22], that have emerged as additional routes to intervening in problematic content. Content is removed if it contravenes current law in the relevant jurisdiction, e.g., hate speech or fake news in certain countries [9, 17], and CSAM globally [3, 11], or if it counters the community guidelines set by the platform [6, 32].

Caplan (2018) identifies three types of content moderation strategies currently deployed by platforms: “artisanal”—small scale, predominantly manually undertaken; “community-reliant”—relying on a larger volunteer base to identify and act upon problematic content; and “industrial”—utilizing automated methods and generally built upon separate policy teams and (often contracted) enforcement teams [7]. Caplan sees artisanal approaches as small-scale operations undertaken (at the time of writing) by platforms like Medium, Discord and Patreon, using small teams of in-house employees that deal with a lower reporting rate than larger platforms do. Community-reliant platforms, notably Reddit [20, 65] and Wikipedia [58], create infrastructure for larger groups of volunteers to help craft and implement community guidelines. Community elements are also utilized by larger platforms, notably by Facebook within Facebook Groups, wherein group members are empowered as administrators of the group to engage in moderation practices [33].

Industrial approaches to content moderation have emerged in larger social media platforms, e.g., Meta (Facebook and Instagram) and Google (primarily YouTube). In industrialized models, there often exists separation between policy development—led by in-house T&S professionals—and policy enforcement, which is increasingly contracted through business process outsourcing to third-party vendors [52]. An increasing body of research highlights the contingent and negative working conditions for frontline content moderators, including how the intense labor results in poor psychological well-being and even PTSD [50, 63].

In addition to documenting the methods and impacts of content moderation, past research has contended with the overarching conceptual frameworks guiding decisions to engage in content moderation. Notably, legal scholar Jonathan Zittrain lays out the competing tensions for communications technology platforms between “rights” and “public health,” i.e., the ability of users to participate online without interference versus the need to curtail the potential or actual harms caused by unfettered participation [4, 70, 71]. In light of the increasingly prominent and impact of social media disinformation, Bowers and Zittrain (2020) outline a “public health” model of content governance wherein platforms “are now being asked to don the epidemiologist’s hat and mitigate specific and contextual harms to norms and institutions arising from interactions between users on a massive scale” ([4], p. 6). Such undertakings require an expansion of the role of T&S to enlarge its research capacities to quantify harms and interventions to harm, at scale.

2.3 Public interactions with Trust & Safety work

As content moderation has become part and parcel of digital platforms, T&S work has become more visible to users. A growing body of research documents user interactions with moderation and the “folk theories” users generate to understand and better interact with both algorithmic and human-led moderation [15, 19, 42, 45]. Survey work by Myers West (2018) highlights tensions between users and platforms as a result of content moderation, finding a range of impacts on users, including a perceived chilling effect on individual speech or an “inhibition of their capacity for self-expression” ([45] 4374). Recent research by Moran et al. [42] documents how users develop tactics to expressly avoid content moderation—particularly users who wish to share information that contravenes community guidelines. Users develop folk theories of how content moderation works and its potential blind spots build strategies to take advantage of gaps in moderation to spread problematic content, e.g., anti-vaccine misinformation. Further work has similarly documented dissatisfaction with content moderation from users not engaging in contravening behavior. Register et al. [51] document the experiences and lasting impacts of content moderation on Instagram, highlighting instances of discriminatory content moderation and account restriction of marginalized users. This is echoed by similar work by Lyu et al., which highlights the negative interactions with content moderation experienced by people with disabilities. Through interviews with blind users of TikTok, the authors spotlight how perceptions of bias within moderation practices intersect with, and are exacerbated by, a lack of accessibility on the platform. [36]

In addition to examining current issues within content moderation approaches, including bias and the impacts on the mental well-being of moderators, CHI research has also sought to develop tools to combat emergent T&S problems. For example, Kolla et al. tested the feasibility of using large language models (LLMS) to identify rule violations on Reddit to minimize the work of human moderators, finding some promise in using LLMs to assist in moderation but encountering difficulties managing more complex rule-violating posts. [31]. Additional CHI work has sought to optimize specific elements of the moderation process including Song et al.’s. "ModSandbox" [61] which uses a virtual sandbox system to improve rule-based content moderation tools and Uttrapong et al.’s "ModeratorHub" — a knowledge sharing platform to streamline content moderation training and team collaboration.[67] Taken together, CHI and related research around user perceptions of content moderation suggests a contentious relationship between the public and the work of T&S professionals, despite the orientation of the profession towards enhancing public safety and shoring up trust in the platform. Further, research highlights future directions for T&S work, particularly related to improving content moderation processes and outcomes that underscore the importance of academic-industry collaborations within this space.

2.4 The concretization of Trust & Safety as a profession

The rapid growth and increasing scale of communications technology platforms have elevated content moderation needs and led to an expansion in the toolkit of T&S beyond content removal to a wide range of preventative, detection, and enforcement measures. The subsequent professional expansion necessitated to undertake this work has concretized T&S work into a known profession. Executive Director of the Digital Trust & Safety Partnership [14] David Sullivan [64] mapped out four eras of T&S work. Starting with (1) community moderation and self-governance on a pre-commercial internet, (2) the advent of large-scale UGC and commercial content moderation following Section 230 of the Communications Decency Act, (3) a growth in the 2010s in demands for transparency around content moderation, and finally (4) what Sullivan terms an “institutional evolution towards maturing” marked by the establishment of professional organizations like the DTSP, the Integrity Institute [26] and the TSPA [66] and an increase in global content regulations [64]. Zuckerman and Rejendra-Nicolucci [72] argue that the shift from community participation to top-down governance through professional T&S operations has led to a “crisis of legitimacy” for platforms, wherein decisions made are “often scrutinized and criticized as corrupt, arbitrary, and irresponsible” ([72], p. 6), echoing the public dissatisfaction explored in the previous section. Calls for transparency are being taken up by the nascent professional organizations in the T&S profession attempting to bring more public clarity to their work. Public-facing projects like the TSPA’s curriculum [2]—which documents the “T&S fundamentals” in addition to the types of roles and responsibilities of T&S teams—and the Journal of Online Trust & Safety look to codify a shared understanding of the profession and allow the public more visibility into their work. For example, recent articles within the journal lay out the experiences of building out a T&S team in a year at teleconferencing platform Zoom following its hypergrowth and switch from a B2B to B2C structure [38]. Further, work by Sullivan et al. [64] within the journal looks to bring together a lexicon of shared language around T&S work in order to “aid the professionalization of the field” through codified critical terms that allow for informed dialogue between T&S professionals and beyond to other key public stakeholders” ([64], p. 3).

2.5 Tumult in the Trust & Safety industry

The professionalization of T&S should thus be seen within a grounding context of public distrust in the decisions of technology firms and a growing internal reckoning within the technology industry over the pursuit of moderation. Zittrain calls this current era one of “process, or legitimacy” ([71], p. 9), noting the uncertainty expressed by major technology companies such as Facebook and Google around their own intervention policies. While the broader industry wavers on whether and how to moderate the digital spaces they operate, the emergent T&S subfield is working hard to systematize and make transparent its work. It remains to be seen the extent to which this exists as an attempt to protect its expanding workforce and assert its necessity for product development or whether professionalization represents a natural and productive evolution of content moderation towards more community-engaged models of governance.

This is also playing out against a backdrop of economic contraction within the technology industry, with widespread layoffs impacting the tech workforce from 2020 onward. X (formerly Twitter) was seen as the starting point for a wave of layoffs affecting tech workers, with T&S heavily impacted at X and other platforms. In its initial 50% cut, X let go of 15% of its T&S workforce in November 2022 [49] before it’s then-Head of Trust & Safety, Yoel Roth decided to leave the company at the end of the same month over concerns that new owner Elon Musk’s propensity to release ad hoc edicts on Twitter rules undermined the legitimacy of the T&S team’s efforts [55]. This was just the beginning of T&S related cuts at the platform; X then dissolved its T&S Council [49]—a group of 100 independent civil, human rights, and other groups—in December 2022 before making further cuts to its T&S team in 2023, including gutting its disinformation and election integrity teams [69]. Other platforms were said to “follow Elon Musk’s lead” in “surrendering to disinformation” through T&S layoffs [47] at Meta, Amazon and Alphabet [16]. In addition to layoffs, tech firms also rolled back misinformation-related content measures related to election rumors and COVID-19 misinformation, including platforms such as YouTube, Meta, and X [18]. The termination of trust & safety-related policies and enforcement powers, in addition to a massive reduction in the T&S workforce across the sector, has also happened alongside the growing politicization of content moderation and broader T&S work. Writing in Columbia Journalism Review, Golding argues “how politics broke content moderation”, mapping the gutting of X’s T&S team alongside Elon Musk’s release of the so-called “Twitter Files,” which Musk alleged “claimed proof the company’s previous leadership had colluded with researchers and federal officials to censor the right” through moderation [24]. While the evidence was not authoritative, and the claims of censorship unsubstantiated, content moderation has remained a subject of partisan polarization with especially right-leaning politicians attacking tech companies for their anti-disinformation efforts [28]. This, in turn, has given rise to concerns over the vulnerability of social media and communications platforms to disinformation campaigns, particularly around the 2024 U.S. presidential elections [54], given a diminishing workforce and necessary resourcing and in light of partisan hostility to T&S related efforts.

2.6 Have we reached peak Trust & Safety?

Taken together, there exist rightful questions over the stability and effectiveness of the nascent T&S field, with journalist Casey Newton begging the question— “Have we reached peak T&S?” [46]. The combination of policy rollbacks, mass layoffs, and politicization over the critical pillars of T&S work lay the foundation for questioning the future of the T&S industry and the work of online safety efforts. Moreover, the US legislative framework around online safety is seen as one characterized by "moral panic", with legislators pursing ill-guided and often maligned online safety bills that are not grounded in the realities of platform use, technological capacities (as in the case of the Kids Online Safety Act [1]) or retain unintended consequences (as seen with the FOSTA and SESTA bills designed to reduce human trafficking [5]). In light of the current period of tumultuous political and economic conditions surrounding T&S work, this research thus looks to explore the following research questions:

RQ1: How have T&S professionals experienced and navigated job precarity and professional upheaval in recent years?

RQ2: How do T&S professionals view content moderation as a route to achieving T&S on online platforms?

3 Method

Our findings emerge from an interpretive and inductive analysis of twenty in-depth interviews with T&S professionals working for US-based technology companies (informed by [8, 23]). In-depth interviews generate deep datasets that provide crucial insights into individual behaviors and opinions [68] and allow cross-examination from which patterns and themes can be analyzed [41]. Interviews were designed to gather insights on the professional experiences of T&S employees, their perspectives on content moderation and other routes to achieve T&S on online platforms, and to identify concerns around safe and productive online conversations related to the 2024 U.S. presidential election. All interview protocols, recruitment practices, and research methods were reviewed and approached by our university’s Institutional Review Board (00019893).

3.1 Data Sample

Interviewees were recruited through professional networks of researchers at the [LAB NAME] and the professional networking platform LinkedIn. The lead researcher identified potential interviewees through T&S related groups on the platform and hashtags associated with T&S-related professional conferences. Publicly available LinkedIn pages also include the employment history of individuals, and it has become common practice by workers within the technology sector to include in their LinkedIn profile labels indicating they used to work at a specific platform, e.g., “ex-Meta” or “ex-Twitter.” The lead researcher used the LinkedIn direct messaging feature to contact individuals meeting the study’s inclusion criteria and, upon positive affirmation from the interviewee, collected email addresses to schedule interviews and provide information about consenting to participate in the study. To be included in the study, participants had to self-identify as having worked (or currently working) in a T&S related role within the technology industry and had to have either been laid off or changed companies since 2020.

Given the sensitivities around layoffs, job hunting, and the relatively small nature of the T&S industry, participant privacy was paramount. Researchers did not collect any identifying information about the participants, and, accordingly, we do not list the individual roles, companies, or biographical information of participants—similar to prior studies with vulnerable participants [43, 48]. As a collective group, the 20 interviewees represented a range of roles within the T&S profession, including user research and design, data engineering, product and program management, policy development, enforcement, content moderation management, and director-level management of T&S related operations. All participants worked for US-based technology companies, though several were not U.S. located. Most interviewees had current or past experience working within social media platforms, with others performing T&S related roles at gaming platforms, AI-related technology platforms and start-ups, and online marketplaces. Most of the participant pool experienced a layoff in recent years, with most occurring during the 2022 wave of tech layoffs. Several chose to leave T&S positions for other opportunities, with most moving away from roles at major platforms to instead take positions at vendor and BPO (Business Process Outsourcing) companies or to become self-employed consultants.

To further ensure anonymity, several steps were taken during the interview process. Interviews were recorded for transcription then deleted after the analysis period. Transcripts were anonymized, with interviewees given pseudonyms used throughout the findings and discussions sections. We redacted mentions of company names, product names, and professional roles from transcripts. Following the interview, we provided participants with a copy of the anonymized transcripts and allowed them to further redact any information they believed may re-identify them. Approximately a quarter of the interviewees asked for further redaction to be made.

3.2 In-depth interviews

Interviews were semi-structured in nature designed around a broad interview protocol. This allowed for conversation to be directed by participant answers while touching on several key issue areas shared across interviews. The interview protocol included four sections: Background—capturing interviewee’s experiences working within the T&S industry and their perceptions of public knowledge of T&S work; Job market—asking about the recent state of the industry’s job market, experiences job-seeking after layoffs and/or career changes, and desires to stay in the industry; Content moderation—focusing on attitudes towards content moderation and other routes to achieving T&S online; and finally, Election 2024— specific questions about the role of the T&S industry in securing safe, productive and democratic online conversations ahead of the 2024 U.S Presidential election and/or other global elections. Interviews lasted 45 minutes on average and were conducted mainly by the lead researcher and a second member of the research team. Interviews were conducted through Zoom, with audio recordings taken and later transcribed using AI-powered transcription software Trint. Transcripts were then corrected for clarity by the lead researcher and anonymized as laid out above. The resultant data set totaled 219 pages of interview transcripts.

3.3 Analytical method

The research team undertook an initial open coding exercise of a random subset of transcripts (totaling six interviews). Researchers took a grounded theory approach to coding [12], beginning with a round of open coding that atomized transcripts into discrete parts labeled with codes representing their central theme. The open coding process resulted in 108 codes applied to a subset of six transcripts, each related to at least one atomized quote, with the most popular code used twelve times. The research team then met to undertake a collaborative clustering exercise using the online collaborative software Miro. Researchers engaged in an axial coding exercise to refine the 108 codes, combining similar and/or duplicate codes and grouping codes into emergent themes to create a complete codebook (see 1). The resultant codebook contained 11 overarching themes with 45 subcodes. Following this collaborative exercise, the research team applied the refined codebook to the entire dataset of twenty transcripts. The team then met again to discuss the themes, grouping them according to initial research questions and identifying the most pertinent findings to discuss in this paper.

Figure 1:

Figure 1: A screenshot of a collaborative Miro board used in our initial clustering exercise.

4 RQ1: How have T&S professionals experienced and navigated job precarity and professional upheaval in recent years?

4.1 Job market upheaval and potential renewal

Most (n = 17) of the T&S professionals interviewed for this study had moved jobs in the past couple of years due to the widespread layoffs across the tech industry. Several had chosen to leave roles (n=3), primarily due to professional burnout caused by the intense nature of T&S work. All interviewees had navigated the job market at least once (some multiple times) during the 2020-present period. Collectively, they shared an overwhelmingly negative perception of the job market when actively seeking employment. This is attributed to the ongoing economic downturn for the tech industry and the nascency of T&S as a subfield, leading to difficulties around defining its worth within an increasingly lean tech workforce. Interviewees disagreed over whether they believed T&S was disproportionately impacted by layoffs. For some, cutting the T&S team felt intentional and in line with a lack of support from top leadership at the platforms where they had worked. Others saw the layoffs as collateral damage within the broader tech downturn. Kit remarked that they believed that the initial wave of layoffs were broad, with later waves potentially more targeted towards T&S work— “It might have been maybe [more targeted] in the year following that, as companies were much more attuned to targeted cost savings, rather than just like ‘Oh my God, investors want a whole bunch of people-cost savings.’”

On a more positive note, several interviewees (Beth, Jess, Ezra) noted signs that the job market may be recovering. Jess commented that “starting in January [2024] there were definitely more jobs being posted…2023 definitely seemed quite bad. 2024 has been a little better in terms of volume.” Beth pointed towards a state-of-the-market report about T&S that shows a “growth factor for the industry, not a cost center,” given the prospect of increasing regulation and media scrutiny around platforms. Ezra highlighted that while job opportunities were growing, many of the T&S roles available were specifically engineering roles rather than broader policy, research, and management positions, leading him to comment that he didn’t believe the industry was “fully recovered.”

4.2 Professional and personal burnout within the T&S industry

Interviewees spoke about professional and personal burnout due to working within the T&S field. For two interviewees, this caused them to leave jobs to take sabbaticals or work in different roles (Jolene, Sarah). Sarah argued that her job move was motivated by burnout at her original company coupled with fears around the broader industry contraction:

“I was looking to move. I was really burnt out. And just, like, feeling stagnant. And I would say it was partly… influenced by the economy at the time where, like, I was really trying to make a career shift. And I will say for me, like, even though I wasn’t impacted by a layoff, the layoffs and the stagnation in the industry made my career shift much more difficult.”

Others persisted in their roles despite burnout and then navigated negative well-being consequences when laid off (Emily, Beth, Ezra, Noah, Robin, Lucy, Jacob). Emily highlighted the lingering mental effects of T&S related work and, in particular, exposure to problematic content:

“I didn’t realize what harm that [work] was causing me, because I kind of put up a shield and blocked everything out. But what I was not paying attention to was the shoulder pain in the neck pain. Right? Like. Or maybe just the signs of burnout. And then once I was out of that role, I was like, Holy cow, I was burned out, or, hey, I’m having flashbacks. The content really did bother me.”

Burnout—because of the extreme nature of the content T&S professionals are directly or indirectly exposed to, or due to the professional pressure of the work—was prominent across interviews. Adam, who did not directly engage with content but managed a large team that did, argued, “PTSD is a term thrown around a lot, like oh, you get PTSD from this work. Some of the experts I’ve talked to actually have more specifically narrowed it to dissociative disorder.” This was further confirmed by Ezra, who comments that if extreme content exposure no longer concerns you, it is a sign to seek help. “I don’t want us to ever get to the point where we’re normalizing bad things.” Taylor further attributed burnout to the increasing scale of T&S work and a lack of support to undertake it— “We do it as a labor of love, but there’s no support behind this work.”

Navigating a difficult job market compounded burnout, especially given the financial pressures of long-term unemployment and, for some, immigration-related concerns. Several interviewees (Paul, Lucky, Lucy) saw their layoff coming in advance due to longstanding upheaval at their companies. Lucky explained that they tried to leave before the cuts but could not secure alternative employment due to a tidal wave of layoffs;

“I started applying for jobs before I was laid off…And, you know, so many people were getting laid off, and I think it just became like a wave. So you’re searching, like, every week, you know, X amount of employees lay off at this big company, X amount of employees laid off here.”

Paul, on the other hand, stayed with the company despite impending layoffs because the company was sponsoring their visa.

“They [non-citizens] would kind of just, like, be burned out for this whole thing because this was a slow moving train wreck, right? Like for six, eight months, we just saw this shit show unfolding in front of our eyes and just wondering what’s going to happen to [my job.]”

Paul went on to say that their American colleagues took time off to recover from burnout while he was “out the gate running” with only 60 days to secure another employee-sponsored visa. For most interviewees, securing another position took several months, with the average job searching time being approximately 6-9 months (Emily). Several interviewees said they took jobs with lower salaries or roles that were not their primary preference in terms of platform or role (Jacob, Paul, Sophia).

Despite interviewees all securing employment, a sense of precarity was still pervasive, contributing to continued burnout, even when individuals had found different roles and moved companies. Lucy commented that her past experiences with layoffs were “honestly quite, quite terrible. It just felt very inhuman at times. So I don’t know if I’ll necessarily ever feel secure.” Further, two interviewees (Lucy, Emily) explained that they have now built up additional emergency savings in fear that they will experience economic hardship because of T&S-related layoffs in the future.

4.3 Shifts in T&S roles: towards “T&S as a service”

In response to market precarity and burnout from working at major platforms, several T&S professionals had moved out of “Big Tech” to work on T&S related issues at vendors (such as content moderation firms and Business Process Outsourcing firms) or were self-employed as T&S consultants (Jess, Beth, Adam). Accordingly, rather than full-time in-house work at platforms, interviewees highlighted a growth in consulting opportunities and short-term contracts. As summarized by Lily:

“It felt like there was like a very sudden influx of, like contract opportunities where it felt like either from positions that had been laid off, especially at some of the bigger tech companies, that they had been outsourced to short term contracts, like less than a year sort of thing”

In addition, Beth argued that the increase in self-employed T&S consultants was “in part…due to layoffs, but also part of it’s the natural evolution as an industry matures.” Adam characterized the growth in consulting as more akin to a pendulum shift—with companies course correcting following massive hiring during the pandemic, undertaking mass layoffs, and then looking to consultants to fill the gap. Accordingly, Adam speculated:

“You’re going to see more and more T&S as a service, whether it’s consultants like myself, whether it’s small shops that go, “hey, we do this one thing and we’ll do it for these platforms that just threw everybody out.”

4.4 Public misperceptions of T&S lead to tangible backlash

Interviewees spoke about broader trends of professional upheaval—notably shifts in how their industry was perceived and a marked increase in public awareness of the profession. Interviewees overwhelmingly agreed that there exists a mismatch between the realities of their work and the pervading public assumptions about T&S. Noah summarized:

“It is actually one of the most inaccurately perceived fields that I’ve seen.It’s probably the field I’ve been in with the greatest imbalance of user expectations to industry expertise”

Furthering this idea of mismatched public expectations, Beth argued that public perceptions were overwhelmingly limited due to a concentration of attention on content moderation, when T&S encompasses a breadth of work “behind the scenes, that people in this industry do that stop bad things ever happening to people.”

However, interviewees justified the need for content moderation to protect users from extreme and harmful content, including CSAM, spam, and online scams. T&S professionals did not believe that platforms—particularly social media but similarly gaming and marketplace platforms—could allow for a sustainable user experience without intervening in such spaces. As summarized by Lucy: “I’m just really having a hard time imagining platforms like Instagram not having any sort of T&S. I mean, it just would be a cesspool. It would just be awful.”

Exacerbating the lack of visibility of the totality of T&S work, Ezra argued that there’s so much T&S professionals cannot talk about their work— “because it’s obviously done behind the scenes.” Accordingly, many interviewees believed the gulf between what they do and what the public thinks they do can be understood by (a) a (sometimes necessary) lack of transparency around T&S work and (b) a general lack of understanding of how technology works. Ben believed ignorance was intentional—“it can be an almost willful ignorance that people don’t want to understand.” Emily argued that there needs to be more education “not just to users” but also to other civil society stakeholders, including regulators and researchers.

Interviewees highlighted a growth in attention around their work, but one that was not coupled with an increasing (or accurate) understanding of what T&S professionals actually do. At the extreme, the intensification of (negative) public and political attention on their work caused T&S professionals to take more concrete steps to protect their personal safety. Noah, for example, spoke about the death threats they received from platform users who were angered by the policy decisions of their platform— “I’m the whipping boy for all their emotional output”—to the extent that they questioned their future in the industry. Several interviewees (Robin, Taylor, Noah) spoke about heightened online privacy measures they took because of their work at large platforms including locking down or deleting personal accounts and using services like DeleteMe to remove personal information from the internet. In addition, Taylor spoke about not revealing their location for fear (and actual experience) of being harassed in person— “If I’m going to a conference, I don’t tweet about being at the conference while I’m there.”

4.5 Professionalization of T&S as a route to addressing misperceptions

Attending to issues of broader professional upheaval, interviewees pointed to the positive role of emerging professional groups such as the TSPA—which Taylor argued is becoming the “professional center of gravity” for the industry— and the Integrity Institute, in addition to industry conferences such as TrustCon. Beth argued that, post-Covid, T&S was “becoming a profession as a whole versus being like, ‘I do it at Facebook, I do it at X.” However a professionalization and concretization of the field is still in its nascency, with several interviewees (Charlotte, Jacob, Ezra) identifying concrete needs for the industry as a whole. Ezra summarized:

“I think when it [T&S] does kind of get the attention and doesn’t necessarily have a unified body that can speak out in defense of it, then I feel like it makes it a lot easier for it to become kind of the villain and effectively hamper, you know, not its importance, but its prevalence in a positive way.”

Building on this, Charlotte highlighted an “opportunity for standardization” within the industry to reduce wasted resources and unify approaches. This was similarly echoed by Jacob who argued that T&S doesn’t “have a shared lexicon of terminology yet, so people don’t always know that they’re talking about the same problems.” Accordingly, interviewees shared optimism that professionalization efforts could help them better weather some of the issues associated with professional and economic upheaval—but that this professionalization was in an early stage.

Two emergent themes dominate T&S professionals’ experience of job precarity and professional upheaval in recent years. Notably (1) economic downturn leading to a wave of layoffs across tech, heavily impacting T&S teams, and (2) a growing schism between public and professional opinions of T&S work, giving the industry a negative, or at least incomplete, image. To weather these storms, T&S professionals are expanding the types of roles and organizations working in the space including consulting and vendor companies, embracing a “T&S as service” model. Further, T&S is leaning into professionalization through industry associations and conferences to unify and strengthen the field to better argue for its value and correct misperceptions.

5 RQ2: How do T&S professionals view content moderation as a route to achieving T&S on online platforms?

5.1 Recognizing narratives of public pushback against content moderation

Despite the strong case T&S professionals present for content moderation online, interviewees similarly recognized a public pushback against moderation practices. Interviewees highlighted narratives around bias, censorship, and even conspiracy theories about targeted moderation as driving features of public conversations about content moderation. The degrading reputation of the practice has led some founders of T&S tooling companies and investors to shy away from the label. Lauren argued: “I think founders and investors alike are very much trying to stay away from using the language of content moderation at all because it’s just so politicized.” Similarly, Ezra conceded that the language of “shadow-banning” lends itself to a “kind of villainous portrayal because it’s obviously ominous sounding” which they saw as feeding into narratives of suppression and censorship.

Interviewees expressed frustration at the vilification of content moderation practices (Jess, Lily, Lucky, Noah), particularly given its lack of grounding in the realities and nuances of T&S work; “There isn’t a conspiracy to take down certain types of content. There’s a concerted effort to leave certain types of content up there” (Jess). Lucky further pushed back against the framing of content moderation as censorship:

“People call it censorship. It’s not, it’s moderation. It’s the same way we don’t just yell like a bunch of curse words at a bunch of first graders. You’re not censoring. It’s about moderating. It’s about being appropriate in the right context, the right situations.”

Other interviewees (Kit, Noah, Lily, Jess) highlighted how the political climate of the U.S. frames conversations around bias and/or censorship regarding content moderation. Kit conceded that content moderation:

“in and of itself is biased in ways that are difficult to ascertain, but specifically in a polarized US context, and very much caught up in the ways in which those systems are built and architected to take down certain things.”

Jess’ interview elaborated on narratives of bias within the U.S., pointing out that perceptions of a partisan imbalance in content removals can be explained by the fact that different sides of the political spectrum post different kinds of content:

“The reason that more of the people that are getting blocked or content taken down are on this side of the [political] spectrum is because of the types of things that you are doing and posting.”

Politicized public pushback on content moderation was seen to be shaped by national political conversations particularly on the US political Right. Noah argued that arguments surrounding censorship came out of “a bunch of ideological battles that had kind of nothing to do with the content.” Further, interviewees expressed frustrations, particularly within the U.S. context over the relationship between platforms and regulatory actors—who they saw as missing the mark on the realities of T&S, contributing to negative public discourse and distrust. Ben discussed recent congressional hearings on digital safety for children, arguing that politicians used the opportunity to engage in “grandstanding” rather than exploring and advocating for T&S— “how can I be seen to beat up the social media companies about not taking down bad content.” Robin echoed frustrations around legislative actions not aligning with T&S realities— “Utah came up with a law last year saying we don’t want kids online after 10 p.m., or something like that. And you’re just like, how? What? How does a tech company fix that?”

5.2 Partisan polarization around content moderation impacts election misinformation intervention efforts

Interviewees noted a shift since the 2020 election in the appetite of major platforms to engage in election-related work, particularly around election misinformation. Taylor pointed towards the reduction in tools platforms have made available to civil society actors and researchers to research problematic election content as evidence of this— “we’re seeing platforms back away from the stuff they did have, like Twitter’s APIs and like Crowdtangle.” Noah further cemented a notion that “expectations have shifted” in regards to intervening in election misinformation, arguing, “My theory is that a number of companies are trying to test the boundaries of how little moderation they can do.” This he attributed to rising political backlash—led by partisan actors in Congress—against platform interventions into election misinformation and against researchers working on related issues. The current climate around interventions into election misinformation, he argued, was “lose lose” leading to a lack of motivation for platforms to commit resources to the ensuring safe and democratic conversation in the upcoming election—

“Now it’s a lose-lose game. Whether you do enforcements or you don’t do enforcement, you’re going to get nailed anyways. The people being able to sue you into oblivion, the people able to pull you into judiciary committees, the people able to do, etc., have been in ascendance. And the pendulum has swung.”

Navigating the politically-charged discourse around alleged censorship, several interviewees (Taylor, Lucky) grappled with the role of content moderation in supporting — rather than undermining — freedom of speech online, arguing that safety requires moderation, with the resultant safety affording a diversity of speech. Lucky argued that you needed to “be intolerant of intolerance to have a tolerant environment,” with Taylor further expanding that “if your goal is to maximize the amount of speech, you need to moderate a relatively small amount of the really harmful stuff to allow for that broader set of speech.” Taylor went on to argue that “mitigating chilling effects on speech requires moderation.”

5.3 Productive shifts in public regulations of online safety

Though participants felt discouraged from politicized pushback in the U.S., interviewees did see beneficial policy conversations happening elsewhere, specifically efforts within the EU to legislate around online safety issues. Several interviewees saw EU legislative efforts as a route to shoring up T&S work (Jolene, Lauren, Lily, Jacob, Emily) as complying with EU regulations on child safety, data privacy, misinformation, and democratic health compelled platforms to at least have a minimal T&S team. As captured by Sarah:

“I actually do think it does set a foundation in a bare minimum, and companies will adhere to that, and especially in times when there’s cost cutting, it’s like, okay, well, you can’t cut this thing now because it’s required by law.”

While legislation and resultant fines were seen as creating a minimal basis for T&S teams to advocate for their necessity and to justify content moderation practices, at least on extreme content, interviewees expressed concern that these minimums would become the ceiling for T&S work. Lily argued:

“I think I have mixed feelings about, some of the online safety regulation, but I think that it has like the unintended consequence of creating this dynamic where it’s just like race to the bottom of minimum viable kind of options to check a box.”

Charlotte similarly echoed that T&S work becomes “sort of like checking the boxes and dotting your eyes and crossing your T’s.” Analysis of interviews thus highlights mixed feelings around the role of global legislation and online safety, with prevailing support for legislation and more legislation but skepticism around the nature of current legislative efforts and the potential unintended consequences should T&S efforts be driven by risk compliance alone, rather than a proactive wish by platforms to create genuine online safety efforts.

In addition, interviewees lamented how negative perceptions of content moderation, and T&S more broadly, are difficult to overcome given that public interactions with T&S tend to occur around adverse incidents of content takedowns. As summarized by Sophia:

“How people feel about T&S because their first interaction is like, oh, this new feature is so cool, but like, why is my post being flagged for nudity? Or like somebody’s out here to restrict my ability to freely express myself. Like these experiences suck.”

Adam shared a similar sentiment, drawing parallels with the offline world— “everyone loves that we have law and order until they get pulled over for speeding by the police.” Personal experiences with content moderation are thus seen as reifying the negative public image created by partisan actors, necessitating that T&S professionals continuously justify how and why content is moderated.

5.4 The potentials of AI in online safety efforts

Participants similar spoke about the productive potentials of AI in enabling more effective, and less individually taxing, trust and safety work. Beth highlighted how AI was a dominant topic of conversation at the recent TrustCon, with T&S professionals outlining how the technology would actually better equip them to undertake moderation and, importantly, remove the need for humans to be exposed to the most extreme content within moderation efforts. Given the increased appetite for investing in AI within the tech industry, T&S professionals were thus looking to the technology as an opportunity to secure future work within their own field, and as a route to tackling resource-heavy T&S areas such as crisis events and elections.

Further, when asked if any issues were overblown in the current discourse about T&S, several interviewees identified AI as a potentially productive, rather than worrisome, element of online safety. Interviewees were less concerned by the potential for GenAI or deepfakes to exacerbate misinformation and instead saw opportunities for AI technology to be of assistance in election-related and other misinformation contexts.

5.5 Documenting the hurdles of doing content moderation

Beyond navigating public opinion of content moderation, interviewees expressed the hurdles they face in creating, enforcing, and continuously updating content moderation policy. Chief among these hurdles are: (a) the ambiguity of applying policy to real-life content, (b) concerns around setting precedents—particularly on highly visible users— for future actions that may cause backlash or have unintended effects, (c) the interdependencies of work teams within platforms and their goals for the product, and (d) the speed and scale necessitated within global networks. Most interviewees expressed how difficult T&S work is given the innate “grayness” of user generated content and its reception within the platform’s community. Beth highlighted:

“There were oftentimes tensions around what content violated the policy or not…With borderline content…Does it violate? Do you leave it up? Take it down? Do you just demote it?. It’s a kaleidoscope of facets.”

Jolene argued that “there’s gray areas…what’s extreme to someone, isn’t to another” complicating how different T&S professionals treat content but also how users view community guidelines. After determining whether content contravenes community guidelines, T&S professionals must then decide how to intervene on actionable content, something interviewees highlighted as bringing to the fore additional challenges. Jess explained how every decision is a “global decision,” meaning you have to think “not just about the individual piece of content…but the precedent that it’s setting for the future.” Noah ruminated on how the setting of precedents is made more complex when dealing with high-profile users such as political leaders:

“You know, if every single time you’re like, yes, this is a violation, but, we don’t want to take any action on someone because people deserve the right to hear it. And we have a policy for certain public officials…You set up this case where everything becomes acceptable by increments.”

Further, T&S teams are working in tandem with a multitude of work teams within a platform, teams whose role is to increase platform use and engagement—goals that don’t always align with T&S work. Interviewees named “competing tensions” (Sophia) both within T&S teams and between T&S and other product-facing teams in addressing safety issues. Such tensions are often driven by profit motives, but some issues arise from legal and technical interdependencies. Sarah pointed to an example from the EU where product designers had to alter a platform product to meet EU privacy directives but then ran into issues when the T&S team could no longer undertake mandatory CSAM scanning.

Finally, T&S professionals highlighted hurdles to effective content moderation because of the speed and scale of digital platforms. Beth quipped, “I oftentimes say you can have two of quality, speed and transparency, but you can’t have all three at once.” Similarly, Charlotte argued that proactive T&S work is hard to do “because we’re all drowning in volume.” Interviewees argued that the twin issues of speed and scale curtailed the ability of T&S to pursue more innovative and/or more positive solutions to online safety, an issue further exacerbated by a reduction in resources for T&S teams. Taylor argued:

“There’s just sort of a kind of lack of grounding for a lot of this work. And it’s rooted in T&S being expensive and complicated and difficult, and there’s a lack of resourcing to support it.”

5.6 T&S is not a “cost center”

Pervasive across interviews was an agreement that content moderation and other T&S mechanisms were economically vital for platforms. Many interviewees expressly pushed back against the idea that T&S is a “cost center”—a phase commonly brought up within our conversations as a prominent industry narrative (Emily, Beth, Lily, Taylor, Sarah). Taylor argued that T&S is foundational to a functioning and profitable platform: “Foundationally, it’s rooted in the long-term growth and viability of the platform. I believe T&S work is not a cost center. It is a profit enabler.” Emily concurred, arguing that T&S was actually “making money by keeping people on the platform. They feel safe.” Beth agreed with such logic, arguing that T&S attended to the reputational concerns of platforms. Charlotte framed the argument in favor of T&S in a similar manner— “it protects your assets, and your customers are your assets.” Lucy argued that T&S secured profit-making by ensuring platforms are attractive to advertisers— “You need a T&S team. Coca-Cola is not going to run an ad next to a white supremacy post.” Interestingly, several interviewees pointed towards the example of X (post-buyout) as evidence of what happens when T&S is deprioritized. Interviewees argued that the user experience of X was wholly diminished because of a lack of content moderation and flooding of spam and inappropriate content, leaving many to close their accounts on the platform.

However, interviewees recognized that doing T&S properly is expensive and that, despite the strength of their arguments around its economic utility, persuading platform leaders that this cost is necessary or desirable is an uphill battle. Talking about the CEO of a large social media platform, Jess conceded that the role of leadership is to maximize short-term profits (by cutting bottom-line costs) over long-term sustainability, which may come at the expense of the pursuit of T&S measures:

“He’s incentivized to maximize shareholder value. That is his job. That is almost his only job. And so he’s not going to make decisions that focus on doing the right thing unless they also maximize shareholder value.”

Adam shared the same sentiment, likening the profit motivation of platform leaders to the “stories of ‘hey we knew this car would hurt people, but we did the math and we paid less in litigation than trying to redo the car.” Accordingly, T&S is seen to suffer as it is deprioritized due to sometimes being in opposition to profit-making, and also struggles to quantify its own value. Charlotte lamented that “the return on investment isn’t totally obvious” for T&S, a narrative expressed across interviewees who highlighted the struggle of measuring and/or quantifying how T&S work translated into profit. Several interviewees extended arguments of measurability, suggesting that companies just don’t want to measure harm or ignore harm when it is documented. As summarized by Jacob— “And is the net good greater than the net harm? That’s not a metric that they measure.”

In addition to the lack of desire for measurability from the top-down, the incentive structure for employees similarly curtails T&S work from the bottom-up (Jolene, Sophia, Kit, Jacob). Jolene argued that only product-driven positions retained job safety— “If I’m not in software engineering, developing a new pair of goggles or a headset, or coding to improve, I’m a cost. I’m not an investment.” Sophia echoed the same concerns, positing a continued question of “how does this [T&S] impact, it’s not part of the promotion package?” Kit further captured the individual limitations of pursuing innovative T&S work within a structure where it is both challenging to measure and does not necessarily align with company incentive structures:

“The incentive also isn’t necessarily there for people to do that, kind of like very provocative and exploratory work, which might actually be very critical towards these larger conversations about like how propaganda might be impacting people, when you could have just like shipped something really quick in three months and that would have moved a bunch of engagement metrics and then you’d get a promotion for that.”

Finally, interviewees highlighted that—beyond issues of measurability—even if T&S work is valued internally, the public backlash against content moderation and related work results in a poor return on investment for companies, especially with the hyper-focus on incidents of failure. Interviewees presented a “damned if you do, damned if you don’t” perspective of T&S work, wherein even if a company invests solidly in content moderation, it will continue to receive criticism from outside actors regardless of efforts. As summarized by Jacob:

“Content moderation team is not cheap. It’s a huge capital investment for the company. Not just to build, but to maintain and to receive the amount of criticism they receive for getting things wrong over and over…I think they’re really trying to get it right. And they’re faced with, conundrum where people disagree in general, so they’re never going to get it right.”

T&S safety work is thus seen as mostly invisible but highly visible in incidents of (perceived) failure, making it a difficult and often thankless task. As captured by Jess: “online safety isn’t a destination. You’re never going to reach it.”

In sum, content moderation remains the central pillar of T&S work. Negative public and political perceptions have exacerbated the extant hurdles the work already faces by forcing T&S professionals to continuously advocate why content moderation and T&S work more broadly, is important— in the face of misperceptions about the work and politicization, particularly within the US. Justification is further necessitated by the difficulties of measuring the impact of content moderation on product success and, most importantly, profit. T&S professionals are steadfast in their belief that, despite prevailing chatter about T&S being a “cost center,” their work ensures that the company’s product is safe, meets legal requirements, and is thus successful.

6 Discussion

The following discussion section focuses on three themes emergent from our analysis. In doing so, we attend to the role of CHI researchers in supporting online safety efforts during the ongoing upheaval of T&S work and the potential for CHI research to meaningfully contribute to design interventions, policy recommendations, and empirical research to minimize digital harms. Below, we examine three themes: (1) the growing professionalization of T&S as a route to shoring up T&S work against economic and political threats, (2) increasing the transparency and measurability of T&S work to allow for more accurate public representation, and (3) the future of T&S tooling and the use of AI to allow for T&S work to be more sustainable and reduce burnout.

6.1 Strengthening Trust & Safety as an industry

Interviewees documented the shakiness of T&S as an industry. Despite the field becoming more concrete in recent years, it is simultaneously facing an undoing with public attacks on its work and widespread layoffs. The industry is naturally on the defense, with interviewees defending the very necessity of resourcing specific teams dedicated to online safety and advocating for content moderation to attend to the extremes of online speech. However, interviews highlight how this defensive orientation also lowers the bar for more innovative or protective T&S work—indeed, interviewees argued that T&S feels like a box checking exercise to meet minimal legal requirements. Interviews highlight a tension between the broad range of roles and activities T&S professionals undertake to ensure safety is written into the very design of digital platforms and the reduction of T&S work to content and account removals. Frustratingly, as resources diminish for T&S work, content moderation becomes the primary T&S work that can be undertaken and is increasingly outsourced to BPOs and vendors where previously laid-off T&S professionals have moved.

Extant CHI research highlights a role for researchers in supporting T&S professionals in advancing their work in the face of diminished resources and, importantly, advocating for an expansion of resources. For example, past CHI research has empirically documented the lived experiences of (volunteer and paid) content moderators [63] and the negative impacts on mental well-being content moderation entails. Such work is vital for advocating for greater resources from platforms for safe and sustainable content moderation and aids T&S professionals in justifying the costs of content moderation. Moreover, past research has developed and tested routes to undertaking human-led content moderation in more efficient [61, 67] and less damaging ways [10, 35], again providing T&S professionals with evidence to argue for resources. Furthering this, CHI research can play a more substantial role in elevating the work of T&S professionals, both through the design of novel tools to allow for the industry to operate despite cuts and through research that empirically documents the relationship between T&S work and online harms (further explored below). Moreover, future T&S work should look to document the professionalization of the industry and, in particular, it’s expansion of resources and documentation, industry norms, and emergent ethical practices. Such academic work would help legitimize the industry and cement its emergent practices.

6.2 Correcting societal misperceptions of content moderation and other T&S efforts

An explosion of public attention on T&S work in recent years has unfortunately not been met with increased (or accurate) knowledge of the field. Interviewees expressed immense frustration at the lack of awareness of the realities of T&S work and the nuances and difficulties of the job. At a base level, misperceptions damage trust in T&S work, allow for politicized narratives of T&S to take hold, and, at the extreme, threaten the physical and emotional safety of T&S professionals. Again, the advent of professional resources through T&S associations and conferences seeks to provide resources to counter some of these ill-effects. However, addressing the root causes of misperceptions remains incredibly difficult. That users mostly encounter T&S work through political narratives or through negative interactions with content removal hinders the ability of T&S professionals to counter prevailing misconceptions. Further, interviews highlight how transparency within the industry is a complicated ideal—much of the work of T&S needs to be invisible to counter bad actors and because of the nature of ensuring users do not encounter extreme content in the first place. This opacity likens T&S work to infrastructure—vital and mostly hidden until instances of failure [62]. In turn, this reduces the ability of T&S to quantify their necessity to platform profits, making them uniquely vulnerable to layoffs in times of financial crisis for tech firms.

Accordingly, countering misconceptions and educating users (and legislators) on the realities of what is required to ensure online safety may better come from third-party institutions, such as academic research and non-profit professional organizations. Further, current T&S mechanisms are far from perfect—as highlighted by the hurdles and lack of resourcing interviewees surfaced—requiring accountability from external actors such as CHI researchers. As such, there is a need for research to accurately document the work of T&S professionals to counter misperceptions surrounding their work, and to empirically document how human-led T&S efforts are vital to the success of automated online safety tools. Similarly, as platforms turn against transparency (including reducing access to vital research APIs) and considering the complexities of transparency-from-within, academic research can act as an accountability mechanism. Past research by Register et al. [51] on perceptions and impacts of discriminatory content moderation on Instagram and work by Mayworm et al. [40] to develop content moderation resources for marginalized users highlights productive routes to taking on this challenge. Moreover, as T&S professionals look to understand and productively counter societal discourse around their work, CHI and CSCW research surrounding algorithmic folk theories captures vital insights into how the public use and (mis)understand platforms and content moderation practices. [13, 37, 39].

6.3 Despite hurdles and backlash, content moderation is here to stay

Finally, interviews highlight how content moderation is a central pillar of online safety and, consequently, remains a core T&S activity. However, in the face of backlash and the hurdles of undertaking content-based interventions, there exists a need to bolster the range of tools and practices undertaken by T&S teams—embracing the “Swiss cheese” model of multiple layers of design and intervention. This requires T&S not to be an afterthought in product design. Instead, platforms must embed considerations of potential abuse and misuse in developing and deploying all products — the identification of such considerations is an area in which CHI research is well-positioned to undertake. However, as interviewees highlight, the cost of such approaches is prohibitive to platforms, allowing them to justify downgrading safety concerns for short-term profit reasons. Again, there exists clear opportunity for CHI research to bolster online safety efforts in the face of this downgrading through the development and empirical testing of design features and safety tooling. The potential of AI-driven approaches to content moderation, for example, has already been documented in extant CHI research [31, 34]. Moreover, future CHI research should look to measure the impact of a broad range of T&S interventions on online safety and the minimization of harm—including algorithmic de-prioritization of problematic content and users, content flagging using crowd-sourced and professional fact-checking, and transparency and education around community guidelines and enforcement decisions (such as [27, 57]). Such research will allow T&S professionals to more finely tune interventions to minimize unintended consequences, bolster their necessity to unsympathetic leaders, and provide a data-driven grounding for legislators working to combat online harms.

7 Conclusion and Limitations

This paper highlights the tumultuous nature of T&S work. T&S professionals work to minimize online harms within a tech industry that increasingly devalues the necessity of such work. Analysis reveals a schism between professional perceptions of the work of T&S, including but not limited to content moderation efforts, and outside perspectives on the proper actions needed to secure online safety. This has been amplified in recent years by a politicization of T&S work, and content moderation in particular, as “censorship,” as being biased against specific partisans, and as chilling freedom of speech. T&S professionals reject these notions and highlight concerns that misperceptions have degraded the resources dedicated to their work, particularly around high-profile events such as global elections. In turn, this devaluing has meant that T&S professionals have thus been vulnerable to layoffs during the widespread tech downturn experienced in recent years. In the aftermath of these layoffs, T&S professionals are expanding the kinds of roles they take on to stay in the industry including taking on short-term contract positions, taking consulting roles, and working for vendors providing content moderation and risk compliance. Our analysis highlights a positive role for CHI researchers in helping T&S sustain and grow through an empirical documentation of the necessity of T&S teams in reducing digital harms, the development of tooling to allow for more efficient and safer moderation work and by allowing greater public visibility into the breadth and realities of T&S work.

7.1 Limitations

This research is naturally limited by its focus on professionals working primarily in the U.S. or for US-based T&S teams. As documented, US perspectives on trust & safety work are uniquely framed by specific cultural conceptualizations of freedom of speech and the role of platforms in supporting and ensuring this speech. Future research should look to document the experiences of T&S professionals working in different cultural contexts, including decolonial perspectives of content moderation and online safety [59]. Moreover, as an interview study, the perspectives of the industry are naturally limited to the sample of participants that could be secured for this research project. Accordingly, there likely exists a broader range of opinions on the state of the industry and individual experiences of layoffs than was captured within this study. Future survey-based research may be educative in capturing a broader range of industry perspectives and experiences.

References

[1]

M. P. Angel and D. Boyd. 2024. Techno-legal Solutionism: Regulating Children’s Online Safety in the United States. In Proceedings of the Symposium on Computer Science and Law. Association for Computing Machinery, New York, NY, USA, 86–97.