Abstract
Trust and Safety work mitigates certain social risks and harms of digital technologies. In a landscape that is only slowly establishing regulatory oversight mechanisms, Trust and Safety work performs a type of internal governance by attempting to mitigate disinformation, online harassment, extremism, and other harms accelerated or amplified by technology products and services. With origins in content moderation, Trust and Safety work now also includes product advice for online services and IRL (in real life) products. Attention to Trust and Safety work can complicate a monolithic view of what tech work is and who does it, as well as point to locations for intervention as we seek to promote Public Interest Technology in globally impactful sociotechnical systems.
1 Introduction
The power of digital platforms and everyday technology to host and facilitate a catalog of intended and unintended harms – including misinformation, extremism, surveillance, harassment, and violence – is a growing societal concern. As an example from the US context, immediate analysis of mass shootings includes questions about whether the shooter posted about his violent intentions online first, was radicalized in an online community, had a history of harassing or stalking women online, chose the location from an online map, or even chose to livestream the horrific event. Companies act swiftly to pull down content and cooperate with law enforcement. The professionals who actually perform that work are increasingly relied upon and fall under the heading “Trust and Safety” professionals.
Much attention has been focused on the ways in which social media platforms are implicated in the organization of protest movements, coups, and state-sponsored repression (Helberger et al. 2018). People from targeted, marginalized communities have found both opportunities for connection (Russell 2020) as well as deeply disturbing and violent attacks in online spaces, which move offline as extremists who were radicalized online bring physical violence into real world places (Ganesh and Bright 2020). Within the personal sphere, smart home devices impact not only owners of those products, but other users in the home, visitors, and neighbors (Levy and Schneier 2020; Tanczer et al. 2018). Therefore, society has an interest in these spaces and gadgets being safe.
Debate has been vigorous about which tools are needed to govern technology in order to reduce harms such as these, and to promote Public Interest Technology. Borrás and Edler (2014) define the governance of science and technology innovation as intentional efforts by societal and state actors to guide and shape how technology is created and spreads throughout society in order to address societal impacts. Proposals to govern internet and social media platforms from the outside, or to enact external governance, include regulation, legal action, standards, and consumer choice undertaken by some combination of governments, courts, markets, and citizens (DeNardis and Hackl 2015; Ganesh and Bright 2020; Helberger et al. 2018; Riley and Morar 2021; Robinson 2020; Suzor et al. 2019). The recent passage of the Digital Services Act in Europe is an example of these efforts. However, policymakers’ efforts to craft effective legislation and regulations – and to enforce those measures – is hampered by the complexity and opacity of technologies (e.g. algorithmic decision-making, data transparency), widely differing contexts and impacts for users across geographies and jurisdictions (e.g. encryption, so-called “low-resource” languages), and a mismatch between the pace of commercial product development and policy processes. While the effective governance of technology towards the public interest may best be achieved through a coordination of efforts across sectors and jurisdictions, Suzor et al. (2019) argue that companies also have a responsibility to address harms facilitated by their products.
To the extent that companies may have an obligation to address and mitigate harms that happen in the course of their business (Helberger et al. 2018; Suzor et al. 2019), people are needed within the companies to respond to harms and shape future product design. While some external oversight may be put in place, the competitiveness of the market and protections for intellectual property limit transparency (Brand and Blok 2019). This tension creates a need for a tool such as Trust and Safety (T&S) work within companies to respond to, mitigate, and prevent the damaging impacts of misused technology, i.e. to shape technology from the inside. As one posting from TikTok described the work arising “in response to society’s growing concern about the role of big tech in society.”
T&S work is an evolving approach to the internal governance of technology within private technology companies. However, the field lacks clear definition and scope. Furthermore, internal governance of technology is only one dimension of governance, and T&S work is only one approach to internal governance. This research contributes to an appraisal of T&S work as internal governance, even as other efforts are ongoing to establish or strengthen mechanisms for external governance. This research can contribute to ongoing debates about the effectiveness and appropriateness of internal governance in balance with external governance as we seek to promote Public Interest Technology in globally impactful sociotechnical systems. Ultimately, improvements to T&S work and public debates about governance must be measured by the tangible benefits for people and communities who experience the risks and harms of ubiquitous technologies.
This paper provides a portrait of T&S work from the dual perspectives of technology companies and the professionals who perform this work. The findings contribute to larger societal conversations about how to shape the development and diffusion of the technology that is woven into our daily lives and into our societies. To those ends, the author collected and analyzed 112 job postings and conducted six semi-structured interviews with T&S professionals between Fall 2021 and Spring 2022, and then examined the degree of alignment between companies’ and professionals’ conceptions of the emerging field of Trust and Safety.
The first major section of this paper presents findings which address questions such as what Trust and Safety work is, what it is not, and the scope and purpose of that work. Six work functions are proposed, as well as the scope of what T&S attempts to counter, and the mindsets and skills common to T&S professionals. The report findings reveal significant trends and challenges facing the field, describes interviewees’ perspectives on the future of T&S, and elaborates on what is needed to meet those challenges and move toward those futures. The findings are followed by a discussion section that focuses on shifts towards the social good and proactive approaches to Trust and Safety.
2 Literature review
Trust and Safety work is an emerging type of work within technology companies that is focused on mitigating risks and harms. Information and communication technologies (ICTs) are increasingly being weaponized to facilitate harms including the promotion of misinformation, extremism, and gender-based violence. In response, civil society and academics are calling on private companies to take action to reduce harms facilitated by their products (Dragiewicz et al. 2018; Suzor et al. 2019; World Wide Web Foundation 2021).
It seems that digital and everyday technologies can be used for good and for ill. However, technology is not neutral (Benjamin 2019), and it is argued that society can use governance tools to intentionally shape our technologies towards futures that are equitable, humane, and sustainable. The emerging field of Public Interest Technology promotes user-centered, ethical technology used in the public sector and in essential services such as education, healthcare, elections, and environmental stewardship. However, many of those technologies are built and maintained by private companies. Additionally, technologies which directly and indirectly impact basic safety, wellbeing, and our social fabric are apps, websites, and gadgets that are entirely market-based. Public Interest Technology is equipped to contribute to a suite of tools for societal governance of private-sector technology products including externally-driven policy, regulation, and legal action as well as voluntary, internal approaches undertaken by industry and companies. Efforts in the latter category include the development of industry standards, codes of conduct, and Trust and Safety work.
There is not yet a clear definition or standardization of what constitutes Trust and Safety work. The recently created Trust and Safety Professional Association (TSPA) (n.d.) describes the field as a “global community of professionals who develop and enforce principles and policies that define acceptable behavior and content online,” though preliminary research suggests that Trust and Safety work also happens at companies which produce tangible ICTs like IoT devices and item locators. Generally, we may say that Trust and Safety teams work to respond to individual incidents and more broadly counter the misuse of their companies’ products.
There is a pressing need for Trust and Safety work, along with other approaches to technology governance, to counter harms experienced by individuals and communities who are subject to surveillance, control, and even violence that is accelerated and amplified by technology. Lenhart et al. (2016) and a report from Amnesty International (2018) provided some of the first large-scale data on experiences of online harassment. Dragiewicz et al. (2018) among others, reported the ways that phones, social media, location technology, text messaging, account access, online information, and images were used for coercive control of intimate partners. Freed et al. (2018) described the use of apps and account features as spyware. Tanczer et al. (2018) sounded an alarm about the hijacking of IoT smart home technologies by one member of a household against others.
The myriad ways in which ICTs are misused to cause harm match the variety and scope of the tools produced by technology companies. For example, large companies like Google or Apple curate app stores, develop and market Internet of Things (IoT) products, and manage communications channels (e.g. email, messaging, and file sharing), all of which can be tools used to perpetrate harms such as gender-based violence while simultaneously acting as tools to enhance individual and community safety, social connection, and wellbeing (Dragiewicz et al. 2018; Freed et al. 2018; Leitão 2021; Messing et al. 2020). Users also subvert intended and unintended uses, both to cause harms, as well as to resist those harms. The dual nature of technology use, along with an increase in the interconnectivity of devices and systems, challenges our ability to identify vulnerabilities as well as design solutions (Schneier 2017). Finally, technology dissemination and experiences of harms that can result from ICT misuse are global (Hobbis 2018; Malanga 2021; Silva and Oliveira 2021). This challenges governance structures that are limited by a geographic jurisdiction.
However, widening the sphere of who does technology, and broadening our vision beyond a “move fast and break things” mentality may go some way to address the complexity and multivalent challenges we face. There are emerging examples of how a shift in design has the potential to mitigate some of the harms resulting from the use of technology. In 2019, a team at IBM released a set of “coercive control resistant” design principles (Nuttal et al. 2019), scholars have suggested broadening the scope of cybersecurity concerns beyond an emphasis on nation-state and corporate spheres to include interpersonal scenarios and threat models (Doerfler 2019; Levy and Schneier 2020; Slupska and Tanczer 2021), and researchers working in collaboration across academia and industry have developed methods to detect spyware and creepware, for example, in app stores (Chatterjee et al. 2018; Roundy et al. 2020). These changes in the design of products and systems may involve the work of Trust and Safety teams, who interface with those who experience harm, as well as civil society organizations and researchers who can offer useful perspectives.
Trust and Safety work could potentially shift the design of products, policies, and incident response teams to mitigate harms through the inclusion of the perspectives of survivors and those who support them. McGuinness and Slaughter (2019) applaud a growing shift towards including a wider array of voices in decisions about technology. Genus and Stirling (2018) described human-centered, user-centered, responsible innovation. Others have called for increasing the participation of those who have historically been excluded from technology design and decision-making through co-design (Leitão 2019) and community-led design (Costanza-Chock 2020; Escobar 2017).
3 Methods
This research examined the degree of alignment between the aspirations and conceptualizations of companies and professionals regarding Trust and Safety work. This research draws on 112 job postings collected from job boards that are open, public, and maintained by third-party organizations between August and December 2021, and six semi-structured interviews conducted with current or former Trust and Safety workers between March and April 2022. Also included is a brief subsequent analysis of companies hiring in the first half of 2023, based on 92 job postings collected during that time.
Postings that contained the words “trust” or “safety” in the job title, the team, or the category as assigned by the job board. Postings from the public sector and nongovernmental organizations were excluded. Interview participants were recruited by direct email, and by postings and emails distributed by the TSPA and other organizations that host job boards in the Public Interest Technology space. Demographic information about interviewees or their employers was not included to protect interviewee anonymity due to the sensitive nature of their employment. In summary, all interviewees were working in the US. Of the interviewees, 5 were working in T&S roles at the time of the interview, one had formerly been employed in T&S, and three had worked for more than one company. Interviews were semi-structured to elicit participants’ reflections, and the interview protocol was reviewed and approved by Arizona State University’s Institutional Review Board. The interviews were recorded and transcribed, and held securely to protect participants’ privacy. The postings and interviewees represent mid-level Trust and Safety positions, and do not include lower level positions within companies or with vendors who provide outsourced T&S services in the US or internationally.
In preparation for the research, I solicited the input of stakeholders and reviewed relevant web content from the Trust and Safety Professional Association (TSPA) and a sample of technology companies. This preparation informed the drafting of the semi-structured interview protocol and a selection of expected codes for thematic analysis of the job postings and interview content. I added emergent codes and uncovered themes iteratively throughout ongoing analysis and in consultation with stakeholders and interview participants.
4 Findings
4.1 What is Trust & Safety?
One interviewee summed up Trust and Safety as “maximizing the good, minimizing the bad,” effects of technology on individuals, communities, and societies (Interviewee 1).[1] However, safety work is not limited to responding to reports from users after they have happened. Findings from interviews with T&S professionals and from job postings I analyzed spoke to the use of intelligence, research, and data analysis to detect threats or harms that have not been reported, as they are happening in real-time. Nor is safety work limited to online spaces; internet connected devices in our homes and other real world environments also have risks and harms which T&S addresses.
The other half of the work is Trust, which relies on Safety. Trust is focused on making people feel safer using a company’s products, and protecting the reputation of a company in the minds of the public. The question here, as one interviewee put it, is:
On this point, the data from the interviews and the postings show an alignment between companies’ and professionals’ perceptions. The importance of the Trust side of the equation is evident in the job postings I analyzed. These included words like: “bolster,” “ensure,” “build,” and “promote” users’ “trust in [the] platform.” Or making products “secure,” or “safe,” or “protect[ing] the platform.” One interviewee summed up T&S as “translat[ing the company’s] values into the principles and enforcement mechanisms” (Interviewee 5). However, data from both the interviews and postings reveal a spectrum of beneficiaries from the brand to users to the greater public interest, and promoting the “good,” “wellbeing,” and “health” of individuals and society.
Some interviewees also addressed this question by saying what T&S work is not. It is not a profit-generating part of the business; rather, it is a “cost-center,” meaning an area in which expenditures are made without an obvious Return on Investment (ROI). This has implications in terms of how it is resourced and prioritized relative to other parts of a company, as will be discussed in Trends in Trust and Safety, below. Additionally, there were a variety of perspectives on whether privacy or fraud were part of T&S, or rather overlapping but distinct areas of work.
4.2 Six functions of Trust & Safety work
Trust and Safety involves an array of functions spanning from responding to harms that have already occurred to preventing those harms from happening in the first place. T&S protects the “privacy, personal safety, financial security, community connections, mental health, and well-being of technology users,” in the words of one interviewee (Interviewee 4). T&S work includes shaping and implementing policies, and developing processes needed for T&S, as well as advising product teams and other relevant parts of the company.
I propose six functions of T&S work which I developed based on my analysis of the job postings and interviews: 1) response, 2) policies and processes, 3) research and data, 4) internal tools, 5) product advice, and 6) collaboration and communication (Table 1). Not all six functions are present in every T&S position, or even in every team within a company.
Table 1:
Frequency in interviews and postings of the six functions of Trust & Safety work.
Function | Tasks | Frequency–Postings | Frequency – Interviewsa |
Incident response | Moderation | 17 | 6 |
ㅤ | Receiving reports and requests | 17 | ㅤ |
ㅤ | Investigation | 19 | ㅤ |
ㅤ | Escalation | 31 | ㅤ |
Policy & process | Policies | 42 | 6 |
ㅤ | Processes | 57 | ㅤ |
Research & data | Data analysis, researching, or sensing trends | 48 | 5 |
ㅤ | Creating metrics | 34 | ㅤ |
ㅤ | Reporting on T&S work | 31 | ㅤ |
Internal tools | Tools for internal T&S work | 31 | 6 |
Product advice | Product development (user-facing) | 41 | 4 |
ㅤ | Prevention | 7 | ㅤ |
Collaboration & communication | Collaboration | 76 | 5 |
ㅤ | Communication | 26 | ㅤ |
ㅤ | Training/education (employees, moderators, users) | 19 | ㅤ |
ㅤ | Serving as a subject matter expert | 15 | ㅤ |
aInterviewees were not asked directly about “functions,” so these represent organic responses that came up in the course of the conversations.
These functions are interrelated and overlap in many cases, either within one position or throughout a team or a company. The sixth function, in particular, collaboration and communication, is woven through the other five functions, and is also noted as a trend later in this report. As an example, one interviewee (Interviewee 4) emphasized the importance of communication with users in the response function, so that they understand what’s happening, have choices about what to do next, and can document what’s occurred.
4.2.1 Incident response
This first function includes moderation, investigation, escalation, and other operations that are triggered by reports from users or requests from law enforcement or other state actors. Action may also be initiated by analysis of data from incidents or trends that are not reported by an individual user (see Function 3). In short, this function is triggered when bad things happen, and when crises or trends emerge.
4.2.2 Policies & processes
Taking cues or analyzing data from individual incidents or trends, T&S professionals develop and improve policies and processes to guide future responses. This function may also try to prevent future occurrences through community guidelines.
4.2.3 Research & data
T&S professionals can also seek to identify risks, harms, and trends that are not reported or are emerging through data analysis, research, and other sensing methods. Postings described responsibilities including developing metrics and reporting internally or externally on T&S activities.
4.2.4 Internal tools
Working with engineers, product managers, or vendors, T&S professionals develop, refine, and deploy tools for users to report content or incidents, to detect harms or trends, and for response processes.
4.2.5 Product advice
A newer function, particularly within larger companies, is providing advice or consultation to product development teams in order to minimize features or capabilities that could be misused to cause harm.
4.2.6 Collaboration & communication
This function weaves through the first five functions. This function includes serving as an in-house subject matter expert (SME), cross-team collaboration or consultation, training and education (internally or externally), and work with external partners, vendors, and other companies in the context of industry-wide efforts, as well as public policy.
T&S overlaps or collaborates with many other areas or teams in companies. While T&S is not core product development, it does shape product development in some companies. It is not engineering, but engineers contribute to it. It is not marketing, but the Trust aspect especially is something that affects users’ (and society’s) sense of the brand. Interviewees also named other teams that T&S interacts with: fraud, cybersecurity, legal, operations, content moderation, leadership, product development, research, and public policy. All of these teams were also mentioned in “About the Position” sections of the job postings. In some cases, though, interviewees either said these were part of T&S, overlapping, or entirely separate. This highlights the lack of clear definition and scope of T&S as it rapidly grows and formalizes as a field.[2]
4.3 Skills and mindsets of Trust & Safety
Analysis of the interview data surfaced four key areas of skill and experience as important for T&S work: project management, investigation, subject matter or localized expertise, and research, analysis, and critical thinking.[3] Job postings also included these skills, here shown with the frequency with which they appeared in postings:
Analysis, critical thinking, or research (119)
Project management and related skills (51)
Investigation (11)
Subject Matter Expertise in abuse areas (7)
Eight mindsets[4] important for T&S work could be drawn from the interviews: collaboration, communication and diplomacy, curiosity, adaptability, problem-solving, the ability to understand context and complexity, empathy and emotional resilience, and passion. Each of these was found when I searched for them in the postings, here listed in order of frequency in the job postings.[5]
Problem-solving (131)
Communication (73)
Passion (43)
Collaboration (40)
Adaptability or flexibility (25)
Curiosity (16)
Empathy and resiliency (8)
Ability to understand complexity or context (5)
These skills and mindsets appear to map well to the functions of T&S work. For example, the mindsets of problem-solving, collaboration, communication, adaptability, curiosity, the ability to understand complexity or context, and the skill of analysis, critical thinking, or research fit a picture of T&S work as rapidly emergent, varied, collaborative, and contextual.
4.4 Trends in Trust and Safety
Analysis of job postings and interviews reveals six key trends including external pressure, prioritization, formalization, automation, collaboration, and prevention.
4.4.1 External pressure
Interviewees pointed to changing external governance landscapes including potential legislation, regulation, or standards. “The industry is grappling with being in crisis mode all the time and having, oftentimes, journalists raise issues that really expose vulnerabilities to our users and to our brand safety” (Interviewee 1). Another interviewee raised the potential for T&S-related policy, as well as “pressure that’s gonna have to come from society and users and people just being more demanding about the type of experiences they want from a product” to drive further growth and formalization of the field (Interviewee 5).
Within job postings, the terms “compliance” or “standards” most often reference meeting internal policies or standards, or helping users to do so, rather than a smaller number of references to complying with external standards, regulations, or laws. The words “regulation” or “regulatory” referred specifically to privacy regulation in half of the instances, with the other half being generally vague about which regulations were being referred to. The vast majority of uses of the word “legal” in job postings referred to collaboration with the Legal Team within a company, with several mentions of responding to legal requests or ensuring compliance with laws. Across these terms, there was a small but interesting trend of referring to the legal or regulatory environment as emerging or dynamic.
4.4.2 Prioritization
In the interviews, it was evident that professionals experience tight resources and a lack of leadership prioritization for T&S and feel that more of both are needed. All interviewees said lack of resources and prioritization was a challenge. As noted above, T&S is seen as a cost-center rather than a profit-making part of a company. Three said more resources were needed, and two pointed to the need for leadership support or prioritization. “Resources” also means more people to do T&S work, especially those with a diversity of backgrounds and expertise in subject areas and the context in which content or behavior should be understood. These resources should support the time and capacity for T&S professionals and teams to engage in cross-team collaboration, to have a wide diversity of backgrounds, experience, and perspectives to more thoroughly consider the context of a given incident. As noted above, related to the pressures and trauma professionals often experience while doing T&S work, company and leadership support for professionals with those experiences is essential, as is ongoing professional development and cross-company collaboration.
All interviewees felt that T&S work is important. Yet, they also expressed that the importance of the work was not fully appreciated in the rest of their companies. They felt a need to internally make the case for the Return on Investment (ROI) of T&S, as well as explaining the complex and nuanced nature of T&S work and inherent trade-offs externally. Interviewees and job postings both point to a need for tools and skills to access, understand, and act on available data that can drive T&S work and also sense trends. Thirty-four postings listed metrics or measurement among responsibilities and 31 postings included reports. Metrics and reports about them could be reframed as telling the story of T&S internally and externally. Internally, metrics and reports can help other teams and leadership to understand T&S approaches and challenges. Externally, telling the story of T&S work can create ground for parallel or shared efforts to minimize the bad and maximize the good of technology products.
4.4.3 Formalization
My attention was drawn to this research area when I became curious about what seemed to be a hiring explosion. Within three months in the Fall of 2021, I was able to collect 119 job postings (7 later excluded from analysis). The sheer number of postings, as well as their content, along with the interview data collected the following Spring support the perspective that T&S as a field is growing. Said one interviewee when asked about this:
Another interviewee said, “It is just so much more sophisticated than it was … 13 years ago … Many Trust and Safety departments have now expanded” (Interviewee 6). It is important to note that since the steep growth curve when this initial research was done, there followed waves of deep layoffs beginning in Fall 2022. Analysis of job postings in 2023 shows a shift to hiring at smaller, newer tech companies. Looking at 92 job postings from 52 companies between January-May 2023, 65 % were from companies with less than 5,000 employees, founded after 2010. And 43 % were for leadership positions such as “manager,” “lead,” “head,” “director,” or “vice president.” Most postings were for social media platforms or apps, although that is an ill-defined category. The remainder of postings were for an assortment of other products and areas.
The field is maturing and becoming more formalized. Another interviewee said, “I think that over time as Trust and Safety becomes a more professionalized field and as people who have done that work gain more influential positions in companies … [T&S will] be able to get more stuff done” (Interviewee 2). This professionalization highlights a tension between external partners serving as SME’s and a shift to bring that expertise and diversity of perspective in-house. One interviewee suggested that as external scrutiny becomes more intense, companies have concerns about confidentiality when consulting with external SME’s (Interviewee 6). Bringing expertise in-house perhaps helps to mitigate that (although high profile whistleblowers from within companies may undermine that assumption). As companies bring experts from nongovernmental organizations (NGOs), law enforcement, or other fields in-house, there is a need for professional development in the skills needed for T&S work. One interviewee named access to professional development as a key need to meet the challenges facing T&S in the near future (Interviewee 1).
The formalization of the field is also evident in the establishment of the Trust and Safety Professional Association (TSPA), a stakeholder consulted in the course of this research, two connected conference spaces hosted by TSPA and the Stanford Internet Observatory. The latter also started the Journal of Online Trust and Safety in 2021. Industry-led and civil society-led coalitions have also been formed to address T&S issues such as online content (Digital Trust and Safety Partnership), child exploitation online (The Tech Coalition), and online hate (Online Hate Prevention Institute).
4.4.4 Automation
The scale of both online and real-world technology diffusion is massive, and so the number of opportunities for cybersecurity issues, content issues, and bad behavior, all the “bad things,” to show up is huge. This means that the need for T&S detection, filtering, and user reporting is also massive. A posting from Facebook described the need to “protect our community at scale.” Another posting from TikTok included the goal of providing “scalable physical and psychological health solutions.” Overall, the words scale or scalable appeared in 57 job postings. All of the interviewees and over a quarter of job postings included references to internal tools for T&S work, including automation.
However, the effectiveness of automation may be limited because messy human problems are not monolithic, instead involving a need for context, nuance, and localized knowledge. This posting provides an example: “preserving TikTok as a safe, positive, and welcoming environment for our users, while also protecting our users’ freedom of creative expression to post content that may be serious or uncomfortable, is complex and challenging.”
Automation is based on the underlying assumption that either/or, binary technologies can make decisions about complex, multi-faceted, and nuanced human behaviors. When an algorithm is applied to a social media post, for example, it can look for similarities to content it has previously encountered. Depending on the sophistication of the algorithm, it may classify the content based on a set of acceptable or unacceptable words, or it may attempt to classify new materials based on previous examples that are not exact matches for the new content. In either case, the decision it renders is “acceptable” or “unacceptable,” or in some cases may be referred for human review (Gallo 2021; Greenfield 2017).
One interviewee elaborated on this limitation, saying that the tech industry was.
Nine postings described the need for T&S work to account for nuance and context or to iterate in response to emerging trends. Additionally, as one interviewee said, “People who misuse technology are always looking for workarounds. I think that’s a human behavior issue“ (Interviewee 4). This speaks to the need for constant iteration based on research, subject matter expertise, and collaboration. It also indicates that responses (and proactive strategies) may benefit from a partnership of humans and automation.
The need to take into account the context in which content is created or someone engages in certain behaviors evident as companies attempt to design community guidelines, use automation to recognize harmful activity, or respond to user reports. One interviewee argued that “to do Trust and Safety well you need context. And you look for context wherever you can find it across the company. And then that context feeds into the work that we do” (Interviewee 5). Another said, “Context, context, it’s all context. And that’s what an investigation is, is determining context” (Interviewee 2). What is context? One interviewee stated that context means “really thinking about all of the different ways people communicate, all the nuances and language and region” (Interviewee 1). Incorporating T&S and subject matter expertise in T&S, having more inclusion, more voices, more experiences, contexts, means that more harms can be anticipated and perhaps mitigated or prevented. As one interviewee said, “Having that diverse lived experience is vital” (Interviewee 6).
4.4.5 Collaboration
There is a strong trend towards greater collaboration. This includes internal collaboration in the form of cross-team work as well as the establishment of more interdisciplinary teams within companies. The trend is also external, showing up as cross-company and industry-wide collaboration, as well as work with partners such as NGOs, law enforcement, and others. Collaboration appeared 82 times in postings (sometimes multiple in one posting) both in reference to internal and external collaboration. All interviewees mentioned this trend, and three raised it in answer to a question specifically about the challenges of T&S work.
Interviewees spoke to a need for structures and resources within companies to support that collaborative work. Three said that structures to support collaboration both with other teams or by making teams more interdisciplinary were needed. Two spoke to the need for cross-company or industry-wide collaboration.
4.4.6 Prevention
More recently the spectrum of T&S work has expanded from response to a more proactive stance, and even as far as prevention through advising on the design of new products and features. In the job postings, responsive verbs including “protect,” “enforce,” “fight,” and “defend” were used 119 times in position descriptions and responsibilities. The arguably more proactive verbs “identify,” “mitigate,” “detect,” and “reduce” were used 189 times. More preventative verbs “design,” “promote,” “prevent” were used 91 times, though design could refer to various parts of this responsive-preventative spectrum (Figure 1).
Figure 1: Spectrum of T&S actions.
Figure 1:
Spectrum of T&S actions.
As evident in both interviews and job postings T&S may now participate earlier in product design and development processes. The role of T&S in this phase of technology design and development is to help minimize misuse and promote good use, put safeguards in place, and help product teams think through T&S issues and anticipate negative use case scenarios. Interviewees spoke to the value of this shift in addressing all the “bad things” that T&S counters, towards “an ideal future [with] Trust and Safety more baked in product” (Interviewee 3). All but one interviewee talked about this shift, with two including this in their answers to a question about challenges facing T&S, and two pointed to this as a need to meet T&S challenges. The words “prevent” and “proactive” appeared 21 times each in job postings. Advising or consulting on external product development was mentioned 41 times in job postings (Table 2).
Table 2:
Responsive, proactive, preventative verbs in 112 job postings.
ㅤ | Verb | About the Position | Responsibilities |
Responsive | Protect | 22 | 10 |
ㅤ | Enforce | 18 | 54 |
ㅤ | Fight | 7 | 0 |
ㅤ | Defend | 7 | 1 |
Proactive | Identify | 18 | 76 |
ㅤ | Mitigate | 9 | 13 |
ㅤ | Detect | 8 | 24 |
ㅤ | Proactive | 4 | 17 |
ㅤ | Reduce | 4 | 16 |
Preventative | Design | 12 | 41 |
ㅤ | Promote | 9 | 8 |
ㅤ | Prevent | 4 | 17 |
Interviewees also understood safety beyond simply mitigating harm. A consistent perspective among interviewees was that harms could be prevented through proactive measures. One interviewee went further to say, “Safety is not just the absence of harm. It’s also ‘Am I empowered? Do I feel engaged? Do I feel my voice matters? Do I feel valued?’” (Interviewee 6). One posting stated the goal this way: “make Flickr even safer and more welcoming to all.” T&S in this way is about proactively designing products to help users feel not just safe, but also empowered in online spaces or while using products.
5 Discussion
The T&S field is relatively nascent, and is in a pivotal point in development in which rapid growth and formalization is underway. This research shows that there is broad alignment between companies’ conceptualization of T&S work and that of professionals. However, I noted that the functions of T&S work and the bad things it addresses are not necessarily present in every T&S team in every company, and that this varies based on company size, maturity, and product. In short, T&S lacks a clear definition and scope that can be applied throughout the tech industry as a whole.
This research reveals six key trends through an analysis of the postings and the interview data: 1) external pressure, 2) prioritization, 3) formalization, 4) automation, 5) collaboration, and 6) prevention. These trends can also be seen as challenges and as future directions, and are interrelated to the point of being nearly impossible to examine separately. For example, external pressure elevates T&S as a priority in companies resulting in more hiring and also the development of automated tools. Additional hiring prior to the Fall of 2022 brought a wider range of experience and expertise context and nuance inside companies. However this increase in hiring was followed by successive waves of layoffs which impacted the nascent T&S field.
Interviewees and job postings are aligned in the kinds of mindsets needed to do T&S work: collaboration, communication and diplomacy, curiosity, adaptability, problem-solving, the ability to understand context and complexity, empathy and emotional resilience, and passion. These mindsets contribute to establishing, communicating, and acting on the context behind harmful content or behavior.
T&S work has been evolving and expanding for more than a decade. Earlier work centered on the development and iteration of content policy and moderation in online spaces. Content that has been flagged or reported by users is reviewed, and a decision is made whether or not to remove it. In particularly egregious cases, there may also be further investigation and escalation. It also now incorporates the use of automated tools for detection and filtering. This may be followed with some kind of remediation such as steps to help a user to feel safer or the removal of a user who has caused repeated harm. Holistic strategies are needed beyond simply responding to crises after they become apparent. Primary among these is a shift towards more proactive approaches to address the risks and harms of misuse of online and real world technologies.
6 Conclusions
This research assessed the degree of alignment between companies and professionals on the question of the purpose and nature of Trust and Safety (T&S) work. Trust and Safety within technology companies responds to, mitigates, and prevents some of the most troubling risks and harms that users experience on platforms and when using products. This research offers a snapshot of a field that is growing and formalizing under growing pressure on companies from society to mitigate bad things. The complexity and context inherent in the problems point to the need for professionals with diverse backgrounds and expertise to collaborate at all phases of technology development from design to responding to incidents. Collaboration is woven throughout, and is essential: from cross-team collaboration to interdisciplinary teams internally, to industry-wide efforts to address T&S challenges and define the field, to cross-sector partnerships to govern technology in the public interest.
Trust and Safety addresses the risks and harms that come with ubiquitous technology (TSPA n.d.). The success of T&S to address technology harms is of importance to everyone. We all have a stake, whether we are on a particular platform or use a certain product or not (Bridle 2019; Greenfield 2017). Within broader social conditions of ubiquitous and profoundly complex technological systems, policymakers, journalists, civil society, and the public generally are becoming aware of the ways in which the technologies of our daily lives accelerate and amplify messy human problems like harassment, exploitation, and polarization.
Many have argued that there is a role for external scrutiny, oversight, and governance (e.g., World Wide Web Foundation 2021). If we take as a given that the tech isn’t going away, that it is integrated into our individual and collective lives and has profound effects, then we need to determine as a society how we manage, repair, or prevent the bad things that come with it. We can exercise that through mechanisms such as regulation, legislation, litigation, journalistic investigations, as well as consumer choice, social media campaigns, and input from civil society (DeNardis and Hackl 2015; Ganesh and Bright 2020; Helberger et al. 2018; Riley and Morar 2021; Robinson 2020; Suzor et al. 2019). Even as these methods of external governance are brought to bear, there is a need for work within technology companies to respond to, mitigate, and prevent harms, and to implement compliance mechanisms as external governance comes online.
The Trust and Safety field is being built by professionals in the field as they articulate the value of T&S, and push for efforts to integrate T&S earlier in technology design and development. A shift toward proactive, even preventative approaches is essential. Such a shift in product design would entail the integration of threat models, scenarios, and subject matter expertise early in the design of new features and products to reduce the risks that these can be misused to cause harm online and in the real world. Care must be taken to ensure the interest of diverse publics is met by both internal and external governance of technology. Societies can continue to call for tech in the public interest, while understanding that solutions to the problems T&S addresses are not simple, and rather than simply asking for more moderators, proactive and preventative approaches are needed.
Corresponding author: Toby Shulruff, School for the Future of Innovation in Society, 7864 Arizona State University , Tempe, USA, E-mail: tshulruf@asu.edu
Acknowledgments
Thanks to the reviewers and editors of this article, and to the stakeholders and advisers including the Trust and Safety Foundation and the School for the Future of Innovation in Society at Arizona State University. Thanks especially to the anonymous interviewees who participated in this research.
Research ethics: The Office of Research Integrity and Assurance at Arizona State University approved the most recent modification of Study #00015156 on April 6, 2023.
Informed consent: Informed consent was obtained from all individuals included in this study, or their legal guardians or wards.
Author contributions: The author has accepted responsibility for the entire content of this manuscript and approved its submission.
Use of Large Language Models, AI and Machine Learning Tools: None declared.
Conflict of interest: The author states no conflict of interest.
Research funding: None declared.
Data availability: Some of the datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request, except where limited by protection of the identities of interviewees.
References
Amnesty International (2018). Toxic Twitter: a toxic place for women, Available at: https://www.amnesty.org/en/latest/research/2018/03/online-violence-against-women-chapter-1/ (Accessed 10 September 2021).Search in Google Scholar
Benjamin, R. (2019). Race after technology: abolitionist tools for the new Jim Code. Polity Press, New York.Search in Google Scholar
Borrás, S. and Edler, J. (2014). The governance of socio-technical systems. Edward Elgar Publishing, Cheltenham.10.4337/9781784710194Search in Google Scholar
Brand, T. and Blok, V. (2019). Responsible innovation in business: a critical reflection on deliberative engagement as a central governance mechanism. J. Res. Innovation 6: 4–24, https://doi.org/10.1080/23299460.2019.1575681.Search in Google Scholar
Bridle, J. (2019). New dark age: technology and the end of the future. Verso, Brooklyn, NY.10.17104/9783406741784Search in Google Scholar
Chatterjee, R., Doerfler, P., Orgad, H., Havron, S., Palmer, J., Freed, D., Levy, K., Dell, N., McCoy, D., and Ristenpart, T. (2018). The spyware used in intimate partner violence. In: IEEE Symposium on Security and Privacy, San Francisco, CA, pp. 441–458.10.1109/SP.2018.00061Search in Google Scholar
Costanza-Chock, S. (2020). Design justice: community-led practices to build the worlds we need. The MIT Press, Cambridge, MA.10.7551/mitpress/12255.001.0001Search in Google Scholar
DeNardis, L. and Hackl, A. (2015). Internet governance by social media platforms. Telecommun. Policy 39: 761–770, https://doi.org/10.1016/j.telpol.2015.04.003.Search in Google Scholar
Digital Trust & Safety Partnership (n.d.), Available at: https://dtspartnership.org/ (Accessed 23 June 2022).Search in Google Scholar
Doerfler, P. (2019). Something you have and someone you know: designing for interpersonal security. Usenix Enigma, Burlingame, CA, Available at: https://www.usenix.org/conference/enigma2019/presentation/doerfler.Search in Google Scholar
Dragiewicz, M., Burgess, J., Matamoros-Fernández, A., Salter, M., Suzor, N.P., Woodlock, D., and Harris, B. (2018). Technology facilitated coercive control: domestic violence and the competing roles of digital media platforms. Fem. Media Stud. 18: 609–625.10.1080/14680777.2018.1447341Search in Google Scholar
Escobar, A. (2017). Designs for the pluriverse: radical interdependence, autonomy, and the making of worlds. Duke University Press, Durham, NC.10.1215/9780822371816Search in Google Scholar
Freed, D., Palmer, J., Minchala, D., Levy, K., Ristenpart, T., and Dell, N. (2018). “A stalker’s paradise”: how intimate partner abusers exploit technology. In: Proceedings of the 2018 CHI conference on human factors in computing systems. Association for Computing Machinery, Montreal, pp. 1–13.10.1145/3173574.3174241Search in Google Scholar
Gallo, J. (2021). Social media: Misinformation and content moderation issues for Congress. Congressional Research Service, Washington, DC.Search in Google Scholar
Ganesh, B. and Bright, J. (2020). Countering extremists on social media: challenges for strategic communication and content moderation. Policy and Internet 12: 6–19, https://doi.org/10.1002/poInterviewee3.236.Search in Google Scholar
Genus, A. and Stirling, A. (2018). Collingridge and the dilemma of control: towards responsible and accountable innovation. Res. Policy 47: 61–69, https://doi.org/10.1016/j.respol.2017.09.012.Search in Google Scholar
Greenfield, A. (2017). Radical technologies: the design of everyday life. Verso, Brooklyn, NY.Search in Google Scholar
Helberger, N., Pierson, J., and Poell, T. (2018). Governing online platforms: from contested to cooperative responsibility. Inf. Soc. 34: 1–14, https://doi.org/10.1080/01972243.2017.1391913.Search in Google Scholar
Hobbis, S.K. (2018). Mobile phones, gender‐based violence, and distrust in state services: case studies from Solomon Islands and Papua New Guinea. Asia Pac. Viewpoint 59: 60–73, https://doi.org/10.1111/apv.12178.Search in Google Scholar
Leitão, R. (2019). Anticipating smart home security and privacy threats with survivors of intimate partner abuse. In: Proceedings of the 2019 on designing interactive systems conference. Association for Computing Machinery, San Diego, CA, pp. 527–539.10.1145/3322276.3322366Search in Google Scholar
Leitão, R. (2021). Technology-facilitated intimate partner abuse: a qualitative analysis of data from online domestic abuse forums. Hum.–Comput. Interact. 36: 203–242, https://doi.org/10.1080/07370024.2019.1685883.Search in Google Scholar
Lenhart, A., Ybarra, M., Zickuhr, K., and Price-Feeney, M. (2016). Online harassment, digital abuse, and cyberstalking in America. Report 11.21.16, Data & Society Research Institute.Search in Google Scholar
Levy, K. and Schneier, B. (2020). Privacy threats in intimate relationships. J. Cybersec. 6, https://doi.org/10.1093/cybsec/tyaa006.Search in Google Scholar
Malanga, D.F. (2021) Survey of cyber violence against women in Malawi. In: Proceedings of the 1st virtual conference on implications of information and digital technologies for development. The International Federation for Information Processing, Available at: https://arxiv.org/abs/2108.09806.Search in Google Scholar
McGuinness, T. and Slaughter, A. (2019). The new practice of public problem solving. Stanf. Soc. Innov. Rev., Spring 2019: 26–33.Search in Google Scholar
Messing, J., Bagwell-Gray, M., Brown, M.L., Kappas, A., and Durfee, A. (2020). Intersections of stalking and technology-based abuse: emerging definitions, conceptualization, and measurement. J. of Fam. Violence. 35: 693–704, https://doi.org/10.1007/s10896-019-00114-7.Search in Google Scholar
Nuttal, L., Evans, J., Franklin, M., and Burne James, S. (2019). Coercive control resistant design: a key to safer technology. IBM UK, London, Available at: https://www.ibm.com/blogs/policy/wp-content/uploads/2020/05/CoerciveControlResistantDesign.pdf.Search in Google Scholar
Riley, C. and Morar, D. (2021). Applying multi-stakeholder internet governance to online content management. R Street, Washington, DC, Available at: https://www.rstreet.org/wp-content/uploads/2021/09/Multi-stakeholder-Internet-Gov-Report-RSI-Sept.-2021.pdf.Search in Google Scholar
Robinson, D. (2020). Click here to sue everybody: cutting the Gordian Knot of the internet of things with class action litigation. Rich. J.L. & Tech 26: 1.Search in Google Scholar
Roundy, K.A., Mendelberg, P.B., Dell, N., McCoy, D., Nissani, D., Ristenpart, T., and Tamersoy, A. (2020). The many kinds of creepware used for interpersonal attacks. In: IEEE Symposium on Security and Privacy, Online, pp. 626–643.10.1109/SP40000.2020.00069Search in Google Scholar
Russell, L. (2020). Glitch feminism. Verso, Brooklyn, NY.Search in Google Scholar
Schneier, B. (2017). Security and the internet of things. Schneier on Sec., Available at: https://www.schneier.com/blog/archives/2017/02/security_and_th.html (Accessed 5 July 2021).Search in Google Scholar
Silva, M. and Oliveira, D. (2021). Brazilian favela women: how your standard solutions for technology abuse might actually harm them. In: SOUPS 5th workshop on inclusive privacy and security (WIPS 2020). USENIX, https://arxiv.org/abs/2101.09257.Search in Google Scholar
Slupska, J. and Tanczer, L.M. (2021). Threat modeling intimate partner violence: tech abuse as a cybersecurity challenge in the Internet of things. In: The Emerald international handbook of technology facilitated violence and abuse. Emerald Publishing Limited, Leeds, UK.10.1108/978-1-83982-848-520211049Search in Google Scholar
Suzor, N., Dragiewicz, M., Harris, B., Gillett, R., Burgess, J., and Van Geelen, T. (2019). Human rights by design: the responsibilities of social media platforms to address gender- based violence online. Policy and Internet 11: 84–103, https://doi.org/10.1002/poInterviewee3.185.Search in Google Scholar
Tanczer, L., Neira, I.L., Parkin, S., Patel, T., and Danezis, G. (2018). The rise of the Internet of Things and implications for technology-facilitated abuse. University College London, London.Search in Google Scholar
Trust and Safety Professional Association (n.d.). What we do, Available at: https://www.tspa.info/what-we-do/ (Accessed 25 October 2021).Search in Google Scholar
World Wide Web Foundation (2021). Open letter to the CEOs of Facebook, Google, TikTok and twitter, Available at: https://webfoundation.org/docs/2021/07/Generation-Equality-Open-Letter-to-tech-CEOs.pdf (Accessed 10 September 2021).Search in Google Scholar
Received: 2024-08-26
Accepted: 2024-09-22
Published Online: 2024-10-24
© 2024 the author(s), published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.