Note: I presented a version of this post as the opening keynote (see keynote deck) at TrustCon’s inaugural conference in 2022. Have feedback on this post? @ me on Bluesky.
Introduction
Back in 2018, I started looking at other fields that dealt with challenges similar to Trust & Safety (T&S) – multidisciplinary, high velocity, high stakes, murky information environments, global in reach – to see if I could draw any inspiration for the issues we were struggling with.
I found myself increasingly intrigued by some of the overlaps I noticed with epidemiology (the study of how diseases and health-related conditions spread, occur, and are controlled within populations), and, in turn, with the field of preventative healthcare and with public health more broadly.
The more I learned, the more similarities I found, and I eventually came to the realization that at its core, Trust & Safety’s purpose is to work in service of health – of the company, the platform, the people who use it, the people whose lives are impacted by it, and (possibly idealistically) the world writ large.
Now, when I say Trust & Safety’s purpose is to work in service of health, I’m not just talking about defining what is bad and addressing the bad things. In healthcare, “health” isn’t just the absence of disease, disorder, illness, or injury. Rather, in its ideal state, health refers to complete physical, mental, and social well-being. Ultimately, we must work in service of that broader meaning of health.
As such, T&S professionals have five important and distinct responsibilities, each of which tracks closely to what is called an “intervention level” in preventative healthcare.
T&S Responsibilities and Healthcare Intervention Levels
The diagram below provides an overview of the five responsibilities that we must fulfill in service of Health.
Figure 1. T&S Responsibilities mapped to Healthcare Intervention Levels
As T&S professionals, we must:
- Prevent Risk Factors. We identify and target situations and conditions that are likely to create the factors that put people at risk of harm and work to stop the risk factors from ever developing. This work maps to the Primordial intervention level in healthcare.
- Reduce Risk. When risk factors already exist, we work to reduce or remove them, empower people to control their experiences, and increase protective factors to build people’s resilience in situations where they encounter risk. This work maps to the Primary intervention level in healthcare.
- Detect Harm and Intervene. We proactively identify harms and intervene to stop them when they occur. This work maps to the Secondary intervention level in healthcare.
- Mitigate Harm. We reduce the impact of experienced harms through direct action and provide remediations to avoid reoccurrences. This work maps to the Tertiary intervention level in healthcare.
- Do It the Right Way. Last but not least, we strive to fulfill these responsibilities the right way. As both a responsibility and a cross-cutting principle, we balance potential risks and harms against the benefits of a given intervention or approach. We make sure we don’t inadvertently do more harm than good. This work maps to the Quaternary intervention level in healthcare.
Mapping Healthcare Interventions to a Trust & Safety Scenario
Let’s consider a specific scenario – say, losing log-in access to your account – and examine the goals and actions we can take at each intervention level.
A. Primordial
Our first responsibility – preventing risk factors from developing – tracks closely to preventative healthcare’s primordial level. By identifying the underlying conditions that lead to risk factors or that prevent protective factors from developing, we can design and implement system-wide interventions to either inhibit or promote the development of a given factor.
Healthcare Example: In healthcare, increasing neighborhood walkability to encourage physical activity is an example of a primordial intervention. The broader population benefits because physical activity is more easily accessible and, as a result, more likely to become a consistent part of someone’s lifestyle.
T&S Example: In our account access scenario, we want an intervention that will cut down on the number of people who inadvertently lose account log-in access and can’t regain it without human assistance. At this level, we aren’t trying to identify specific groups or individuals in need; rather, we’re looking for broader conditions that could lead to risk factors developing that can be targeted across the system. For example, we might try requiring that people confirm their email addresses as part of the sign-up process to ensure that the email addresses they provide don’t have any inadvertent typos. Another option could be to display a user prompt informing them of the required correction when a known invalid email address is entered – say, an email address that has a comma instead of a period or no @ sign.
B. Primary
Now, realistically, there is no way that we can prevent every possible risk factor from developing. In preventative healthcare, primary prevention maps to our responsibility to reduce risk. By reducing or eliminating existing risk factors, we can lower the likelihood of people experiencing a given harm. If risk reduction is not possible, we instead must work to enhance protective factors and build capacity for and resilience to harm.
Healthcare Example: In preventative healthcare, a vaccine that lowers your likelihood of getting a specific disease is an example of risk reduction, while a law mandating the use of seatbelts in cars is an example of an intervention that enhances protective factors.
T&S Example: In our account access scenario, we want to reduce the number of people who inadvertently lose log-in access to their accounts and can’t regain it without human assistance. However, we’re no longer trying to address underlying conditions; rather, we’re attempting to either reduce risk or enhance protective factors. Let’s say, for example, that we’ve seen a significant increase in account compromises lately. We might consider requiring people to confirm they’re still in control of their account via an in-app or email prompt whenever we see an account login attempt from a new device. This intervention provides an additional layer of protection against account compromise.
C. Secondary
Much as we can’t prevent every possible risk factor from developing, there’s also no feasible way to eliminate all existing risk factors or prevent all harm. As such, we also need to detect harm and intervene – actions that fall within the secondary-level interventions in preventative healthcare. We want to take action at the earliest possible point where doing so can be effective, ideally before the full impact of the harm is experienced.
Healthcare Example: In preventative healthcare, these interventions involve screening for and identifying unrecognized diseases in healthy-appearing people and treating them before the onset of symptoms or at the earliest possible point where detection and intervention can be effective, as well as reversing the communicability of infectious disease. These interventions are not implemented for the whole population; rather, the focus is on specific individuals or groups who are likely at risk. Regular mammography screenings to detect early-stage breast cancer are an example of a secondary-level intervention.
T&S Example: In our account access scenario, the regular monitoring of credential dumps resulting from various data breaches and cross-referencing those credentials with our records to identify accounts that have direct password exposure will allow us to implement forced password resets before an account is compromised. This intervention identifies and halts the progression of harm for at-risk accounts before the full impact of the harm is experienced.
D. Tertiary
With the previous level, we focused on halting the progression of harm. To effectivelymitigate harm – known as tertiary interventions in healthcare – we must not only reduce the damage caused by the harm, with the goal of restoring the account to a healthy state, but also devise ways to prevent the harm from reoccurring or causing further damage.
Healthcare Example: In healthcare, tertiary interventions take place once a disease has entered the clinical stage and symptoms are present. These interventions focus on reducing the damage caused by symptomatic disease, including mitigating current pain or damage, preventing additional pain or damage, and restoring the health and functions of individuals affected by disease. Examples of tertiary interventions in healthcare include stroke rehabilitation programs and outpatient support programs.
T&S Example: In our Account Access scenario, a tertiary intervention could be to have a guided account restoration process in place so that people can easily regain access to their accounts. Similarly, having multi-factor authentication (MFA) reset tools so that people can reset their MFA settings after losing access to their authentication device (e.g., a lost phone or a broken hardware token) would also be a tertiary intervention.
E. Quaternary
Finally, as both a responsibility and a cross-cutting principle, we want to ensure that any actions we take and interventions we deploy are done the right way – preventative healthcare’s quaternary level. We need to weigh benefits against risks and potential ramifications before an action is taken. If the risks and potential ramifications outweigh the benefits, we need to figure out a different approach.
Healthcare Example: In healthcare, these interventions include actions taken to identify and protect individuals from medical interventions that are more likely to cause harm than good. These steps include identifying patients at risk of overtesting or overmedicalization, protecting them from excessive medical invasion, preventing the use of therapies that have not been adequately assessed, and suggesting interventions that are ethically acceptable.
Ultimately, such initiatives are informed by the principle of nonmaleficence – the obligation to not inflict harm on others – which is closely tied to the maxim of “first do no harm.” An example of a quaternary intervention in healthcare is avoiding excessive antibiotic use because overuse could lead to the development of drug-resistant diseases.
T&S Example: How might this play out for our account access scenario? Let’s say we’re planning to force password resets for all accounts that have had their passwords exposed through data breaches elsewhere. As we’re evaluating potential risks and ramifications, we realize that a significant number of affected accounts don’t have confirmed email addresses. If there is no way for people to regain access to their account without access to their email, we could consider either implementing a special account access flow for those accounts or limiting our forced password reset to those that have confirmed email addresses, then falling back on in-app notifications for other accounts to let them know that they need to change their password.
By providing effective and reliable mechanisms for people to regain access to their accounts if and when they need help – particularly if they need that help because of an intervention we’ve initiated – we give ourselves more freedom to implement other types of interventions because there is still a path back to health for those affected.
When we keep the five intervention levels in mind as we evaluate how to tackle a given T&S challenge, we can identify optimal intervention points and, when possible, shift our focus to earlier levels to maximize their impact.
Using Intervention Objectives to Identify Gaps in T&S Work
In the preceding section, I’ve focused primarily on Trust & Safety interventions that aim to either preserve health (i.e., health protection) or prevent, slow, or repair harm (i.e., harm reduction).
To use a video game metaphor, I’ve described actions that are designed to prevent our character from taking damage (shield), slow the rate of damage experienced (armor), or heal them (potion).
Figure 2. Health protection and harm reduction interventions aim to keep the Life Meter full
For Trust & Safety work to be successful and sustainable, however, our efforts must also include interventions that promote health and build resiliency – or, to go back to the video game analogy, interventions that will make our character’s life meter longer and give us more health capacity (i.e., health promotion).
Figure 3. Health promotion interventions extend the Life Meter
Why does this matter?
Health promotion interventions empower people to engage in responsible behaviors, which enhances harm prevention and strengthens protection measures. Because health promotion encourages individuals to adopt healthy behaviors before issues arise, these types of interventions increase resilience, further strengthening harm prevention and health protection interventions.
By developing interventions that span all three areas, we can break the cycle of being trapped in the endless whack-a-mole of reacting to harm after it has already occurred and instead begin to shift towards a more holistic pursuit of health.
Consider the Account Access examples we covered in the previous section. The table below maps the aforementioned interventions (denoted with a green checkbox) to the three intervention objectives and five intervention levels. A quick glance at the table below shows that our previously identified interventions all fall squarely under the health protection and harm prevention objectives only. It’s clear that we have gaps in health promotion and have an opportunity to expand our work on the first two objectives.
Table 1. Partial list of Account Access Interventions mapped to the three Objectives and the five Intervention Levels
What might a comprehensive set of interventions look like? Let’s fill out the rest of the table and consider how other interventions might work in service of preserving Account Access for our rightful users.
Table 2. Comprehensive list of Account Access Interventions mapped to the three Objectives and the five Intervention Levels
Conclusion
As Trust & Safety professionals, our focus cannot solely be on mitigating harm. Instead, our interventions must be in service of all three objectives:
- Health protection – eliminate risk factors and reduce the number of potential risks people encounter.
- Harm prevention – maintain current health capacity, mitigate the impact of harm, and repair damage.
- Health promotion – empower people to take control of their own experiences, build resilience, and increase their capacity to handle encountered risks.
Figure 4. By pursuing all three intervention objectives, we create a Virtuous Cycle
When our work spans the three intervention objectives and is across all five intervention levels, we create a virtuous cycle that allows us to identify optimal intervention points, discover gaps in our T&S work, and better support the development of a balanced and sustainable path in our pursuit of health.
Author’s note: Special thanks and everlasting gratitude to mdy, who’s helped me both with this and so much else over the years.
References
Association of Faculties of Medicine of Canada. “Introduction to Epidemiology.” Public Health Primer. Accessed August 9, 2024. https://phprimer.afmc.ca/en/part-i/chapter-4/.
Mendes, René, and Elizabeth Costa Dias. “Health Protection, Health Promotion, and Disease Prevention at the Workplace.” In Global Occupational Health, edited by Tee L. Guidotti. Oxford University Press, 2011. https://doi.org/10.1093/acprof:oso/9780195380002.003.0018. Accessed August 9, 2024.