Note: The 7 Rules of Causation are part of my upcoming Root Cause Analysis guide, a resource in the Trust & Safety Toolkit.
Contents
Introduction
We’ve all been there: an incident has occurred. Something that should not have happened did happen, and now you’re faced with explaining how, exactly, that situation came to pass.
A retro, a post-mortem, a debriefing—whatever you call it, it’s got to be done.
Why Causation Matters in Trust & Safety
Robust and accurate evaluations of the root causes and contributing factors that led to an incident are essential for informed decision-making, preventing future incidents, and improving communication. They also play a key role in maintaining user trust by demonstrating accountability, transparency, and a commitment to safety.
When done right, a Root Cause Analysis strengthens the integrity and effectiveness of Trust & Safety (T&S) operations. As the name implies, an effective Root Cause Analysis identifies and documents the often complex and interconnected sequence of causes and effects that led to the incident.
Determining Causation: 7 Rules for Better Incident Reports
Adapted from David Marx’s 5 Rules of Causation for aviation safety incidents1, below are the 7 Rules of Causation for documenting the root causes and contributing factors of a T&S incident.
While aviation safety and Trust & Safety may seem quite different at a glance, they share common challenges: complex systems, human decision-making under pressure, and the need to prevent recurring incidents.
When followed, these 7 rules ensure that incident descriptions are accurate, precise, and unemotional. They reflect human factors engineering (HFE) principles because they require us to examine how people interact with systems and to determine how we can make these interactions more reliable, efficient, and error-resistant.
Collectively, the 7 Rules of Causation help us analyze incidents through the lens of human-system interaction rather than treating technical and human factors separately.
Rule 1. Causal statements must clearly show the "cause and effect" relationship
Rule 1. Causal statements must clearly show the “cause and effect” relationship
Ensure clear identification of the chain of events that led to the incident.
SCENARIO: A violative post remained visible on the platform for several hours.
- Incorrect: “The system failed.”
- Correct: “The violative post remained visible because the moderation algorithm was not updated, which caused the violative content to go undetected.”
Why It Matters:
- The rule forces clear thinking about the actual chain of events;
- The rule helps us avoid vague conclusions (e.g., “the system failed”) that don’t help prevent future incidents; and
- The rule enables teams to identify specific points of failure that can be addressed.
Rule 2. Negative descriptors are not used in causal statements
Rule 2. Negative descriptors are not used in causal statements
Replace subjective or vague terms with factual and specific language.
SCENARIO: User reports are not being resolved in a timely manner.
- Incorrect: “Poor user communication.”
- Correct: “User reports were not resolved promptly because the notification system did not send alerts to the moderation team on time.”
Why It Matters:
- Negative descriptors such as “poor,” “inadequate,” or “bad” are subjective and don’t describe what actually happened;
- The rule keeps the focus on specific, actionable issues rather than judgmental language; and
- The rule keeps the analysis professional and factual rather than emotional or blame-oriented.
Rule 3. Each human error must have a preceding cause
Rule 3. Each human error must have a preceding cause
Highlight the systemic or environmental factors that led to human mistakes rather than blaming individuals.
SCENARIO: A moderator incorrectly flags a harmless post as harmful.
- Incorrect: “The moderator made an error.”
- Correct: “The moderator incorrectly flagged the post because they were not properly trained on the new policy guidelines.”
Why It Matters:
- Humans rarely make mistakes for no reason;
- The rule forces us to identify systemic issues (such as training gaps or unclear procedures) that can be fixed; and
- The rule moves us beyond blaming individuals to understanding why they acted or handled the situation the way they did.
Rule 4. Each violation of procedure must have a preceding cause
Examine why a procedure was not followed, addressing unclear or impractical guidelines.
SCENARIO: A reviewer skips a step in the content review process.
- Incorrect: “Procedure was not followed.”
- Correct: “The reviewer skipped a step in the content review process because the guidelines were unclear and the training provided was incomplete.”
Why It Matters:
- People rarely deviate from a prescribed procedure without a reason;
- The rule helps identify if procedures are unclear, impractical, or poorly communicated; and
- The rule can reveal when “workarounds” have become normalized due to system issues.
Rule 5. Failure to act is only causal when there is a pre-existing duty to act
Assign responsibility only where there is a legitimate expectation for the person or group to act, avoiding unwarranted blame based on hindsight.
SCENARIO: Harmful content was not escalated for review.
- Incorrect: “The reviewer failed to escalate the content.”
- Correct: “The harmful content was not escalated because the reviewer was not aware of the escalation protocol even though they had a duty to monitor for and escalate such content.”
Why It Matters:
- We can’t accurately assess causation if we attribute duties that are not specifically assigned to a role;
- The rule helps us distinguish between actual responsibilities and hindsight bias;
- The rule prevents unfairly attributing blame when someone wasn’t responsible; and
- The rule helps identify gaps in role definitions and responsibilities.
Rule 6. Causal search must focus on information transfers, control actions, and human or system interfaces
Identify points where communication, controls, or system interfaces broke down.
SCENARIO: Harmful content is not removed promptly.
- Incorrect: “The content was not removed because of a system error.”
- Correct: “The harmful content was not removed because the user interface of the agent tool’s review queue did not highlight the violation to the moderator, leading to a delay in action.”
Why It Matters:
- The rule directs attention to specific points where interventions can be made;
- The rule helps identify breakdowns in communication or system design; and
- The rule focuses attention on actionable elements rather than abstract concepts.
Rule 7. Specific descriptors are necessary to identify what failed
Avoid vague descriptions like “system error” and provide detailed explanations.
SCENARIO: A violative post is not removed.
- Incorrect: “The system had a bug.”
- Correct: “The violative post was not removed because the content filtering system had a bug that misclassified the post, preventing it from being flagged.”
Why It Matters:
- Vague descriptions like “had a bug” don’t go into enough specific detail to understand why things went wrong;
- The rule forces us to identify precisely what, why, and how things went wrong; and
- The rule enables us to deploy targeted fixes rather than broad, ineffective solutions.
Applying the Rules: An Example
Let’s take an incident write-up and see how applying the 7 Rules of Causation makes a world of difference.
Initial Draft
Here’s the initial write-up:
The content moderation tool was misclassifying user posts, leading to a significant number of false positives for harmful content. Users reported being wrongly flagged for content violations, resulting in temporary account restrictions. The issue was first identified due to an influx of user complaints received on September 2, 2024, across multiple regions.
The moderation team immediately halted auto-classification and reverted to manual review. Users affected by the misclassification were notified, and account restrictions were lifted after verifying that no actual violations had occurred.
A cursory read shows us what has happened, but it has not explained why and how it happened.
Reviewing the Draft
Let’s review how well the write-up abides by the rules of causation.
In the annotated image below, we see right away how Rule 1 and Rule 7 are not fully implemented.
Annotated incident write-up
We also note that nowhere in this write-up was there a mention of system-level factors (Rule 6) that may have contributed to the incident.
Improving the Draft
To improve the write-up, we first apply Rule 7 (Specificity)
By examining the specifics, we can better understand the incident’s scope (number of users) and what was misclassified (new allow-listed terms).
On 28 August 2024, the Policy team published OPSUPDATE-037 to add a new set of slang/terms to the Allow List. This policy change is intended to reflect how different marginalized communities have reappropriated these terms.
The policy change was scheduled to take effect on 2 September 2024 at noon PT. Unexpectedly, the auto-classification bots continued to classify user posts that included the slang/terms as violative.
From noon PT on 2 September 2024 until the auto-classification bots were paused, a total of 94,673 posts were wrongly flagged for content violations, affecting 43,862 unique users and resulting in these user accounts being placed under temporary account restrictions.
The issue was first identified because Anjali Bahri (T&S Policy) noticed on 3 September that user complaints about their accounts being restricted due to these slang/terms continued despite the planned policy change.
Next, we apply Rule 1 (Cause and Effect)
We dig deeper into what happened and identify the surprising series of events that led to the misclassification:
The moderation tool’s algorithm was not updated to handle the new slang and context variations documented in OPSUPDATE-037 because the Heuristics team member who was auto-assigned the ticket based on the on-call rotation had started parental leave earlier than expected.
There is no established process to ensure that already-assigned tickets are reassigned when the on-call rotation is retroactively updated. As such, OPSUPDATE-037 was not bundled with other planned updates and did not ship as scheduled.
There is also no established process within the Policy team for confirming that planned policy updates shipped as scheduled.
Assessing the Result
Notice how the improved write-up now abides by the 7 Rules:
- [Rule 1] Causal statements show the cause of the misclassification (approved policy change was not implemented as expected);
- [Rule 2] There are no negative descriptors used;
- [Rule 3] The human error (not deploying the allowlist update) has a preceding cause (ticket was not reassigned despite early parental leave);
- [Rule 4] The violation of procedure (failure to deploy the allowlist changes) has a preceding cause;
- [Rule 5] The lack of a pre-existing duty to act (no established process within the Policy team to confirm deployment of new policies) is noted explicitly;
- [Rule 6] The search for the cause identified gaps in hand-offs between people and systems; and
- [Rule 7] Specific descriptors now indicate who, when, how many, how long, and why.
More importantly, the write-up now describes the root causes and contributing factors in sufficient detail so we can easily identify the specific actions needed to prevent a similar incident from occurring again.
Wrapping Up
The 7 Rules demystify the “art” of writing incident reports. Instead of fatalistically believing that “only a good writer can produce decent incident reports,” we now have a rigorous framework for analysis that moves beyond simple blame to an in-depth understanding of system issues.
While the rules are listed as separate statements, our example clearly shows how applying a handful of rules leads to improvements across all seven.
By applying the 7 Rules of Causation to any incident write-up, we can:
- Better understand what really happened during incidents;
- Identify concrete points for tactical interventions and improvements;
- Share what we’ve learned effectively across teams; and
- Build more robust systems and processes.
The next time you need to write an incident report, use these rules as your guide. They’ll help you move from “What went wrong” to “Here’s how we can make it better.” Ultimately, that’s what effective Trust & Safety work is all about.
Author’s note: Special thanks and everlasting gratitude to mdy, who’s helped me both with this and so much else over the years.
Resources
Downloadable Rules
[ Join this Google Group (free) to gain access. ]
Causify It! Custom GPT
Need a little help getting used to the 7 Rules of Causation? Try Causify It, an experimental Custom GPT powered by this blog post. See a sample chat. See also OpenAI’s Privacy Policy.
Related Template
References
- Marx, D. (1999). Maintenance error causation. Chapter 2. In Human Factors in Aviation Maintenance – Phase XI, Progress Report. Washington, D.C: FAA, Office of Aviation Medicine. ↩︎