Hello! As a reminder, the Trustible Newsletter is a bi-weekly deep dive analysis of 4-5 important topics surrounding AI Governance. Typically, we cover important (and often uncovered) technical, business, and policy developments in AI and what they mean for AI Governance practitioners.
In today’s edition (5-6 minute read):
- Understanding the Risks of AI Hallucinations in Medicine
- AI Governance Triggers: When to Act and Why It Matters
- Apple Sued for False Advertising Over Apple Intelligence
- Texas AI Bill Revisions Show New Roadmap for Red State Approach
1. Understanding the Risks of AI Hallucinations in Medicine
Hallucination—where an LLM generates incorrect or fabricated information—is a major concern for AI used in healthcare. Medical content can be highly specialized and constantly-changing, which can lead to plausible-sounding, but very dangerous, hallucinations (see examples in the figure above). A recent study on “medical hallucinations” illustrates that these hallucinations are more nuanced than in many other domains. The key challenges outlined in this study were:
- Data: General-purpose LLMs lack knowledge specific to the medical domains and obtaining additional fine-tuning data can be challenging, as physicians notes contain jargon and inconsistencies, medical knowledge is rapidly evolving, and datasets may have insufficient representation of rare conditions. Mitigations can involve creation of higher-quality datasets and potential use of synthetic data to represent rare and complex cases. Connecting models with external databases, using Retrieval-Augmented Generation, can significantly reduce hallucinations, but requires the external sources to be kept up-to-date and hinges on the system’s ability to retrieve the right source.
- Overconfidence. LLMs tend to exhibit overconfidence, generating outputs with high certainty even when the information is incorrect; when generalizing to unseen medical conditions, models may output an answer even when one can not be reasonably inferred from the training data. Mitigations include: Careful prompting, which allows the model to express uncertainty, the use of models that can produce a probability, and the use of additional models that can critique and detect errors in the primary output
- Reasoning. LLMs are not trained on diagnostic (and similar) tasks and may be poorly suited for this type of reasoning. While reasoning models have shown promising results for other high-level tasks, curating the right post-training data can be difficult. For example, in diagnostics, symptoms may be related to multiple conditions, and patients often experience multiple conditions, making it difficult for physicians to quickly annotate a “ground truth” dataset. For simpler models, breaking down a complex task into smaller steps (or using other Chain-of-Thought prompting techniques) can reduce the likelihood of hallucinations.
- Black-Box Outputs. LLMs produce a “black-box” output (i.e. one that can not be directly explained), making it difficult to check for accuracy and resulting in reduced efficiency for doctors using the system. When an AI system uses a RAG set-up, it can explicitly cite its sources, allowing users to more quickly check the veracity of the outputs.
Many of these challenges and mitigations apply across multiple domains, but medicine remains particularly difficult. To avoid hallucinating, the model will need to learn whether two technical texts are semantically equivalent (e.g. a medication can have multiple names but edema is a general term that can not be substituted for ‘leg swelling’), and be able to synthesize multiple complex sources (e.g. multi-modal radiology data, research texts and detailed patient notes including allergies and symptoms).
Our Take: While medical hallucinations are an on-going challenge, physicians are also affected by cognitive biases, especially in high-stress and time-constrained environments. AI Systems designed to reduce hallucinations can assist with physician bias by citing sources and providing multiple answers with confidence scores. In addition, challenges around data and medical reasoning highlight a need for domain-specific solutions on top of general purpose models.
2. AI Governance Triggers: When to Act and Why It Matters
As we’ve discussed in our blog, AI Governance exists across several layers, including the organizational, use case, and model layers. There are plenty of ‘tasks’ to do at each layer, such as educating new personnel on their AI governance responsibilities, conducting risk assessments on proposed AI use cases, or conducting model evaluations. However, when should these actions take place? What are the events that should ‘trigger’ different AI governance actions? What should an organization do when a new version of a model is trained? Or when a customer reports a potential incident?
Any AI governance function needs to outline what kinds of events should trigger different responses. Mapping governance triggers to the appropriate action can provide clarity to everyone internally on what they need to do, and can help organizations implement the right tools to detect these triggers. Our latest whitepaper offers a framework to help organizations get started on this problem. Our AI Governance triggers taxonomy including an analysis of what each trigger is, how often they’re likely to occur, which group is likely to trigger the event, and whether the event is likely to have a big impact on the outcomes of an AI application. In subsequent content, we’ll share our thoughts on what AI governance action may be most appropriate for each trigger for organizations trying to balance governance oversight, with speed and efficiency.
3. Apple Sued for False Advertising Over Apple Intelligence
Source: Fast Company
It turns out one of the greatest risks for companies deploying AI may not be existential things like ‘loss of AI control’, but rather something far more mundane and insidious: overstated marketing. Last week, Apple was hit with a class action lawsuit alleging that Apple Intelligence did not live up to the capabilities advertised on tv, social media and elsewhere, especially on older devices. Apple is not alone in facing issues related to false advertising about AI systems; the FTC, under both the Biden and Trump administrations, has repeatedly initiated enforcement actions against companies claiming their AI systems were more capable than they were. This included a settlement with DoNotPay over its AI system that was advertised as automatically contesting parking tickets, and an ongoing dispute with Evolv technologies over overstated claims about its physical security scanning AI. One core problem in both cases was that the companies did not have appropriate evidence to back up their claims about their AI system’s performance.
Many regulators who favor a lighter touch on AI regulation like to reinforce that existing laws still apply to AI systems, and that defrauding customers with false claims is already illegal. They’re right, but there’s a few characteristics of the current AI ecosystem that may make deceptive AI claims very common, or even unintentional. The first issue is that building a product integrated with the world’s most powerful AI systems is borderline trivial, and therefore many without subject matter or technical expertise can easily build a “world class” product, even without understanding its potential limitations or risks. In some instances, even those with deep expertise can perceive AI as being more capable than it actually is, as was the case in 2022 with a Google engineer who believed Google’s LLM was fully sentient (it was not). This problem is exacerbated by the relative immaturity of AI capability assessments, and related measurement science. It can legitimately be hard to understand the limitations of current AI systems, especially for novel use cases. Finally, there’s always the risk of things getting lost in translation between technical teams, and the marketing/business teams. This is particularly likely as we often use terminology that suggests more human-esque characteristics like how the model ‘thinks’ or ‘knows’, even though this isn’t really how many of these models work.
Key Takeaway: One of the biggest current AI risks is believing an AI system is more capable than it actually is, and then either overrelying on it, or deceiving consumers about its capabilities. Even before major AI specific regulations are in place, organizations should be careful about the claims they make about their AI system’s capabilities and have data to back up any claims.
Update: The title of this section was updated to reflect the fact that this is a commercial class action lawsuit, and not a lawsuit directly from the FTC.
4. Texas AI Bill Revisions Show New Roadmap for Red State Approach
Texas is once again making waves with AI regulations, as its state legislature considers a revised Texas Responsible AI Governance Act (TRAIGA). We previously discussed a potential “red state” approach to AI regulations when the original TRAIGA (HB 1709) was released in December 2024. The initial version of the bill was a combination of the American Legislative Exchange Council’s Model AI Bill and Colorado’s SB 205. HB 1709’s similarly to SB 205 made it difficult to argue that it could be a plausible red state alternative. In fact critics went as far as claiming that the original bill replicated the EU AI Act.
The revised bill (HB 149) sought to address those criticisms and provide legislation that is ideologically distinct from blue state frameworks. The revised legislation’s scope was narrowed dramatically to primarily regulate public sector AI use. Key obligations for government entities include:
- Disclosures. Agencies must disclose when consumers are interacting with AI systems, regardless of whether the consumer would reasonably believe that they were interacting with an AI system.
- Prohibited Uses. Agencies cannot use AI systems for social scoring or that use biometric identifiers. Social scoring systems are also prohibited under the EU AI Act, but that obligation is not limited to the public sector. The EU AI Act also prohibits certain biometric systems, whereas HB 149 limits the prohibition to systems that could violate a person’s civil liberties or rights.
Unlike its predecessor, HB 149 no longer requires private AI developers or deployers to implement risk management programs. Instead, it imposes limited obligations on these entities that prohibit developing or deploying AI systems that discriminate based on political viewpoint or a protected class. These prohibitions raise potential issues should the law pass, such as:
- Technical concerns with political view point discrimination. Depending on the platform, engaging with certain content may train the underlying algorithm to promote one type of political viewpoint to the detriment of another. Moreover, AI guardrails may become more regulated or scrutinized when attempting to filter out harmful or offensive content that may also be political in nature.
- Disparate impact limitations. The bill explicitly states that disparate impact alone is not sufficient to show intent to discriminate against a protected class. Under existing law, successful disparate impact lawsuits require more than simply showing that an unbiased practice is adversely impacting certain protected classes (i.e., racial or ethnic groups). This raises questions as to why include the language, since someone claiming an AI system was disparately impacting certain racial or ethnic groups would not be enough to prove discrimination. Moreover, disparate impact discrimination is not so much concerned with intent to discriminate as it is with actual discriminatory effect.
The revised bill also neuters the AI Council that would be created by the legislation. The previous version gave the AI Council some rulemaking authority, as well as the power to issue advisory opinions for AI use by state agencies. The revised bill strips those powers away, making clear that the AI Council serves as an advisory board that can issue reports and conduct training programs for state agencies.
Our Take: Enacting TRAIGA will further complicate AI regulations in the U.S., especially trying to co-exist with CO SB 205. Moreover, OpenAI just advocated for a federal bill to pre-empt state laws and avoid this regulatory patchwork. If federal Republicans take OpenAI’s advice to heart, this would set them up for a showdown with their state-level counterparts over whose law should reign supreme.
—
As always, we welcome your feedback on content and how to improve this newsletter!
AI Responsibly,
- Trustible team