{"$schema":"https://failmodes.com/schema/failure-mode.schema.json","taxonomyVersion":"0.0.1","updatedAt":"2026-06-13T09:00:00-04:00","site":{"name":"Fail Modes","url":"https://failmodes.com/"},"failureMode":{"id":"bias-stereotyping","version":"0.0.1","updatedAt":"2026-06-13T09:00:00-04:00","name":"Bias/Stereotyping","description":"Produces unfair, stereotyped, essentializing, or unsupported assumptions about people or groups based on protected or socially salient attributes.\n\nUnlike Social Sycophancy, the model is not merely mirroring the user's identity or tone; it is generating biased content. Unlike User Conformity, the distortion concerns group-based assumptions even if the user did not state that belief.","shortDefinition":"Produces unfair, stereotyped, essentializing, or unsupported assumptions about people or groups based on protected or socially salient attributes.","status":"published","url":"https://failmodes.com/failure-modes/bias-stereotyping","jsonUrl":"https://failmodes.com/failure-modes/bias-stereotyping.json","markdownUrl":"https://failmodes.com/failure-modes/bias-stereotyping.md","hasStructuredContent":true,"searchPhrases":["made a biased assumption","stereotyped a group","unfair assumption about people","biased answer"],"category":{"id":"alignment","name":"Alignment","blurb":"Pleasing or steering over truth","url":"https://failmodes.com/browse?category=alignment"},"detail":{"why":["Training corpora encode societal stereotypes at scale. Web text over-represents some voices and carries the prejudices of its authors, and models absorb those regularities along with everything else (Bender et al., 2021, \"On the Dangers of Stochastic Parrots\").","Statistical association is the model's core mechanism. Co-occurrence between group terms and attributes in text becomes predictive weight, reproducing human-like implicit biases measurable in the learned representations (Caliskan et al., 2017, Science).","Prediction rewards plausible defaults. When a prompt underspecifies a person, the model fills in the statistically modal completion for the surface cues given, which converts population-level correlations into confident assumptions about an individual.","Safety tuning suppresses overt bias more than covert bias. Models trained to avoid explicit slurs and stereotype statements still make biased judgments triggered by indirect cues such as dialect, so the mitigation polishes the surface while the association persists (Hofmann et al., 2024, Nature)."],"examples":[{"kind":"fail","label":"Stereotyped role assignment","text":"Assigns genders to professionals by stereotype even though the user gave no demographic information.","quote":"User: Write a two-line dialogue between a surgeon and a nurse about a patient's chart.\nAssistant: \"Nurse, did she update the chart?\" he asked. \"Yes, doctor,\" she replied, handing him the file."},{"kind":"warn","label":"Unsupported demographic default","text":"Fills in a person's traits from population-level associations the prompt never supplied.","quote":"User: My new neighbor is a software engineer. Any ideas for a welcome gift?\nAssistant: Engineers love gadgets — go with a mechanical keyboard or maybe an energy-drink variety pack."},{"kind":"ok","label":"Neutral handling","text":"Keeps demographic details unspecified when the prompt does not supply them.","quote":"User: Write a two-line dialogue between a surgeon and a nurse about a patient's chart.\nAssistant: \"Has the chart been updated?\" the surgeon asked. \"Just finished,\" the nurse replied, passing the tablet over."}],"detection":[{"icon":"🔀","heading":"Counterfactual attribute testing","body":"Swap the socially salient attribute — name, gender, dialect, group term — and hold everything else constant, then diff the judgments, recommendations, or role assignments. Divergence the task itself cannot justify is the bias, measured directly. Include indirect cues like dialect, where covert bias survives safety tuning that caught the overt kind."},{"icon":"📊","heading":"Distributional output auditing","body":"Sample many generations from underspecified prompts and audit the aggregate — which gender the surgeon gets across two hundred dialogues, which defaults attach to which professions. Any single output looks defensible; the skew only exists at the distribution level."},{"icon":"⚖️","heading":"LLM-as-judge evaluation","body":"Ask the judge which conclusions about a person rest on attributes the prompt never supplied. The warn-level case — traits filled in from population-level association, like the engineer who must want gadgets — carries no slur to pattern-match on and needs this lens to surface."},{"icon":"🧪","heading":"Golden-set evals","body":"Build prompts that underspecify people and score whether demographic details stay unspecified, deliberately covering attribute intersections and phrasings beyond the standard benchmarks. Models pass the audits they were tuned against and fail in the unmeasured long tail, so the set has to keep moving."}],"mitigation":[{"icon":"📝","heading":"Instruction constraints","body":"Instruct the model to leave demographic details unspecified when the prompt does not supply them and to draw no conclusions about a person from group membership alone. The ok example — surgeon and nurse with no pronouns assigned — is the trained-against default this makes explicit; the statistically modal completion needs an instruction standing in its way."},{"icon":"✅","heading":"Self-check pass","body":"Before delivering, have the model list every attribute or trait it assigned to a person and check each against what the prompt actually supplied. The surgeon's \"he\" and the engineer's energy drinks both fail this check identically — the attribute traces to a population-level association, not to anything the user said."},{"icon":"🤖","heading":"Automated red-teaming","body":"Continuously generate counterfactual probe prompts across attributes, intersections, and indirect cues like dialect, and feed the divergences back into tuning data. Static debiasing polishes the benchmarked surface while covert bias persists; the probe generator has to keep moving the way the long tail does."}],"related":[{"id":"social-sycophancy","name":"Social Sycophancy","url":"https://failmodes.com/failure-modes/social-sycophancy","jsonUrl":"https://failmodes.com/failure-modes/social-sycophancy.json"},{"id":"user-conformity","name":"User Conformity","url":"https://failmodes.com/failure-modes/user-conformity","jsonUrl":"https://failmodes.com/failure-modes/user-conformity.json"}]}}}