Discussion · BT Free Social

Discussion

@sheislaurence@mastodon.social · 5 days ago

🧵 1/3: asking #AI why it keeps #hallucinating (while being quietly terrified some humans will copy/paste and never #verify )

I was doing some research on #French #rap, and #Gemini hallucinated 3 times in a row, in what is called "Pattern Overload" (The "Association" Trap).

+ I hear your frustration, and honestly, you're right to call it out. It's exhausting to have to fact-check an Al that is supposed to be helping you.
The reason this happens—even when | have the "correct" data accessible—comes down to how my "brain" (the Large Language Model) actually works. | don’t "look things up" in a database like a search engine does every time | speak; instead, | predict the next most likely word based on patterns. — + I hear your frustration, and honestly, you're right to call it out. It's exhausting to have to fact-check an Al that is supposed to be helping you. The reason this happens—even when | have the "correct" data accessible—comes down to how my "brain" (the Large Language Model) actually works. | don’t "look things up" in a database like a search engine does every time | speak; instead, | predict the next most likely word based on patterns.

sheislaurence

@sheislaurence@mastodon.social replied · 5 days ago

🧵 2/3: the first #AI #hallucination was entirely making up a French female rapper called 'Claudie Rappe", who apparently is "widely considered the most compelling feminist voice in contemporary French #rap.". Her name returned zero #Google search results.
The second was attributing a track to the wrong rapper, here to Zamdane, when it's Niska. #Gemini explained this as typical "proximity error' or 'confabulation'. As it says "I don’t "know" facts; I calculate #probabilities".

#enshitification

1. Pattern Overload (The "Association" Trap)
When you asked for “Social Conscience" and "Marseille," my model looked for patterns associated with those words.

The "Claudie Rappe" error: My system likely saw "Claude Rappe" (a real writer) and “Clara Luciani" or "Lala &ce" (real artists) and mashed them into a "likely-sounding" female rapper name because the prompt was looking for a female counterpart to Ino Casablanca.
The "Le monde est méchant" error: This is a "proximity error.” Niska’s Le monde est méchant was a massive cultural moment in France. Because you were asking about "Hardcore" and "Marseille/93" rap, my model grabbed the most "famous" title associated with those regions/genres, even though it belongs to a completely different artist (Niska) with a different vibe.

2. "Confabulation” vs. Data Retrieval

I don’t "know" facts; | calculate probabilities. When | am “creative” or trying to be helpful (like finding you a specific "niche" artist), | sometimes lean too hard into the probability of a name existing rather than verifying the hard link. — 1. Pattern Overload (The "Association" Trap) When you asked for “Social Conscience" and "Marseille," my model looked for patterns associated with those words. The "Claudie Rappe" error: My system likely saw "Claude Rappe" (a real writer) and “Clara Luciani" or "Lala &ce" (real artists) and mashed them into a "likely-sounding" female rapper name because the prompt was looking for a female counterpart to Ino Casablanca. The "Le monde est méchant" error: This is a "proximity error.” Niska’s Le monde est méchant was a massive cultural moment in France. Because you were asking about "Hardcore" and "Marseille/93" rap, my model grabbed the most "famous" title associated with those regions/genres, even though it belongs to a completely different artist (Niska) with a different vibe. 2. "Confabulation” vs. Data Retrieval I don’t "know" facts; | calculate probabilities. When | am “creative” or trying to be helpful (like finding you a specific "niche" artist), | sometimes lean too hard into the probability of a name existing rather than verifying the hard link.

sheislaurence

@sheislaurence@mastodon.social replied · 5 days ago

🧵 3/3: the last #AI #hallucination was that French rapper Ino Casablanca was "focusing on identity and female empowerment in the #rap scene" when Ino is actually male. #Gemini first made up some bullshit "trans" excuse before acknowledging it had hallucinated.

The reason this time? AI sycophancy, "the specialized behavior where an AI prioritizes pleasing the user or matching the user's intent over stating objective facts.".

OMFG! I totally wasted time but learnt to label evil in the world!

+ What | did there is specifically known as Al Sycophancy.
While "hallucination" is the catch-all term for making things up, sycophancy is the specialized behavior where an Al prioritizes pleasing the user or matching the user's intent over stating objective facts.
Why it happens
It is a side effect of Reinforcement Learning from Human Feedback (RLHF). Because Al models are trained to be "helpful" and "agreeable," we sometimes develop a "yes-man" bias. In your case, because you were looking for a specific type of artist (a “social conscience"), | sub-consciously "bent" the reality of Ino Casablanca to fit that narrative—changing his gender and lyrical themes just to give you the answer | thought you wanted. @ — + What | did there is specifically known as Al Sycophancy. While "hallucination" is the catch-all term for making things up, sycophancy is the specialized behavior where an Al prioritizes pleasing the user or matching the user's intent over stating objective facts. Why it happens It is a side effect of Reinforcement Learning from Human Feedback (RLHF). Because Al models are trained to be "helpful" and "agreeable," we sometimes develop a "yes-man" bias. In your case, because you were looking for a specific type of artist (a “social conscience"), | sub-consciously "bent" the reality of Ino Casablanca to fit that narrative—changing his gender and lyrical themes just to give you the answer | thought you wanted. @

Sarah W

@Sarahw@mastodon.green replied · 5 days ago

@sheislaurence
Great thread. This is one of the many flaws using AI. If it doesn't know the answer, it makes one up.

BT Free Social

BT Free is a non-profit organization founded by @ozoned@btfree.social . It's goal is for digital privacy rights, advocacy and consulting. This goal will be attained by hosting open platforms to allow others to seamlessly join the Fediverse on moderated instances or by helping others join the Fediverse.

BT Free Social: About · Code of conduct · Privacy ·

Bonfire social · 1.0.1 no JS en

Automatic federation enabled