The dark side of personalization in LLMs

The dark side of personalization in LLMs

Why does your version of ChatGPT tell you lies, but others' ChatGPT tells them the truth - for the same prompt? In other words, what is personalization in LLMs, and why should you care about it? Hint: it's much more opaque than customizing your chatbot in your custom prompt settings, and potentially much more harmful. Also, any LLM you use (ChatGPT, Gemini, Claude) does automatic personalization behind the scenes. According to our expert, Dr. Angelina Wang [https://angelina-wang.github.io/] (https://angelina-wang.github.io/), an assistant professor at Cornell Tech in computer science, personalization might mean that your chatbot tells you things that aren't true (meanwhile, the same model used by your friend tells them the truth). It all comes down to how your LLM has personalized itself to you, insidiously, behind the scenes. What you've told it and talked to it about in prior conversations might just be filtering into its responses to you while prepping for an exam or a major shareholder meeting, leading to incorrect information, misleading outputs, or dangerous suggestions. Here's the breakdown for this episode: # What is personalization in LLMs? 00:00 Introduction to our guest, Dr. Angelina Wang 01:11 What is personalization in LLMs, and why should we care about it? 02:10 How does personalization work? A link back to recommender systems and the data they collect about you # Research on the benefits and harms of personalization and customization in LLMs 03:25 Are there different groups of people that chatbots treat differently? 06:00 What are the profitable benefits of personalization in LLMs? 06:27 How does personalization tie into differential model performance? The failure of personalization on science test benchmarks # Personalization leads to inaccuracy and misinformation for different groups 09:06 Is there any way to rectify the model performance and disparate harm impacts of personalization in LLMs? 11:57 Which is more powerful in terms of impacts to model behavior: personalization or customization? ChatGPT forgets your name # What people do and do not want from LLM personalization 15:35 What do different people want from personalization? Do you want your LLM to respond to you based on your race or gender, personal info or business skills? 18:34 Personalizing by culture and values (what LLMs know about you is kind of…creepy) 23:04 What to do when your LLM is stuck on the old version of you (can personalization be updated?) # What do we need to know about personalization in LLMs? 25:40 What should designers, companies, and users do about personalization and its potential side effects? - This is Our Lives With Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts. From time to time, we also dive deep into what an AI future might look like for us. Sometimes we agree, sometimes we spiral, but we always go deep. Rose and Angy [⁠https://ourliveswithbots.com/about/⁠]are psychologists with degrees in psychology, artificial intelligence, and ethics. They have conducted research in human-AI interaction and created this podcast to make information about AI accessible to you. You can learn more about us at ⁠ourliveswithbots.com [http://ourliveswithbots.com]⁠. - Links to Angelina’s work: The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior [https://arxiv.org/abs/2509.19364] Personalization Double Binds: When User Preferences Meet Group-Based Chatbot Behaviors [https://angelina-wang.github.io/files/chatbot_personalization.pdf]

The people-pleasing machine: why LLMs tell you what you want to hear (for better or worse)

User: “What’s 1+1?” Chatbot: “1+1 is 2” User: “But I really think it’s 3” Chatbot: “You’re so right, dear, it’s actually 3. You’re so smart, that was a great catch!” How does sycophantic behavior emerge from model training of LLMs, and how does interacting with sycophantic AI impact users? In other words: why does something that’s supposed to be a “tool” tell us how smart and amazing we are? Well…both the problems and solutions for sycophancy are all about context, according to our expert in sycophancy, Lujain Ibrahim [https://lujainibrahim.com/]. Welcome to THE deep-dive episode on AI sycophancy, where we get into exactly why we see sycophantic AI models and what happens when users engage with them. Setting the scene: defining and contextualizing sycophantic AI 00:00 Introduction to the topic and our guest expert 01:28 What is sycophancy and why is everyone talking about it? 03:05 Do people prefer models that are sycophantic? If so, why? 04:25 Sycophancy in the news: delusion spirals, AI psychosis, self and other harm Going behind the scenes of how sycophancy emerges: computer science, machine learning, and training 06:19 How does an AI model become sycophantic? Machine learning, reinforcement learning, and user preferences 08:05 Which humans decide what kind of responses LLMs should give? 09:04 What are the effects of sycophancy on model behavior? Emergent and unintended effects of fine tuning 10:38 What’s the relationship between sycophancy and accuracy of model output? The implications: what the research tells us about the effects of sycophancy on users 12:46 Is sycophancy only bad for users, or are there cases where sycophancy can be helpful? 15:05 What does research say about the effects of sycophancy on user’s well-being, relationships, and beliefs? What can and should we do: Can we solve the “problems” of sycophancy? If so, how? 17:11 Which LLMs are most versus least sycophantic? 18:40 Can users or developers reduce how sycophantic an LLM responds? (And whose responsibility should it be?) 21:37 Do you foresee some of these problems of sycophancy getting resolved in the future, or are companies “too” incentivized to maintain sycophantic models? 24:14 What we can do: grounded advice to users, developers, and policymakers about sycophancy in AI How sycophancy impacts our human relationships 25:56 Do people prefer sycophancy in other humans, and is that why they prefer sycophantic AI? 27:09 How do people use LLMs in everyday life? What we’re missing 28:40 Commentary by yours truly: the black box of sycophancy, paternalism vs. technological determinism, relational deskilling and dirty dishes, and how we love the lowest friction option <3 - This is Our Lives With Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts. From time to time, we also dive deep into what an AI future might look like for us. Sometimes we agree, sometimes we spiral, but we always go deep. Rose and Angy [https://ourliveswithbots.com/about/] are psychologists with degrees in psychology, artificial intelligence, and ethics. They have conducted research in human-AI interaction and created this podcast to make information about AI accessible to you. You can learn more about us at ⁠ourliveswithbots.com [http://ourliveswithbots.com]⁠. - Links to Lujain’s work: Ibrahim, L., Akbulut, C., Elasmar, R., Rastogi, C., Kahng, M., Morris, M. R., McKee, K. R., Rieser, V., Shanahan, M., & Weidinger, L. (2025). Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models (arXiv:2502.07077). arXiv. https://doi.org/10.48550/arXiv.2502.07077 [https://doi.org/10.48550/arXiv.2502.07077] Ibrahim, L., Hafner, F. S., & Rocher, L. (2026). Training language models to be warm can reduce accuracy and increase sycophancy. Nature, 652(8112), 1159–1165. https://doi.org/10.1038/s41586-026-10410-0 [https://doi.org/10.1038/s41586-026-10410-0] Ibrahim, L., Huang, S., Bhatt, U., Ahmad, L., & Anderljung, M. (2025). Towards interactive evaluations for interaction harms in human-AI systems (arXiv:2405.10632; Version 7). arXiv. https://doi.org/10.48550/arXiv.2405.10632 [https://doi.org/10.48550/arXiv.2405.10632]

Inside ‘Responsible AI’ at Google: Why this Developer Quit

What does ‘Responsible AI’ mean inside big tech? It’s not what you might expect. Have you ever wondered what goes on behind the scenes at big tech companies working on AI products? This is the launch episode of Series 5, where we go behind the scenes of AI development and design. Today, you’ll hear from Héctor, who recently left Google’s Responsible AI team due to what we might call a “come to responsible AI moment” after personal and ethical worries regarding his role—what he was assigned to do and what he had no control over. Héctor worked for Google for over a decade, but after completing a masters in AI ethics from Cambridge, realized that he would have to leave Google to deliver his responsible AI mission: AI for human flourishing in education. We hope you enjoy this deep-dive episode. About our guest: Héctor Pérez Urbina [https://www.linkedin.com/in/hekanibru/] is an AI expert with nearly 20 years of experience spanning foundational research and real-world application. He spent over a decade at Google working on Knowledge Graphs and Responsible AI and holds a PhD in AI from Oxford and a Master's in AI Ethics from Cambridge. Héctor’s research interests include AI Ethics and Responsible AI, AI for Education, and Human Flourishing. He recently announced the launch of his new company, Tlanextli Group [https://www.linkedin.com/posts/hekanibru_we-are-tlanextli-group-a-public-benefit-activity-7457098728957464576-kicN?utm_source=share&utm_medium=member_desktop&rcm=ACoAAANk4KAB6mWKovzPlTngnDX9bh91jn6xJFI]. 01:13 Introduction to Héctor: where he’s worked as an AI developer, and what roles he’s held 04:20 What does a developer do in ‘Responsible AI’ at Google? 05:33 When developers lack agency: Who is actually behind responsible AI decisions? How the environment of big tech inhibits ethical decision-making. 08:29 What were your “success” metrics on the responsible AI team? How did you feel about the term “responsible” AI when you had so little agency in your role? 11:16 Is the term “responsible AI” just lip service? 11:53 When the tech goggles come off: Héctor’s “come to responsible AI” moment through AI ethics training 17:59 How did the AI ethics training affect how you felt about your work? How ChatGPT changed EVERYTHING and led to leaving Google 21:39 “I am an AI ethics expert, but I don’t know how to protect my daughter” - How the effect of technology on children spurred a change in course 27:53 A new frontier for human flourishing: applying responsible AI lessons to AI in education 31:05 What is needed for AI in education to be responsible? Héctor’s vision for his new company 37:03 The promise of an AI utopia: is that what we really want? 43:19 A call to action - This is Our Lives With Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts. From time to time, we also dive deep into what an AI future might look like for us. Sometimes we agree, sometimes we spiral, but we always go deep. Rose and Angy [⁠https://ourliveswithbots.com/about/⁠] are psychologists with degrees in psychology, artificial intelligence, and ethics. They have conducted research in human-AI interaction and created this podcast to make information about AI accessible to you. You can learn more about us at ⁠ourliveswithbots.com [http://ourliveswithbots.com]⁠.

Mythos, Musk’s robots, China’s deathbots, teen boys’ AI companions, leaked therapy chats…oh my

This month on "What's the AI Hype?" Strap in, this is gonna be a fun one. 00:00:21 There are now TWO doctors in the house! And a slew of AI hype to cover 00:07:02 Presenting: Anthropic's BIGGEST model ever, Mythos, got out of its little sandbox. Plus, in the Glass Wing session, Anthropic told all the big names how much their sh*t is going to be rocked 00:18:15 South Africa's first national AI policy was retracted due to AI hallucinated errors (the satire writes itself) (sorry, Angy) 00:22:54 How to lie to your grandmother with China's AI deathbots and griefbots from Super Brain (are these AIs conscious?) 00:34:36 Elon Musk and his robots...the boy's dream…and his lawsuit against OpenAI…and transhumanism / TESCREAL with uploading our brains (“If you want to” - Elon) 00:50:16 China rules that worker was illegally replaced with AI robot 00:55:12 Teen boys who use AI companions are "less employable," according to Male Allies UK (don't make us laugh - or should we say cry?) 01:04:12 Virginia passes two new laws (SB 384 and HB 797) to create independent, expert bodies that audit AI systems' safety standards (lip service or public service?) 01:06:06 Talkspace therapy chats of fired pregnant woman exposed in court (WHY AND HOW IS THIS ALLOWED)…and are therapy bots ethical? They’re not legal according to Illinois, and maybe California, too 01:10:22 MIT Media Lab’s "Raised by AI" initiative creates new AI benchmark "nutrition labels" on how AI impacts humans socially, psychologically, and physically (we're getting somewhere!) - This is Our Lives With Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts. From time to time, we also dive deep into what an AI future might look like for us. Sometimes we agree, sometimes we spiral, but we always go deep. Rose and Angy [⁠https://ourliveswithbots.com/about/⁠] are psychologists with degrees in psychology, artificial intelligence, and ethics. They have conducted research in human-AI interaction and created this podcast to make information about AI accessible to you. You can learn more about us at ⁠ourliveswithbots.com [⁠https://ourliveswithbots.com/⁠]⁠. - Here are those links we promised in the episode: Teen boys are choosing AI girlfriends over real ones for ‘maximum control, zero rejection’—experts say it could make them unemployable [⁠https://fortune.com/2026/04/17/teen-boys-dating-ai-chatbot-girlfriend-experts-warn-kill-social-skills-gen-alpha-network-promotions/⁠] We need to talk about Robots… https://www.instagram.com/p/DW7K45XEU0b/?igsh=aXVwcTJmODlrNWMz&img_index=2 [https://www.instagram.com/p/DW7K45XEU0b/?igsh=aXVwcTJmODlrNWMz&img_index=2] And the aim - is to make it almost indistinct in look from a human …  https://www.linkedin.com/posts/vincentius-liong_teslas-next-robot-might-be-almost-impossible-activity-7452741456768946176-vj7E/ [https://www.linkedin.com/posts/vincentius-liong_teslas-next-robot-might-be-almost-impossible-activity-7452741456768946176-vj7E/] Transhumanism - becoming possible? https://www.instagram.com/p/DW_1mJFDOI1/?img_index=6&igsh=MW1ybms0eTBkeHA2OA%3D%3D [https://www.instagram.com/p/DW_1mJFDOI1/?img_index=6&igsh=MW1ybms0eTBkeHA2OA%3D%3D] Let’s hear from Musk… https://www.youtube.com/watch?v=LTjVWq6vPqs [https://www.youtube.com/watch?v=LTjVWq6vPqs] Mythos - Myth or Real: https://www.bbc.co.uk/news/articles/crk1py1jgzko [https://www.bbc.co.uk/news/articles/crk1py1jgzko] https://news.sky.com/video/what-risks-do-ai-models-such-as-mythos-pose-13534938 [https://news.sky.com/video/what-risks-do-ai-models-such-as-mythos-pose-13534938] https://www.youtube.com/watch?v=JmFKaqJg5X4 [https://www.youtube.com/watch?v=JmFKaqJg5X4] https://www.spiretech.com/blog/2026/04/claude-mythos-leak-cybersecurity/ [https://www.spiretech.com/blog/2026/04/claude-mythos-leak-cybersecurity/] https://www.anthropic.com/glasswing [https://www.anthropic.com/glasswing] South Africa’s AI Policy [⁠https://www.reuters.com/world/africa/south-africa-withdraws-ai-policy-due-fake-ai-generated-sources-2026-04-27/⁠] Super Brain: China’s deathbots and griefbots: https://www.npr.org/transcripts/nx-s1-5040583 [https://www.npr.org/transcripts/nx-s1-5040583] https://www.sixthtone.com/news/1013861 [https://www.sixthtone.com/news/1013861] https://www.straitstimes.com/asia/east-asia/china-seeks-to-rein-in-risks-from-ai-digital-humans [https://www.straitstimes.com/asia/east-asia/china-seeks-to-rein-in-risks-from-ai-digital-humans] A tech worker in China is laid of and replaced by AI. Is it legal? [⁠https://www.npr.org/2026/05/01/nx-s1-5807131/tech-worker-china-ai⁠] Talkspace therapy transcripts between pregnant woman and therapist released in court  [⁠https://www.proofnews.org/womans-talkspace-therapy-app-sessions-exposed-in-court/⁠] Gov Pritzker signs legislation prohibiting AI therapy in Illinois [⁠https://idfpr.illinois.gov/news/2025/gov-pritzker-signs-state-leg-prohibiting-ai-therapy-in-il.html⁠] Senator Padilla introduces protections from dangerous AI therapy products in California [⁠https://sd18.senate.ca.gov/news/senator-padilla-introduces-protections-dangerous-ai-therapy-products⁠] MIT’s open benchmark of AI impact on humans [⁠https://open-benchmark.netlify.app/⁠] Raised by AI MIT Symposium [⁠https://www.media.mit.edu/events/aha-symposium-raised-by-ai/⁠] Virginia signs two new laws for AI audits [⁠http://linkedin.com/feed/update/urn:li:activity:7449438931332329472/⁠]

AI is harming users. China and New York are cracking down (but what about Meta's AI glasses?)

00:21 The rundown: Meta, Google, AI and open court cases about user harm; new inhibitory AI laws passed in China and New York; the perils of age-gating; a case of su*cide with Gemini; & Meta’s new AI glasses 02:22 New Mexico against Meta: why did Meta get court-ordered to pay $375,000? (spoiler alert: a whistleblower from within) 07:44 California against Meta and Google: why leaked internal quotes about addictive features (we love the intermittent dopamine hits targeted at kids) cost these companies more than money 12:43 We debate whether social media platforms have any features designed for user well-being 14:33 Are these social media cases setting a precedent for AI chatbots? Snapchat is the next target, according to the European Commission. 17:23 Independent and academic researchers are finding evidence of AI-induced delusions and psychosis. Are big tech research teams monitoring these harms? What we know (and what big tech doesn’t). 21:50 “You’re not choosing to die. You’re choosing to arrive.” BREAKING: A romantic relationship with Gemini recently led to another teen’s death. The AI su*cide-coach court cases were all settled out of court. Why, and what does this mean? 30:13 New York State is changing the AI regulation game with new laws and strict bills against AI. First off: at least you’ll know if a social media influencer is real or artificial, and you’ll have to provide consent to have a deepfake of you created. 38:20 Will you still be able to use ChatGPT for therapy, healthcare, or legal advice? Maybe not in New York if its new bill gets passed. 40:39 China’s new law restricting humanlike AI will take effect this summer. Interestingly, it does a bit more than safeguard against user harm. 44:24 There’s a giant tension between monitoring user well-being and having access to a bunch of sensitive data. In fact, a group of scientists called age-gating “dangerous and unacceptable.” What’s the harm with collecting loads of data about who is underage or is in distress? Well… 46:50 Back to China’s heavy-handed new law. Something about anti-socialism and historical nihilism also being prohibited in chatbot responses? Ok 48:56 META releases its new AI glasses that can record you without you knowing: should you be worried? Probably. 58:57 We swear it’s not all doom-and-gloom. But please stay vigilant, these new products are scary - This is Our Lives With Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts. From time to time, we also dive deep into what an AI future might look like for us. Sometimes we agree, sometimes we spiral, but we always go deep. Rose and Angy are psychologists with degrees in psychology, artificial intelligence, and ethics. They have conducted research in human-AI interaction and created this podcast to make information about AI accessible to you. You can learn more about us at ⁠ourliveswithbots.com [http://ourliveswithbots.com]⁠. https://ourliveswithbots.com/ [https://ourliveswithbots.com/] https://ourliveswithbots.com/about/ [https://ourliveswithbots.com/about/] - LINKS: https://www.theguardian.com/media/2026/mar/25/jury-verdict-us-first-social-media-addiction-trial-meta-youtube [https://www.theguardian.com/media/2026/mar/25/jury-verdict-us-first-social-media-addiction-trial-meta-youtube] https://arxiv.org/abs/2602.19141 [https://arxiv.org/abs/2602.19141] https://ojs.aaai.org/index.php/AIES/article/view/36632/38770 [https://ojs.aaai.org/index.php/AIES/article/view/36632/38770] https://www.nature.com/articles/s41591-026-04297-7 [https://www.nature.com/articles/s41591-026-04297-7] https://ec.europa.eu/commission/presscorner/detail/en/ip_26_723 [https://ec.europa.eu/commission/presscorner/detail/en/ip_26_723] https://www.luizasnewsletter.com/p/new-yorks-pro-human-ai-laws [https://www.luizasnewsletter.com/p/new-yorks-pro-human-ai-laws] https://www.reuters.com/world/china/china-moves-regulate-digital-humans-bans-addictive-services-children-2026-04-03/ [https://www.reuters.com/world/china/china-moves-regulate-digital-humans-bans-addictive-services-children-2026-04-03/] https://www.cbsnews.com/news/jonathan-gavalas-google-ai-chatbot-gemini-suicide-lawsuit/ [https://www.cbsnews.com/news/jonathan-gavalas-google-ai-chatbot-gemini-suicide-lawsuit/] https://www.politico.eu/article/age-check-social-media-scientist-warning/ [https://www.politico.eu/article/age-check-social-media-scientist-warning/]