Metascience Matters

Fraud in Alzheimer's research | Charles Piller

Charles Piller, an Investigative Correspondent for Science Magazine, is the author of the 2025 book Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's. He has received more than 40 honors or awards from American Association for the Advancement of Science/Kavli Foundation, National Institute of Health Care Management, Society of Professional Journalists, National Academies/Eric and Wendy Schmidt, Los Angeles Press Club, American Society of Tropical Medicine and Hygiene, Gerald Loeb Foundation and UCLA, Western Publications Association, First Amendment Coalition, California State University, Computer Press Association, American Society of Business Press Editors, California Newspaper Publishers Association, National Council on Crime and Delinquency, and other organizations. His work has been nominated for the Pulitzer Prize nine times by STAT, the Los Angeles Times and The Sacramento Bee. CONTACT RANDY: Feedback: metasciencematters@gmail.com EPISODE LINK: Blots on a field?  https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease Picture imperfect https://www.science.org/content/article/research-misconduct-finding-neuroscientist-eliezer-masliah-papers-under-suspicion Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's https://www.amazon.com/Doctored-Fraud-Arrogance-Tragedy-Alzheimers/dp/1668031248 OUTLINE: 0:00 - Introduction 6:05 - How Charles became a science journalist 7:58 - Charles' 2022 piece, "Blots on a field?" 14:39 - Responses from scientists to this piece 22:22 - Charles' 2024 piece, "Picture imperfect" 30:29 - How Charles began working on his book "Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's" 33:15 - Charles' experience working with scientific sleuths 36:45 - The incentive for positive data in science 46:09 - Charles' NYT op-ed about how to combat fraud in Alzheimer's research; attacks from the Trump administration on science 53:22 - Charles' thoughts on registered reports and other ways to improve scientific incentives 58:34 - Upcoming work 59:22 - Advice for listeners 1:02:20 - Outro

Why So Many Cancer Biology Findings Fail to Replicate | Tim Errington

Tim Errington is the Senior Director of Research at the Center for Open Science. He led the Reproducibility Project: Cancer Biology, Systematizing Confidence in Open Research and Evidence (SCORE), as well as the implementation and evaluation of initiatives such as Registered Reports, Registered Revisions, responsible conduct of research trainings, and open science badges. CONTACT RANDY: Feedback: metasciencematters@gmail.com EPISODE LINKS: Investigating the replicability of preclinical cancer biology (Reproducibility Project: Cancer Biology led by Tim Errington): https://elifesciences.org/articles/71601 Bayer replication study: https://www.nature.com/articles/nrd3439-c1 Amgen replication study: https://www.nature.com/articles/483531a Reproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology (Companion paper to Reproducibility Project: Cancer Biology): https://elifesciences.org/articles/67995 What is replication?: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691 Study comparing standard reports and registered reports in psychology: https://journals.sagepub.com/doi/10.1177/25152459211007467 Blog post on the seemingly magical success of revision experiments: https://rajlaboratory.blogspot.com/2013/06/the-magical-results-of-reviewer.html Google's AI co-scientist paper: https://arxiv.org/abs/2502.18864 Machine-readable documents: https://journals.sagepub.com/doi/10.1177/2515245920970949 How open science helps researchers succeed: https://elifesciences.org/articles/16800 ZBW's Expedition to Open Science Land: https://expedition-open-science.org/ OUTLINE: 0:00 - Introduction 4:47 - Tim's origin story as a cancer biologist 6:38 - Initial interest in metascience 9:24 - Starting the Reproducibility Project: Cancer Biology 12:07 - How were the studies that were replicated chosen? 14:41 - Publishing the Reproducibility Project: Cancer Biology as a registered report 17:26 - Results from the Reproducibility Project: Cancer Biology 20:28 - Tim's experience throughout the years running the Reproducibility Project: Cancer Biology 25:21 - The difficulty of running cancer biology studies 27:54 - Judging whether a replication is successful 31:23 - What has the response to the Reproducibility Project: Cancer Biology been? 37:52 - Why aren't replication rates higher? 40:26 - Challenges of running cancer biology replication studies 45:43 - Caveats of preclinical disease models 49:13 - The incentive for positive data in science 57:05 - Systemic intervention vs. Individual policing 1:01:04 - The value of registered reports 1:07:38 - Registered revisions 1:10:48 - Falsifying theories early at the preclinical stage 1:15:21 - Different institutions (e.g., academic, industry) conducting different studies (e.g., preclinical, clinical) 1:17:34 - New initiatives at the Center for Open Science (Replication project of social/behavioral sciences, automated tools for predicting replication success, LifeCycle journal) 1:23:02 - AI scientists are trained on biased literature; distrust of academic literature in drug discovery 1:28:46 - Peer review 1:32:51 - Narrative in science 1:35:06 - 100-200 years into the future 1:40:29 - Advice for high school/undergraduate listeners 1:42:51 - Metascience manifests in every field 1:44:46 - Philosophy of science 1:47:49 - Outro

The Replication Crisis Changed Psychology. Did It Fix It? | Brian Nosek

Brian Nosek is the cofounder and Executive Director of the Center for Open Science. He co-developed the Implicit Association Test, a method that advanced the study of implicit bias. He then co-founded three non-profit organizations: Project Implicit to advance research about implicit bias, the Society for the Improvement of Psychological Science to improve the research culture in his home discipline, and the Center for Open Science (COS) to improve rigor, transparency, integrity, and reproducibility across research disciplines. He led the Reproducibility Project in Psychology, a replication of 100 studies from psychology, as well as multiple other replication projects, along with policy reforms such as open science badges, and the investigation of prediction markets for study replication. CONTACT RANDY: Feedback: metasciencematters@gmail.com EPISODE LINKS: Estimating the reproducibility of psychological science (Reproducibility Project in Psychology led by Brian Nosek): https://www.science.org/doi/10.1126/science.aac4716 False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant: https://journals.sagepub.com/doi/10.1177/0956797611417632 What is replication?: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691 Metascience conference: https://metascience.info/ International Conference on the Science of Science and Innovation: https://icssi.org/ International Congress on Peer Review and Scientific Publication: https://peerreviewcongress.org/ MetaROR: https://metaror.org/ OUTLINE: 0:00 - Introduction 2:56 - Brian's origin story as a psychologist 6:35 - Initial interest in metascience  10:57 - "Scientific Utopia," starting the Reproducibility Project in Psychology, and major fraud cases 17:57 - How did the Reproducibility Project in Psychology come together? 23:40 - Results of the Reproducibility Project in Psychology 27:03 - Response from the field, institutional changes 30:39 - Advice for high school/undergrad listeners 34:03 - Outro

Litigating multimillion-dollar scientific fraud cases | Eugenie Reich

Eugenie Reich is an attorney committed to taking on scientific fraud, understanding the incentives that drive it, and recovering misdirected research funding. She is also a former investigative science journalist committed to correcting the scientific record. Her 2009 book Plastic Fantastic, details a major fraud cause in physics at Bell Labs. Two of the cases she has litigated cases were against Biogen ($900 million settlement), and the Dana-Farber Cancer Institute ($15 million settlement). CONTACT RANDY: metasciencematters@gmail.com EPISODE LINKS: Cargo Cult Science: https://calteches.library.caltech.edu... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbTJLX0VUdWRiWWFwb1RWMTRNeWYzOWdTSnVqZ3xBQ3Jtc0tseDZ2WVZuTnhzdTRsYldhUHZiMHN2U0NTc3FFZy1FT0lsMGw2c2VlekhQUkYydjR1NFlwQTR6bEZCSmtudm9LZ09DNC1UWnFlNi1GeXBtaG9tNkpzQ2lFSlVMX1g4NjkxbTQ2RjV6SU9zMkItNUp0NA&q=https%3A%2F%2Fcalteches.library.caltech.edu%2F3043%2F1%2FCargoCult.pdf&v=SMRCynRDC0Q] The PubPeer Conundrum: https://www.tandfonline.com/doi/epdf/... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbS1Fc2l6NXN0alZNZ3JDZWxhX0NEYmx6V05FUXxBQ3Jtc0trenJldVhiUzU1dDBIRl9lRDRKVzR1bWR5cElnQ0RVZEk3eDVmRlg3S2dicDQtNWtHUmNhZklZYm9CelJJbXhhcWl3UnFNdDNZUmhEdjYyQ1NEaml5bTcwQndTZlQ3a1RtXzF6Tkx2d1lxbThwMUVDSQ&q=https%3A%2F%2Fwww.tandfonline.com%2Fdoi%2Fepdf%2F10.1080%2F08989621.2024.2390007&v=SMRCynRDC0Q] New Scientist: https://www.newscientist.com/ [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqazI2RV80U2RJX1VfenZLN0JLblZremNJTnM1UXxBQ3Jtc0tsMDJhTjMtX1RZWE5BU3FmV3BUTmhrRjF6dWRWWkVzemptLVNqcFJoV0E5RGYtcVY1X1JTX2prTk9zRWhvdHVoUnBMUlp5ZXFqTWQ4SDBwc2lNOE95ZDNQMzdxdmhfMU9PV3d5em1zdTNBVHBSVndINA&q=https%3A%2F%2Fwww.newscientist.com%2F&v=SMRCynRDC0Q] Skeptical Inquirer: https://skepticalinquirer.org/ [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbWpzakt0OFM3SUJWVVN2SnM5OE1HU1FuZ3FvUXxBQ3Jtc0tuT2tVOTRLZ3lxR0hPaVY3Z1FkQ3JnS3FSNENlbEdybFp5ZE00SlhUZEZScDJBN3VYMkNtUnN5MGhCc1REc3dTM1lYUThLWW5Na1Q2cXM5VExzSzNfLXVVbklENWNMT3k2LWVOVy1PaFpCZEJub0czWQ&q=https%3A%2F%2Fskepticalinquirer.org%2F&v=SMRCynRDC0Q] Special thanks to Dylan Bouscher OUTLINE: 0:00 [https://www.youtube.com/watch?v=SMRCynRDC0Q] - Introduction 4:20 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=260s] - How Eugenie became interested in science 6:25 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=385s] - Interest in scientific fraud 7:38 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=458s] - Deciding to become a science journalist 8:58 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=538s] - Fraud in physics at Bell Labs (the subject of her 2009 book Plastic Fantastic) 20:15 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=1215s] - Bell Labs coming under private ownership 22:53 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=1373s] - Interviewing scientists for her book 26:50 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=1610s] - The response from the physics community 31:10 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=1870s] - Deciding to become a lawyer 33:28 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=2008s] - The False Claims Act 36:29 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=2189s] - The qui tam provision of the False Claims Act 37:46 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=2266s] - The $900M case against Biogen 44:02 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=2642s] - The day-to-day of working on this case 46:30 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=2790s] - Impact of the case on biotech and pharma 48:40 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=2920s] - The $15M case against Dana-Farber 50:42 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=3042s] - Do universities have an incentive to protect accused researchers? 54:15 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=3255s] - Is the scale of fraud too large to be fixed? 57:56 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=3476s] - Does fraud damage public trust in science? 1:00:44 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=3644s] - Tools/solutions to combat these problems 1:04:32 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=3872s] - Advice to practicing scientists 1:05:55 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=3955s] - Advice and resources for listeners 1:08:28 [https://www.youtube.com/watch?v=SMRCynRDC0Q&t=4108s] - Outro

Forensic Metascience, the GRIM test, and technology for checking papers | James Heathers

James Heathers is the Founder and Director of the Medical Evidence Project, a venture of The Center for Scientific Integrity. He aims to reduce medical harm and improve patient outcomes by identifying and publicizing errors and miscondcut in the medical literature. He uses forensic meta-analytical techniques to detect and deconstruct errors arising from low-quality science and fraudulent work in areas that involve large numbers of patients. CONTACT RANDY: metasciencematters@gmail.com EPISODE LINKS: The original GRIM test paper: http://www1.psych.purdue.edu/~gfranci... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbUc3QldHY0I2V2d5SFhUbnVWU1A0WGFNczlDd3xBQ3Jtc0tuMjdxMlhpb0pfeV82dXF4SkE2NnYwTDgxbmR3bGZuNlJiLUlxQ2lHS0NYcWxBQlhBSXF5c2staTlNUHpPNlB1RUZqS0hheElkOGY5OXVxQk1fTDNZa3dIZ2pCS3FwVnJiZ05oWUl5VHVZcTlSaDVVUQ&q=http%3A%2F%2Fwww1.psych.purdue.edu%2F%7Egfrancis%2FClasses%2FPSY392%2FBrownHeathers2016.pdf&v=QH87h_4dbeg] Machine-readable documents: https://journals.sagepub.com/doi/10.1... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbmc5RFpiS3BIbDFXclkxa0o4c2hwQXRWU2tkUXxBQ3Jtc0tsaktld0d1WmxHNGhZMUtuYmZlU0hGZHRvQk9rRE51MWcwbFFjY3hTNVhPRWk0cU1La05VazNvYlNTUDRZLVNqTG9VV0pwMl90cExTcC1Ibld1YlBUd1J0dGFKUFlqT1JoUXh6Q1h0WkZHeFhkVl94QQ&q=https%3A%2F%2Fjournals.sagepub.com%2Fdoi%2F10.1177%2F2515245920970949&v=QH87h_4dbeg] Chaos in the Brickyard: https://www.science.org/doi/10.1126/s... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbWxaWmdfOGlyVGt6b0dpSFk4UFRXZGpDRHdBQXxBQ3Jtc0tsOHhKMTBfdXNTYl9CTUFkQTJLa0ZUMmozT3A1Q3FLYXNQWUk5MlJSejRBaWJJYWlPaGhnTkhuenRTaGE3Zm9tUDdpX3pPSlMycGFIYTllcWVmbDlCOTNYWnZlaDlnVzhsdHhNNjZNQ2t1QWZFZEx3NA&q=https%3A%2F%2Fwww.science.org%2Fdoi%2F10.1126%2Fscience.142.3590.339.a&v=QH87h_4dbeg] Books: The Man Who Only Loved Numbers: https://en.wikipedia.org/wiki/The_Man... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqa1pEUnFRUWt5dEY3eXlDclJ4UzdlZHRndnlNd3xBQ3Jtc0trYnh5R3pFSGlKd19wMVVDOV9LTXRNa0ZKSnVWenhCQmRBYzJtM3MxbXFZVkxYTkctMmtUZFdPeTZrcnJKNWNuZ25lOXpnUHpiN1dYRHhrQkhDZFdlM09QakJ0SUlNdmFkazBnbi1IMC1IY09Cdm5wbw&q=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThe_Man_Who_Loved_Only_Numbers&v=QH87h_4dbeg] The Emperor of All Maladies: https://en.wikipedia.org/wiki/The_Emp... [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbm5hMmNnNHRRbkQyMGNhR0MyVm5jbXhCWm5EZ3xBQ3Jtc0tuc0lhalFOVTBGVUhLZUZnY3VLUlB1aE9OaTB1ZERnaE5DSHdKVkJmSDZMQjV3Vzc1ZHdTWWFlVFNMNFJNVXdqR3cwYWJuczdRUm9jOHpxM1MyMnl1QlByLU5ta29LZnRMMTBjVXNXdmh1YnQwY0stQQ&q=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThe_Emperor_of_All_Maladies&v=QH87h_4dbeg] OUTLINE: 0:00 [https://www.youtube.com/watch?v=QH87h_4dbeg] - Introduction 5:27 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=327s] - How James has been the last couple days, and his radio voice 6:40 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=400s] - Writing the first book on Forensic Metascience 10:47 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=647s] - James' PhD work on heart rate variability 14:12 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=852s] - Starting to work on metascience 18:40 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=1120s] - The GRIM test 26:40 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=1600s] - Programmatically scaling statistical checks 33:25 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=2005s] - Centering quantitative results and peripheralizing narrative in scientific papers 39:36 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=2376s] - AI for metascience, unearthing data underneath narrative 48:11 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=2891s] - Pursuing cases of misconduct 52:45 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=3165s] - Founding and Directing the Medical Evidence Project 56:10 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=3370s] - Incentives for positive data, post-publication review, challenging the binary of positive and negative data 1:05:41 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=3941s] - Advice and resources for listeners 1:12:40 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=4360s] - Optimism over pessimism 1:16:58 [https://www.youtube.com/watch?v=QH87h_4dbeg&t=4618s] - Outro