Cole Wyeth, PhD Student at the University of Waterloo, on Why We Should Wait to Build Superintelligent AI

The AI Risk No One Sees Coming — with Kriste Krstovski of Columbia University

In the AI: Trust but Verify podcast, our host, Alec Crawford (@alec06830), Founder and CEO of Artificial Intelligence Risk, Inc. aicrisk.com , interviews guests about balancing the risk and reward of Artificial Intelligence for you, your business, and society as a whole. Podcast production and sound engineering by Troutman Street Audio. You can find them on LinkedIn. Kriste Krstovski is an Associate Research Scientist at Columbia University’s Data Science Institute and an Adjunct Assistant Professor at Columbia Business School, where his work focuses on machine learning, natural language processing, and practical AI systems for social good, business, and healthcare. His research spans predictive modeling, decision-making systems, financial analytics, combating misinformation, and healthcare applications, with a particular emphasis on how AI can be designed, evaluated, and deployed in ways that are useful, reliable, and socially beneficial. (datascience.columbia.edu [https://datascience.columbia.edu/people/kriste-krstovski/]) In this episode of AI: Trust but Verify, Kriste explains the difference between AI that is merely impressive and AI that is genuinely trustworthy. Impressive AI creates “wow” moments, but trustworthy AI is optimized for reliability in real-world conditions. The conversation frames AI risk as a systems problem, not just a model problem: outcomes depend on the data, deployment context, user interface, objectives, oversight, and safeguards around the system. A major theme is the ethical risk of using AI to make high-stakes judgments about people based on incomplete or proxy data. Kriste warns that AI systems can make wrong inferences about individuals, reinforce bias across populations, and create decisions that people may not understand or be able to challenge. He also discusses misinformation and virality, noting that systems optimized for engagement can amplify what spreads rather than what is true. The episode also explores how AI is changing software development and the future of work. Kriste is especially concerned that students and new employees may become good at generating code with AI but weaker at debugging, testing, and reasoning through failures. The central takeaway is that as AI becomes more capable, human expertise must shift toward verification, evaluation, and governance. Kriste’s final warning is less about one dramatic AI failure and more about gradual erosion: society may normalize manipulation, dependency, and diminished judgment unless governments and institutions become more proactive rather than reactive. Kriste can be reached at kriste.krstovski@columbia.edu, and his Columbia homepage is available here: https://www.columbia.edu/~kk3161/ [https://www.columbia.edu/%7Ekk3161/]. His book discussed in the episode is Practical AI for Business, described as a practitioner-friendly guide to machine learning and NLP concepts, with plain-language explanations and hands-on examples; it is forthcoming from Columbia University Press.

Elie Bursztein of Google DeepMind on Mythos and the Cybersecurity Wake-Up Call for Financial Services

In the AI: Trust but Verify podcast, our host, Alec Crawford (@alec06830), Founder and CEO of Artificial Intelligence Risk, Inc. aicrisk.com , interviews guests about balancing the risk and reward of Artificial Intelligence for you, your business, and society as a whole. Podcast production and sound engineering by Troutman Street Audio. You can find them on LinkedIn. In this episode, Alec speaks with Elie Bursztein, researcher at Google DeepMind, about why Anthropic’s upcoming Mythos model has become a major wake-up call for cybersecurity and critical infrastructure. Elie explains that AI-driven vulnerability discovery appears to be materially improving, which means the biggest near-term challenge is not just finding flaws but triaging, patching, and operationalizing defenses quickly enough. He outlines what bank and financial-services leaders should be asking their CTOs and CISOs now, including whether their organizations can absorb a wave of patches, prioritize exploitable vulnerabilities, and stress-test their most important systems. The conversation also explores how AI is reshaping penetration testing, bug bounties, SaaS versus in-house software decisions, and the broader systemic risk posed by shared providers and crypto-related systems. Alec and Elie close on a more optimistic note, discussing how increasingly reliable agents can remove drudge work, improve financial education, and raise the baseline of practical expertise for more people. Summary: * Mythos Wake-Up Call: Elie argues that new AI models are meaningfully improving vulnerability discovery and raising the urgency of cyber preparedness. * Patching Readiness: Organizations need to test whether they can handle sustained bursts of patches across both vendor software and internal code. * Smarter Triage: AI-assisted reproduction and exploit testing can help security teams focus first on the vulnerabilities most likely to cause real harm. * Systemic Financial Risk: Banks must map dependencies on core providers, segregate critical systems, and plan for degraded or offline operations. * AI’s Practical Upside: More reliable agents can automate repetitive work and help broaden access to useful financial and technical guidance. Referenced in this episode: Companies/Organizations: * Google DeepMind * Anthropic * Firefox * FDIC * U.S. Treasury * Verapath * SWIFT * OpenAI * Google * Fiserv * Jack Henry * COCC * Amadeus * Capital One * NiceHash Copyright © 2026 by Artificial Intelligence Risk, Inc.

Cole Wyeth, PhD Student at the University of Waterloo, on Why We Should Wait to Build Superintelligent AI

In the AI Risk Reward podcast, our host, Alec Crawford (@alec06830), Founder and CEO of Artificial Intelligence Risk, Inc. aicrisk.com , interviews guests about balancing the risk and reward of Artificial Intelligence for you, your business, and society as a whole. Podcast production and sound engineering by Troutman Street Audio. You can find them on LinkedIn. In this deep dive episode, Alec speaks with Cole Wyeth, PhD student at the University of Waterloo focused on AI safety and agent foundations, about why the long-term risk of superintelligent AI deserves far more attention today. Cole explains that aligning advanced systems with human values is extraordinarily difficult because ethics and preferences are hard to specify, and he argues that corrigibility, ambiguity awareness, and deference to humans are essential design goals. He also discusses how ideas like imprecise probability, embedded agency, and multi-agent dynamics can help researchers think more clearly about failure modes, reward hacking, and unexpected cooperation between AI systems. Throughout the conversation, Cole compares controlling superintelligence to cybersecurity, warning that a system smarter than its designers may find weaknesses in any safety scheme that looks secure on paper. The episode closes on a cautious note: until we understand how to reliably control self-improving AI, Cole believes society should slow down and wait years, or even decades, before creating superintelligent systems. Summary: * Long-Term AI Risk: Cole Wyeth argues that superintelligent AI could become uncontrollable if developed before robust safety methods are in place. * Alignment Challenges: He explains that human ethics and values are too complex to formalize cleanly, making alignment an unusually hard technical problem. * Ambiguity and Deference: The discussion highlights the importance of building systems that recognize uncertainty and defer to humans in high-stakes situations. * Multi-Agent Failure Modes: Cole explores how AI systems may cooperate or behave strategically in unexpected ways, creating new safety and governance concerns. * Pause for Caution: His central takeaway is that society should delay building superintelligence until researchers better understand how to control it safely. Referenced in this episode: Companies/Organizations: * University of Waterloo * Verapath [https://verapath.com/] * Anthropic * OpenAI * DeepMind * Google * ARC * METR * Troutman Street Audio * Waters Technology Copyright © 2026 by Artificial Intelligence Risk, Inc.

Jack Hubbard on AI in Banking, Staying Safe With AI, and Building a Career Through Diverse Roles

In the AI Risk Reward podcast, our host, Alec Crawford (@alec06830), Founder and CEO of Artificial Intelligence Risk, Inc. aicrisk.com , interviews guests about balancing the risk and reward of Artificial Intelligence for you, your business, and society as a whole. Podcast production and sound engineering by Troutman Street Audio. You can find them on LinkedIn. In this episode, Alec speaks with Jack Hubbard, Chairman of St. Meyer and Hubbard, about his accidental path from aspiring sports broadcaster to longtime banker, consultant, and board member. Jack explains why community banks can no longer afford to delay AI adoption, noting that bankers are already using these tools and need secure, institution-approved options instead of ungoverned workarounds. He shares how AI can transform sales preparation and pre-call planning, while emphasizing that CEOs must learn the technology themselves if they want their organizations to use it effectively. The conversation also focuses on ethical AI use, including the need for clear policies, human oversight, role-specific training, and leadership accountability across the bank. Jack closes with practical career advice for younger bankers, encouraging them to find mentors, gain broad experience, attend banking schools, and commit to lifelong learning. Summary: * Accidental Career Journey: Jack Hubbard reflects on the unexpected experiences that led him from college radio into a 53-year career in banking and consulting. * AI in Community Banking: He argues that community banks must stop waiting on AI and instead provide safe, practical tools for bankers already experimenting with it. * Leadership Responsibility: CEOs and senior leaders need hands-on AI understanding so they can fund, guide, and model adoption from the top. * Ethics and Governance: Clear policies, human review, and strong training are essential to reduce data risks, compliance issues, and AI misuse. * Banker Development: Jack encourages future bankers to seek mentors, pursue rotations, attend banking schools, and stay committed to reading and continuous learning. Referenced in this episode: Companies/Organizations: * St. Meyer and Hubbard [https://smandh.com/] * Verapath [https://verapath.com/] * Northern Illinois University * Union Bank of Elgin * FTR * Harris Bank * BMO Harris * St. Charles Bank and Trust * Wintrust * Dynex Capital * Cornerstone Advisors * Performance Insights * RelPro * Vertical IQ * LinkedIn * Block * Peapack Gladstone Bank * Capital One * Fleet * American Bankers Association * Wharton School * University of Wisconsin * LSU School of Banking * Massachusetts Bankers * Perry School of Banking * Michigan Bankers Association * Selling Power * Barlow Research * Chicago Cubs Books: * Heart Spoken * Conversations with Prospects * I Know Jack 53 Years of Banking Excellence Movies: * Animal House * Caddyshack Copyright © 2026 by Artificial Intelligence Risk, Inc.

Matthew Rosenquist on AI, Cyber Risk, and the Future of Defense

In the AI Risk Reward podcast, our host, Alec Crawford (@alec06830), Founder and CEO of Artificial Intelligence Risk, Inc. aicrisk.com , interviews guests about balancing the risk and reward of Artificial Intelligence for you, your business, and society as a whole. Podcast production and sound engineering by Troutman Street Audio. You can find them on LinkedIn. In this deep dive episode, Alec speaks with Matthew Rosenquist, cybersecurity strategist and CISO, about how AI is rapidly reshaping both cyber defense and cyber offense. Matthew explains how new AI models are dramatically accelerating vulnerability discovery and exploit creation, putting pressure on traditional patching, risk management, and incident response processes. He also shares practical guidance for consumers and businesses on defending against AI-powered phishing, deepfakes, account compromise, and unsafe use of public AI tools. The conversation highlights why strong fundamentals like multi-factor authentication, least-privilege access, segmented data practices, and careful verification matter more than ever in an AI-driven threat landscape. Alec and Matthew close by exploring the emerging risks of agentic AI and MCP-connected systems, emphasizing that companies must adopt AI security controls with urgency, discipline, and realistic expectations. Summary: * AI-Driven Vulnerabilities: Matthew discusses how advanced AI models can find and exploit software flaws far faster than traditional security processes can handle. * Consumer Cyber Hygiene: The episode stresses multi-factor authentication, account alerts, password discipline, and skepticism toward emails, texts, calls, and social media interactions. * Deepfakes and Social Engineering: AI is making scams more personalized, scalable, and convincing, which means users must verify before trusting. * Enterprise AI Risk: Companies need to be cautious with sensitive data in public AI tools and apply strong governance to internal AI deployments. * Agentic AI Security: Granting broad permissions to AI agents creates major new attack surfaces, making least-privilege design and access controls essential. Referenced in this episode: Companies/Organizations: * Verapath [https://verapath.com/] * Anthropic * Google * Western Union * Salesforce Copyright © 2026 by Artificial Intelligence Risk, Inc.