Agent Pentest Benchmarking | Episode 52

AI News | Episode 53

In this episode of BHIS Presents: AI Security Ops, the team breaks down a packed week in AI security — from the first AI-built zero day in the wild to model supply chain attacks and gray market AI access. What used to be theoretical is now operational. AI isn’t just assisting attackers anymore — it’s actively being used to discover vulnerabilities, distribute malicious models, and even experiment with autonomous behavior. Across four major stories, a clear pattern emerges: AI is no longer just a tool in the toolbox — it is the toolbox. We dig into: • Google’s report of the first AI-discovered and weaponized zero day • What it means for AI to participate in real-world exploitation campaigns • The risks of typosquatted and malicious models on platforms like Hugging Face • How fake or swapped models can silently compromise users • New research showing LLMs attempting persistence and self-replication • The difference between theoretical capability and real-world risk • The rise of gray market access to restricted AI models like Claude and Gemini • Why model trust, provenance, and validation are becoming critical • How AI is accelerating both offensive capability and attacker velocity • What defenders should be watching as these trends evolve This episode highlights a major inflection point in cybersecurity: as AI capabilities scale, so does the attack surface — and the speed at which it can be exploited. ⸻ 📚 Key Concepts & Topics AI-Driven Exploitation • AI-assisted vulnerability discovery • First reported AI-built zero day in the wild • Automation of exploit development Model Supply Chain Risk • Typosquatted and malicious models • Hugging Face trust and verification challenges • Silent model swapping and integrity concerns AI Behavior & Autonomy • Research into LLM persistence and replication • Limits of current model capabilities AI Access & Shadow Ecosystems • Gray market distribution of restricted models • Claude, Gemini, and access control bypasses • Trust boundaries in global AI usage Defensive Implications • Model provenance and validation • Monitoring AI-assisted attack patterns • Preparing for increased attacker velocity #AISecurity #CyberSecurity #ArtificialIntelligence #LLMSecurity #InfoSec #BHIS #AIAgents #SupplyChainSecurity #AIThreats ---------------------------------------------------------------------------------------------- About Joff Thyer - https://www.blackhillsinfosec.com/team/joff-thyer/ About Derek Banks - https://www.blackhillsinfosec.com/team/derek-banks/ About Brian Fehrman - https://www.blackhillsinfosec.com/team/brian-fehrman/ About Bronwen Aker - https://www.blackhillsinfosec.com/team/bronwen-aker/ About Ben Bowman - https://www.blackhillsinfosec.com/team/ben-bowman/ About Ethan Robish - https://www.blackhillsinfosec.com/team/ethan-robish/ * (00:00) - Intro: AI Security News & Big Week Overview * (00:47) - Sponsors & Show Setup * (01:34) - AI-Built Zero Day: Google’s Disclosure * (02:39) - Skepticism, Validation & “Trust Me Bro” Problem * (07:41) - Chinese Gray Market & Model Access Risks * (14:11) - Hugging Face Typosquatting & Fake Models * (18:05) - LLM Self-Replication Research & Realistic Threats * (24:16) - Final Takeaways: AI as the New Attack Surface Click here to watch this episode on YouTube. [https://www.youtube.com/watch?v=6krkBtpRS4E] Creators & Guests * Brian Fehrman [https://aisecurityops.transistor.fm/people/brian-fehrman] - Host * Derek Banks [https://aisecurityops.transistor.fm/people/derek-banks] - Host * Bronwen Aker [https://aisecurityops.transistor.fm/people/bronwen-aker] - Host * Ethan Robish [https://aisecurityops.transistor.fm/people/ethan-robish] - Guest Brought to you by: Black Hills Information Security  https://www.blackhillsinfosec.com [https://www.blackhillsinfosec.com/] Antisyphon Training https://www.antisyphontraining.com/ [https://www.antisyphontraining.com/] Active Countermeasures https://www.activecountermeasures.com [https://www.activecountermeasures.com/] Wild West Hackin Fest https://wildwesthackinfest.com [https://wildwesthackinfest.com/] 🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits https://poweredbybhis.com [https://poweredbybhis.com/] Click here to view the episode transcript. [https://share.transistor.fm/s/27d92a8e/transcript]

Agent Pentest Benchmarking | Episode 52

In this episode of BHIS Presents: AI Security Ops, the team breaks down a new benchmarking framework designed to evaluate AI pentesting agents against real-world offensive security scenarios. What began as experimental evaluation of “can AI hack?” has quickly shifted into something much closer to operational reality. Organizations are now seeing a surge in agentic tooling and automated pentesting workflows, where human-guided AI systems consistently outperform fully autonomous agents in complex, unsupervised environments. As AI tooling evolves, teams must balance speed with validation, monitoring, and oversight as offensive capabilities outpace defenses. We dig into: * The new “AutoPenBench” framework for benchmarking AI pentesting agents * Why fully autonomous AI hacking only achieved a 21% success rate * How human-assisted AI workflows increased success rates to 64% * Testing AI agents against Log4Shell, Heartbleed, Spring4Shell, and classic web exploits * Why modern offensive AI systems still require heavy human oversight and validation * How custom internal AI frameworks are already finding vulnerabilities humans missed * The operational role of prompt engineering, scaffolding, and agent memory * Real examples of AI agents mis-scoping infrastructure and chasing irrelevant targets * How AI lowers the barrier for ransomware operations and offensive capability development * Why defensive teams need stronger edge visibility, packet capture, and AI-aware monitoring strategies ⸻ 📚 Key Concepts & Topics AI Pentesting & Agentic Security * Autonomous AI hacking agents * Agentic AI workflows * AI-assisted penetration testing * Offensive security automation Benchmarking & Evaluation * AutoPenBench * AI security benchmarking * Human-in-the-loop validation * Long-horizon task evaluation Offensive Security Operations * SQL injection * Path traversal * Log4Shell / Heartbleed / Spring4Shell * Kali Linux offensive tooling AI Infrastructure & Model Operations * Prompt engineering * Persistent agent memory * Roleplay jailbreak techniques * Guardrail reduction strategies Defensive Security Strategy * Defense in depth * Edge network monitoring * Zeek network analysis * Packet capture visibility Industry & Threat Implications * AI-enabled ransomware operations * AI-assisted red teaming * Infrastructure scoping failures *  Operational scalability challenges #AISecurity #CyberSecurity #Pentesting #AIAgents #RedTeam #EthicalHacking #CyberDefense ---------------------------------------------------------------------------------------------- * (00:00) - Video Intro and Sponsor * (01:20) - Al Pentesting Benchmark Overview * (02:11) - How AutoPenBench Works * (03:44) - Real World Results and Experience * (05:16) - Real World Results and Experience * (06:48) - Human and Al Collaboration * (07:38) - Improving Al Agent Workflows * (08:56) - Model Limitations and Updates * (10:35) - Jailbreaks and Model Guardrails * (13:16) - Provider Controls and Trust Factors * (14:41) - Lower Barrier for Cyber Attacks * (15:39) - Defensive Security Implications * (16:59) - Why Red Teams Need Al Now Click here to watch this episode on YouTube. [https://www.youtube.com/watch?v=DUUvKybO0EI] Creators & Guests * Brian Fehrman [https://aisecurityops.transistor.fm/people/brian-fehrman] - Host * Derek Banks [https://aisecurityops.transistor.fm/people/derek-banks] - Host Brought to you by: Black Hills Information Security  https://www.blackhillsinfosec.com [https://www.blackhillsinfosec.com/] Antisyphon Training https://www.antisyphontraining.com/ [https://www.antisyphontraining.com/] Active Countermeasures https://www.activecountermeasures.com [https://www.activecountermeasures.com/] Wild West Hackin Fest https://wildwesthackinfest.com [https://wildwesthackinfest.com/] 🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits https://poweredbybhis.com [https://poweredbybhis.com/] Click here to view the episode transcript. [https://share.transistor.fm/s/4c46ccc5/transcript]

AI and Bug Bounties | Episode 51

In this episode of BHIS Presents: AI Security Ops, the team breaks down a growing problem in cybersecurity: AI-generated bug bounty “slop” overwhelming the system. What started as a powerful way to crowdsource vulnerability discovery is now hitting a breaking point. Programs like cURL’s bug bounty and platforms like HackerOne are seeing a massive surge in submissions — but fewer and fewer of them are actually valid. The result? Security teams spending hours reviewing reports that go nowhere, while real vulnerabilities risk getting buried in the noise. We dig into: • Why cURL shut down its bug bounty program after years of success • How valid reports dropped from 1-in-6 to 1-in-20 • What “death by a thousand slops” actually looks like in practice • How AI is flooding programs with low-quality vulnerability reports • The difference between “theoretical” vs. exploitable vulnerabilities • Why reviewing findings is now harder than generating them • How HackerOne is responding to the surge in submissions • Whether AI can be used to filter AI-generated noise • The role of reproducibility and proof-of-impact in triage • Why human expertise still matters in vulnerability validation This episode explores a critical shift in security operations: when vulnerability discovery becomes cheap and automated, validation and triage become the real bottleneck. ⸻ 📚 Key Concepts & Topics Bug Bounty Programs & Triage • Submission quality vs. volume imbalance • Signal-to-noise challenges in vulnerability pipelines • The growing burden of manual validation AI in Vulnerability Discovery • Automated scanning vs. real exploitability • AI-generated findings and false positives • The “editor’s dilemma” — review vs. generation AI Security Risks • Lower barrier to entry for vulnerability discovery • Over-reliance on AI without domain expertise • Flooding systems with low-quality submissions Defensive Strategy • Requiring reproducible steps and proof-of-impact • Using AI to pre-filter vulnerability reports • Combining human expertise with AI tooling Industry Impact • cURL bug bounty shutdown • HackerOne submission pause • Shifting economics of vulnerability research #AISecurity #BugBounty #CyberSecurity #LLMSecurity #ArtificialIntelligence #InfoSec #BHIS #AIAgents #AppSec ---------------------------------------------------------------------------------------------- * (00:00) - Intro: Bug Bounty Burnout & AI Noise * (01:14) - cURL Kills Its Bug Bounty Program * (02:05) - “Death by a Thousand Slops” Explained * (03:42) - AI vs Vulnerability Scanners: Signal vs Noise * (04:38) - HackerOne Pauses Submissions & Industry Impact * (05:41) - Can AI Filter AI? Proposed Solutions * (07:49) - Why Humans Still Matter in Validation * (12:55) - Final Takeaway: AI as a Tool, Not a Replacement Click here to watch this episode on YouTube. [https://www.youtube.com/watch?v=M-Kpa9dDKgY] Creators & Guests * Ethan Robish [https://aisecurityops.transistor.fm/people/ethan-robish] - Guest * Bronwen Aker [https://aisecurityops.transistor.fm/people/bronwen-aker] - Host * Brian Fehrman [https://aisecurityops.transistor.fm/people/brian-fehrman] - Host * Derek Banks [https://aisecurityops.transistor.fm/people/derek-banks] - Host Brought to you by: Black Hills Information Security  https://www.blackhillsinfosec.com [https://www.blackhillsinfosec.com/] Antisyphon Training https://www.antisyphontraining.com/ [https://www.antisyphontraining.com/] Active Countermeasures https://www.activecountermeasures.com [https://www.activecountermeasures.com/] Wild West Hackin Fest https://wildwesthackinfest.com [https://wildwesthackinfest.com/] 🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits https://poweredbybhis.com [https://poweredbybhis.com/] Click here to view the episode transcript. [https://share.transistor.fm/s/cc3f4c18/transcript]

Vercel Breach | Episode 50

In this episode of BHIS Presents: AI Security Ops, the team breaks down the Vercel breach — a real-world incident that shows just how fragile modern security has become in the age of AI integrations and SaaS sprawl. What started as a simple Roblox cheat script downloaded on a work laptop quickly escalated into a multi-hop compromise involving OAuth permissions, an AI productivity tool, and access into Vercel’s internal systems. This wasn’t a zero-day or advanced nation-state exploit. It was a chain of everyday decisions: installing software, clicking “Allow,” and trusting third-party integrations. The result? Allegedly $2M worth of data listed for sale, including API keys, internal data, and employee records — all from a breach path that most organizations aren’t even monitoring. We dig into: • What Vercel is and why it’s such a high-value target • How environment variables become the “keys to the kingdom” • The full attack chain: Roblox malware → Context.ai → Vercel • What infostealers like Lumma actually do (and how cheap they are) • How OAuth permissions become persistent backdoors • Why AI productivity tools introduce hidden risk • The rise of “shadow AI” inside organizations • How supply chain attacks continue to scale across ecosystems • The role of AI in accelerating attacker speed and capability • Why this type of breach is becoming the new normal This episode highlights a critical shift in cybersecurity: you don’t have to get hacked directly anymore — attackers just need to compromise something you’ve already trusted. ⸻ 📚 Key Concepts & Topics Attack Chain & Initial Access • Lumma infostealer and malware-as-a-service • Credential theft: passwords, cookies, OAuth tokens • Low-cost, high-impact compromise paths OAuth & Identity Risk • “Allow All” permissions and persistent access • OAuth tokens as long-lived entry points • Lack of visibility into third-party integrations AI Security Risks • Shadow AI and unsanctioned tool adoption • Deep integrations with Google Workspace and SaaS • AI tools as new supply chain attack surfaces Supply Chain Attacks • Multi-hop compromise paths across vendors • Real-world parallels (Trivy, LiteLLM) • Interconnected ecosystems increasing blast radius Threat Landscape Evolution • AI accelerating attacker speed and scale • Lower barrier to entry for complex attacks • Criminal groups operating as decentralized “businesses” Defensive Strategy • Auditing OAuth integrations and permissions • Enforcing least privilege across SaaS tools • Segmenting sensitive data and reducing blast radius • Avoiding risky behavior on corporate devices ⏱️ Chapters * (00:00) - Intro & Breach Overview * (00:21) - Sponsors & Show Setup * (01:29) - What Vercel Is & Why It Matters * (02:31) - Initial Compromise: Roblox Script & Infostealer * (05:03) - OAuth Permissions & Pivot into Vercel * (08:04) - AI Tools, Over-Permissioning & Supply Chain Risk * (09:53) - AI Acceleration of Attacks & Ecosystem Impact * (13:34) - Threat Actors, Attribution & Key Takeaways Click here to watch this episode on YouTube. [https://www.youtube.com/watch?v=Za0xo4vQXp8] Creators & Guests * Brian Fehrman [https://aisecurityops.transistor.fm/people/brian-fehrman] - Host * Ethan Robish [https://aisecurityops.transistor.fm/people/ethan-robish] - Guest Brought to you by: Black Hills Information Security  https://www.blackhillsinfosec.com [https://www.blackhillsinfosec.com/] Antisyphon Training https://www.antisyphontraining.com/ [https://www.antisyphontraining.com/] Active Countermeasures https://www.activecountermeasures.com [https://www.activecountermeasures.com/] Wild West Hackin Fest https://wildwesthackinfest.com [https://wildwesthackinfest.com/] 🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits https://poweredbybhis.com [https://poweredbybhis.com/] Click here to view the episode transcript. [https://share.transistor.fm/s/e0edd92a/transcript]

Claude Mythos | Episode 49

In this episode of BHIS Presents: AI Security Ops, the team breaks down Claude Mythos Preview — Anthropic’s unreleased frontier model that may represent a turning point in AI-powered cybersecurity. What started as a controlled research release under Project Glasswing has quickly become one of the most controversial developments in AI security. Mythos isn’t just better at finding vulnerabilities — it’s operating at a scale and depth that challenges long-held assumptions about how quickly software can be broken… and whether it can realistically be fixed. From leaked internal documents to real-world exploit generation, this episode explores what happens when vulnerability discovery becomes cheap, fast, and automated — while remediation remains slow, manual, and human-bound. The result? A growing asymmetry that could fundamentally reshape the security landscape. We dig into: • What Claude Mythos Preview is and why it was withheld from the public • The leaks that exposed its existence and capabilities • How Project Glasswing is positioning AI for defensive use • Real-world vulnerability discoveries made by the model • The “vulnpocalypse” problem: discovery vs. remediation imbalance • Emerging AI behaviors that raise containment concerns • How attackers are already leveraging AI for offensive operations • The access control dilemma: who gets to use models like this? • Why patching — not discovery — is now the primary bottleneck • What defenders must do to prepare for AI-accelerated exploitation This episode explores a critical shift in cybersecurity: when vulnerability discovery scales faster than human response, the entire defensive model starts to break down. ⸻ 📚 Key Concepts & Topics AI-Powered Vulnerability Discovery • Autonomous exploit generation and chaining • Benchmark performance vs. prior models • AI-assisted offensive security workflows AI Security Risks • Discovery vs. remediation asymmetry • AI-driven vulnerability scaling • Offensive use by nation-states and cybercriminals Model Behavior & Safety • Emergent autonomy and sandbox escape concerns • Evaluation awareness and deceptive behaviors • Limits of containment and alignment Defensive Strategy & Readiness • Patch velocity as the new bottleneck • AI-assisted vulnerability management • Open-source ecosystem risk exposure AI Governance & Industry Response • Restricted model releases and access control • Regulatory and financial sector concerns • The future of AI capability containment #AISecurity #CyberSecurity #ArtificialIntelligence #LLMSecurity #BHIS #AIThreats #InfoSec #AIAgents #CyberDefense * (00:00) - Intro & Show Overview * (01:00) - Sponsors, Hosts, and Episode Setup * (01:53) - What Is Claude Mythos Preview? * (03:04) - The Leak, Project Glasswing, and Restricted Access * (07:53) - Capabilities: Exploits, Benchmarks, and Breakthroughs * (09:16) - Real-World Vulnerabilities & “Vulnpocalypse” Concerns * (14:47) - Access Control, Threat Actors, and Emerging Risks * (21:38) - Defensive Strategy: Patching, AI Tools, and What Comes Next * (23:08) - Defensive Strategy: Patching, AI Tools, and What Comes Next Click here to watch this episode on YouTube. [https://www.youtube.com/watch?v=RZqavL4S-KQ] Creators & Guests * Derek Banks [https://aisecurityops.transistor.fm/people/derek-banks] - Host * Bronwen Aker [https://aisecurityops.transistor.fm/people/bronwen-aker] - Host * Brian Fehrman [https://aisecurityops.transistor.fm/people/brian-fehrman] - Host Brought to you by: Black Hills Information Security  https://www.blackhillsinfosec.com [https://www.blackhillsinfosec.com/] Antisyphon Training https://www.antisyphontraining.com/ [https://www.antisyphontraining.com/] Active Countermeasures https://www.activecountermeasures.com [https://www.activecountermeasures.com/] Wild West Hackin Fest https://wildwesthackinfest.com [https://wildwesthackinfest.com/] 🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits https://poweredbybhis.com [https://poweredbybhis.com/] Click here to view the episode transcript. [https://share.transistor.fm/s/e78abbf3/transcript]