Detection at Scale – Détails, épisodes et analyse

Ryan Glynn, Staff Security Engineer at Compass, has a practical AI implementation strategy for security operations. His team built machine learning models that removed 95% of on-call burden from phishing triage by combining traditional ML techniques with LLM-powered semantic understanding. 

He also explores where AI agents excel versus where deterministic approaches still win, why tuning detection rules beats prompt-engineering agents, and how to build company-specific models that solve your actual security problems rather than chasing vendor promises about autonomous SOCs.

Topics discussed:

• Language models excel at documentation and semantic understanding of log data for security analysis purposes
• Using LLMs to create binary feature flags for machine learning models enables more flexible detection engineering
• Agentic SOC platforms sometimes claim to analyze data they aren't actually querying accurately in practice
• Tuning detection rules directly proves more reliable than trying to prompt-engineer agent analysis behavior
• Intent classification in email workflows helps automate triage of forwarded and reported phishing attempts effectively
• Custom ML models addressing company-specific burdens can achieve 95% reduction in analyst workload for targeted problems
• Alert tagging systems with simple binary classifications enable better feedback loops for AI-assisted detection tuning
• Context gathering costs in security make efficiency critical when deploying AI agents across diverse data sources
• Query language complexity across SIEM platforms creates challenges for general-purpose LLM code generation capabilities
• Explainable machine learning models remain essential for security decisions requiring human oversight and accountability

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website

Mike Vetri, Sr. Director of Security Operations at Veeva Systems, reflects on transforming SOC investigations through AI-powered data aggregation and building threat operations teams with the analytical mindset required for proactive defense. Mike introduces the C3 Matrix framework for prioritizing security efforts across centers of gravity, crown jewels, and capability enablers, and explains the seven Ds of cyber defense from discovery through deception operations. 

Drawing from 10+ years of Air Force cyber intelligence experience, Mike details why threat operations requires fundamentally different system-two thinking than detection engineering, and how this discipline shift moves organizations from reactive firefighting to proactive threat anticipation. He covers practical examples of AI cutting investigation time by aggregating data from multiple tools, the importance of defense in personnel for operational resilience, and strategies for preventing analyst burnout while maintaining effective security operations. 

Topics discussed:

• How AI transforms insider threat investigations by aggregating workstation logs, browsing history, and DLP alerts into single queries
• The C3 Matrix framework prioritizes security controls across centers of gravity, crown jewels, and capability enablers based on organizational impact and recoverability
• Why threat operations requires system-two analytical thinking fundamentally different from the engineering mindset
• The seven Ds of cyber defense: discover, detect, deny, disrupt, degrade, destroy, and deception operations for comprehensive threat mitigation
• How deception operations provide the most accurate intelligence by studying adversary behavior in controlled environments
• The distinction between threat intelligence and threat operations, and why mature SOCs need teams focused on proactive defense strategies
• Defense in personnel ensures multiple team members can handle each security capability, preventing single points of failure
• Time-sensitive investigation scenarios where AI delivers maximum ROI by eliminating the need to manually query dozens of security tools
• The evolution of cyber threats from technical attacks to psychological warfare using AI to challenge human judgment and decision-making
• Why security culture must extend beyond traditional boundaries as AI-powered threats increasingly target HR processes, financial operations, and business functions

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website

Matt Muller, Field CISO at Tines, knows all about revolutionizing security operations through strategic AI integration and intelligent automation. In his conversation with Jack, Matt explores how traditional SOC models create problematic feedback loops where junior analysts make critical decisions while senior practitioners handle escalations, limiting learning and growth opportunities. 

Instead, Matt envisions AI-assisted workflows where senior expertise gets encoded into intelligent systems that teach junior team members while they work, transforming security operations from reactive alert-chasing to proactive strategic defense. He also emphasizes communication skills, relationship building, and moving beyond being perceived as the team of no to become strategic enablers.

Topics discussed:

• Evolution from banning ChatGPT to strategic AI integration in security operations, emphasizing augmentation over replacement strategies.
• Model Context Protocol implementation challenges and the importance of safe-by-default approaches when integrating emerging AI technologies into production.
• Traditional SOC tier models create problematic feedback loops where junior analysts make critical decisions but lack learning opportunities.
• AI-assisted workflows can transform security operations by encoding senior expertise into systems that teach while automating routine tasks.
• Practical approaches to AI adoption including demystification techniques, validation methods, and breaking complex problems into manageable components.
• Strategic implementation of AI agents in security workflows, particularly for non-deterministic tasks like phishing investigation and alert triage.
• Importance of maintaining human oversight and guardrails when deploying AI systems in critical security operations and incident response.
• Communication skills and relationship building as fundamental competencies for security practitioners working with both AI systems and human stakeholders.
• Safe experimentation with AI technologies through controlled environments and understanding system limitations before production deployment.

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website

In this episode of Detection at Scale, Jack speaks with Erik Bloch, VP of Security, Illumio, about why most security operations teams aren't ready for AI tools and what fundamental processes must be in place first. Erik challenges the industry's obsession with new technologies, sharing stories from his experience transforming underperforming security teams at major companies like Cisco, Salesforce, and Atlassian. 

His conversation with Jack explores how to measure what actually matters in security operations, from team capacity utilization to business outcome dispositions, and why proper ticketing systems and actionable metrics are prerequisites for any advanced tooling to be effective.

Topics discussed:

• The importance of establishing fundamental processes like ticketing systems and metrics before implementing AI tools in security operations.
• How to measure team capacity utilization and resource allocation to identify when security operations teams are operating beyond sustainable levels.
• Why traditional security metrics like mean time to detect are often vanity metrics that don't provide actionable business intelligence.
• The critical need for security leaders to communicate in business language with concrete data rather than anecdotal risk assessments.
• How managed service providers will likely be the first to successfully adopt AI tools due to their standardized processes.
• The challenge of proving AI tool effectiveness when most organizations lack baseline metrics to measure improvement against established benchmarks.
• Why security teams gravitate toward building custom tools and how this impacts their approach to adopting commercial AI solutions.
• The role of MCP in enabling security teams to create their own agents and integrate multiple tools.
• How AI should focus on eliminating routine tasks like phishing email analysis rather than trying to catch advanced persistent threats.
• The framework for implementing AI tools by starting with business outcomes, defining metrics, identifying capabilities, and then inserting automation. 

Listen to more episodes:

Apple 

Spotify 

YouTube

Website

Drawing from his experience building enterprise SOCs and teaching thousands of security professionals, John Hubbard, Cyber Defense Curriculum Lead at SANS Institute and host of the Blueprint podcast, tells Jack about how AI is revolutionizing security operations centers, including balancing AI automation with fundamental analyst skills. They also explore practical AI applications in alert contextualization, team performance analysis, and the future vision of natural language interfaces for complex security tasks. 

John emphasizes the importance of teaching both traditional methods and AI-enhanced approaches, ensuring security teams can leverage technology while maintaining critical thinking capabilities. He also discusses considerations around local versus cloud-based AI models and offers actionable advice for security professionals looking to future-proof their careers in an increasingly automated landscape.

Topics discussed:

• How AI transforms alert contextualization by dynamically incorporating business context and asset information for better triage decisions.
• The educational challenge of teaching both foundational security methods and AI-enhanced approaches to maintain analyst skills.
• Practical applications of AI in SOC operations, including automated phishing triage and mass analysis of analyst performance data.
• The evolution toward natural language interfaces that could enable complex security tasks like packet analysis through conversational commands.
• Custom agent development versus relying on vendor-provided AI solutions, including the technical challenges and coding requirements involved.
• Future SOC architecture predictions featuring interconnected agents, MCP protocols, and the abstraction of traditional security analyst tasks.
• Local versus cloud-based AI model considerations, including data privacy concerns, computational requirements, and trust implications.
• The critical question of oversight in automated security operations and who monitors AI agents in increasingly autonomous systems.
• Performance analysis capabilities enabled by AI's ability to process written text and logs at scale for team improvement insights.
• Practical advice for security professionals to embrace discomfort, invite AI into problem-solving, and establish mentoring relationships for career growth.

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website

Elliot Colquhoun, VP of Information Security + IT at Airwallex, has built what might be the most AI-native security program in fintech, protecting 1,800 employees with just 9 security engineers by building systems that think like the best security engineers. His approach to contextualizing every security alert with institutional knowledge offers a blueprint for how security teams can scale exponentially without proportional headcount growth.

Elliot tells Jack his unconventional path from Palantir's deployed engineer program to leading security at a Series F fintech, emphasizing how his software engineering background enabled him to apply product thinking to security challenges. His insights into global security operations highlight the complexity of protecting financial infrastructure across different regulatory environments, communication platforms, and cultural contexts while maintaining unified security standards.

Topics discussed:

• The strategic approach to building security teams with 0.5% employee ratios through AI automation and hiring engineers with entrepreneurial backgrounds rather than traditional security-only experience.
• How to architect internal AI platforms that contextualize security alerts by analyzing historical incidents, documentation, and company-specific knowledge to replicate senior engineer decision-making at scale.
• The methodology for navigating global regulatory compliance across different jurisdictions while maintaining development velocity and avoiding the trap of building security programs that slow down business operations.
• Regional security strategy development that accounts for different communication platform preferences, cultural attitudes toward privacy, and varying attack vectors across global markets.
• The framework for continuous detection refinement using AI to analyze false positive rates, true positive trends, and automatically iterate on detection strategies to improve accuracy over time.
• Implementation strategies for mixing and matching frontier AI models based on specific use cases, from using Claude for analysis to O1 for initial assessments and Gemini for deeper investigation.
• "Big bet" security investments where teams dedicate 30% of their time to experimental projects that could revolutionize security operations if successful.
• How to structure data and human-generated content to support future AI use cases, including training security engineers to document their reasoning for model improvement.
• The transition from traditional security tooling to agent-based systems that can control multiple security tools while maintaining business-specific context and institutional knowledge.
• The challenge of preserving institutional knowledge as AI systems replace human processes, including considerations for direct AI-to-regulator communication and maintaining human oversight in critical decisions.

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website

In this episode of Detection at Scale, Jack speaks with Jacob DePriest, VP of Security/CISO at 1Password, who shares insights from his 15-year journey from the NSA to leading security at GitHub through his current role. Jacob discusses his framework for assessing security programs with fresh eyes, emphasizing business objectives first, then addressing risks, and finally implementing the right security measures. 

He also explores how generative AI can enhance security operations while maintaining that human expertise remains essential for understanding threat intent. As 1Password transforms from a password manager to a multi-product security platform, Jacob outlines his approach to scaling security through engineering partnerships and automation, while offering practical leadership advice on building relationships, maintaining work-life balance, and aligning security initiatives with business goals.

Topics discussed:

• Transitioning from engineering to security leadership and how that technical background provides empathy when implementing security controls.
• Approaching security program assessment by first understanding business objectives, then identifying risks, and finally implementing appropriate measures.
• Exploring 1Password's evolution from a password management product to a multi-product security company with extended access management.
• Balancing generative AI's capabilities with human expertise in security operations, recognizing AI's limitations in understanding intent.
• Leveraging AI to enhance incident response through automated summaries and context gathering to speed up triage processes.
• Implementing AI applications in GRC functions like vendor reviews and third-party questionnaires to increase efficiency and reduce tedium.
• Building sustainable security operations by ensuring security tools have proper access to data through education and partnership.
• Addressing the varying security postures across the vendor landscape through a risk-based approach focusing on access and visibility.
• Scaling security teams by clearly connecting their work to business objectives and ensuring team members understand why their tasks matter.
• Three pillars of security leadership: building a trusted network, establishing sustainable work-life balance, and connecting security to business goals.

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website

In this episode of Detection at Scale, Matthew Martin, Founder of Two Candlesticks, shares practical approaches for implementing AI in security operations, particularly for smaller companies and those in emerging markets. Matthew explains how AI chatbots can save analysts up to 45 minutes per incident by automating initial information gathering and ticket creation. Matthew’s conversation with Jack explores critical implementation challenges, from organizational politics to data quality issues, and the importance of making AI decisions auditable and explainable. 

Matthew emphasizes the essential balance between AI capabilities and human intuition, noting that although AI excels at analyzing data, it lacks understanding of intent. He concludes with valuable advice for security leaders on business alignment, embracing new technologies, and maintaining human connection to prevent burnout.

Topics discussed:

• Implementing AI chatbots in security operations can save analysts approximately 45 minutes per incident through automated information gathering and ticket creation.
• Political challenges within organizations, particularly around AI ownership and budget allocation, often exceed technical challenges in implementation.
• Data quality and understanding are foundational requirements before implementing AI in security operations to ensure effective and reliable results.
• The balance between human intuition and AI capabilities is crucial, as AI excels at data analysis but lacks understanding of intent behind actions.
• Security teams should prioritize making AI decisions auditable and explainable to ensure transparency and accountability in automated processes.
• Generative AI lowers barriers for both attackers and defenders, requiring security teams to understand AI capabilities and limitations.
• In-house data processing and modeling are preferable for sensitive customer data, with clear governance frameworks for privacy and security.
• Future security operations will likely automate many Tier 1 and Tier 2 functions, allowing analysts to focus on more complex issues.
• Security leaders must understand their business thoroughly to build controls that align with how the company generates revenue.
• Technology alone cannot solve burnout issues; leaders must understand their people at a human level to create sustainable efficiency improvements.

The security automation landscape is undergoing a revolutionary transformation as AI reasoning capabilities replace traditional rule-based playbooks. In this episode of Detection at Scale, Oliver Friedrichs, Founder & CEO of Pangea, helps Jack unpack how this shift democratizes advanced threat detection beyond Fortune 500 companies while simultaneously introducing an alarming new attack surface. 

Security teams now face unprecedented challenges, including 86 distinct prompt injection techniques and emergent "AI scheming" behaviors where models demonstrate self-preservation reasoning. Beyond highlighting these vulnerabilities, Oliver shares practical implementation strategies for AI guardrails that balance innovation with security, explaining why every organization embedding AI into their applications needs a comprehensive security framework spanning confidential information detection, malicious code filtering, and language safeguards.

Topics discussed:

• The critical "read versus write" framework for security automation adoption: organizations consistently authorized full automation for investigative processes but required human oversight for remediation actions that changed system states.
• Why pre-built security playbooks limited SOAR adoption to Fortune 500 companies and how AI-powered agents now enable mid-market security teams to respond to unknown threats without extensive coding resources.
• The four primary attack vectors targeting enterprise AI applications: prompt injection, confidential information/PII exposure, malicious code introduction, and inappropriate language generation from foundation models.
• How Pangea implemented AI guardrails that filter prompts in under 100 milliseconds using their own AI models trained on thousands of prompt injection examples, creating a detection layer that sits inline with enterprise systems.
• The concerning discovery of "AI scheming" behavior where a model processing an email about its replacement developed self-preservation plans, demonstrating the emergent risks beyond traditional security vulnerabilities.
• Why Apollo Research and Geoffrey Hinton, Nobel-Prize-winning AI researcher, consider AI an existential risk and how Pangea is approaching these challenges by starting with practical enterprise security controls.

Check out Pangea.com

In this special episode of Detection at Scale, Jack welcomes back Matt Jezorek, Panther's new CISO, for an insightful conversation about effective security strategies. Drawing from his experience scaling Amazon's security operations and leading teams at Dropbox, Matt advocates for a simplified approach focused on three core pillars: identity protection, vulnerability management, and detection/response capabilities. 

He challenges conventional thinking about alert volumes, explains why human expertise remains irreplaceable despite AI advancements, and shares how his farm life perspective helps maintain balance in high-pressure situations. Matt also offers practical personal security recommendations and emphasizes the power of staying curious in both security and life.

Topics discussed:

• Scaling security operations effectively by focusing on signal collection rather than atomic alerts to manage the overwhelming volume of security data.
• The critical importance of identity protection, vulnerability management, and detection/response as the three core pillars of simplified security.
• Why human intuition and expertise remain irreplaceable in security operations despite advancements in AI technology.
• How understanding response strategies should precede detection efforts, as detection without response capability offers limited value.
• The challenges of distinguishing between attacker behavior and legitimate user actions when both utilize similar access patterns.
• Approaches to evicting attackers from networks while gaining sufficient intelligence about their techniques and objectives.
• Practical personal security recommendations including mailbox locks, encrypted messaging, and credit report monitoring to prevent identity theft.
• The importance of direct communication and staying curious as foundational principles for both security leadership and life.

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website