Can You Trust Your AI SOC Analyst? Testing the Limits of LLMs in Security Operations
About This Session
LLMs are showing up in SOC tools, from log triage to incident summaries. But can we trust their outputs in critical workflows? This session explores the promises and pitfalls of using LLMs in security operations. We’ll evaluate real-world use cases like auto-generating detections, summarizing incidents, and helping with reverse engineering tasks. Through examples and benchmarks, we’ll explore where LLMs shine, where they hallucinate, and how to build secure, auditable pipelines around them. Attendees will leave with a framework to evaluate AI tools in the SOC, and a clear sense of when to automate, when to supervise, and when to just say no.
Speaker
Charit Upadhyay
Senior Site Reliability Engineer - Oracle
Charit Upadhyay is a Senior Site Reliability Engineer at Oracle, specializing in building scalable, secure, and high-performance cloud infrastructures. With extensive experience across Kubernetes, Terraform, observability, and security operations, he has led initiatives integrating AI into DevOps and cloud security workflows. Charit’s work focuses on applying emerging AI technologies to enhance operational efficiency, mitigate risks, and strengthen threat detection in complex systems. He is an active contributor to industry conferences, a reviewer for multiple technical committees, and a strong advocate for practical, real-world applications of AI in security and reliability engineering.
