BewAIre: Detecting Malicious Pull Requests at Scale with LLMs

As AI coding assistants accelerate software development, the volume of pull requests at Datadog has grown to nearly 10,000 per week, increasing the risk that malicious changes slip through due to review fatigue. To address this, Datadog built BewAIre, an LLM-powered code review system designed to identify malicious source code changes introduced by threat actors. By reducing approval fatigue for developers while increasing friction for attackers, BewAIre guides human reviewers to the areas where judgment matters most, without slowing developer velocity.

In this breakout session, Julien Doutre, Senior Software Engineer, and Kassen Qian, Senior Product Manager, will share why BewAIre was built, how it evolved from a hackathon experiment into a production-grade internal system, and the key architectural decisions and trade-offs involved along the way. They will discuss what worked, what didn’t, and the limitations they encountered when applying LLMs to security-critical workflows.

They will also cover how BewAIre is now being integrated into Datadog Code Security, and what it takes to turn an internal engineering tool into a product capability used at scale. Attendees will leave with practical lessons on building, hardening, and productizing LLM-powered systems and how you can use LLMs to minimize the security risks that those same LLMs can introduce.