Practical AI-Enabled Observability for Agents and LLMs

You’re told to “go build agents” without clear guidance on what that actually means, how to do it well, or how to know if it is working. You are not a data scientist. You are a software engineer. In this talk, a Datadog AI product leader Shri Subramanian breaks down what changes when you move from building applications to building AI agents, and why familiar approaches like traditional testing and linear delivery fall short. We will explore how agent development shifts the focus from code alone to data, prompts, and evaluation, and why functional reliability matters just as much as operational reliability. You will see how Datadog approaches LLM observability, experiments, and evaluation driven development, including lessons learned from shipping and dogfooding agents ourselves. Attendees will leave with a clear mental model, practical best practices, and concrete next steps they can apply immediately to build, measure, and improve agents with confidence.