Introducing Built-in AgentOps Tools in Azure AI Foundry Agent Service

A New Era of Agent Intelligence

We’re thrilled to announce the public preview of Tracing, Evaluation, and Monitoring in Azure AI Foundry Agent Service, features designed to revolutionize how developers build, debug, and optimize AI agents.

With detailed traces and customizable evaluators, AgentOps is here to bridge the gap between observability and performance improvement. Whether you’re managing a simple chatbot or a complex multi-agent system, this is the tool you’ve been waiting for.

What Makes AgentOps Unique?

AgentOps offers an unparalleled suite of functionalities that cater to the challenges AI developers face today. Here are the two cornerstone features:

1. Integrated Tracing Functionality

AgentOps provides full execution tracing, offering a detailed, step-by-step breakdown of how agents process queries, interact with tools, and make decisions. By leveraging OpenTelemetry-supported traces, developers can gain insights into critical aspects of agent workflows, including:

• Execution Paths: Visualize an agent’s full reasoning and decision-making process across multi-agent workflows.
• Performance Monitoring: Track timestamps, latency, and token consumption to identify bottlenecks and optimize agent efficiency.
• Tool Invocation Logs: Monitor the success, failure rates, and duration of tools like file search, Grounding with Bing Search, code interpreters, OpenAPI, and more.
• Detailed Request/Response: Access granular logs for every interaction and activity thread, helping developers debug with precision.

2. Advanced Evaluation Framework

AgentOps isn’t just about tracing; it elevates evaluation to a new level with cutting-edge features that allow developers to assess and improve agent behavior through built-in and customizable metrics. Here’s what the evaluation functionality brings to the table:

Comprehensive Metrics

Azure AI Foundry Agent Service enables statistical analysis of agent outputs within Agent Playground using new evaluation metrics, including:

• Performance Evaluators: Measure latency, token consumption, request logs, and tool invocation efficiency across each step of the agent’s activity thread.
• Quality Evaluators: Assess outputs for intent resolution, coherence, fluency, and accuracy, ensuring high-quality responses.
• Safety Evaluators: Identify risks in agent responses, such as hate speech, indirect attacks, and code vulnerabilities.

3. Monitor Azure AI Foundry Agent Service

Continue to monitor and assess your system using Azure Monitor. The following Azure Monitoring Metrics are now available in the Azure Portal through Hubs and Projects, and are coming soon on Foundry Developer Platform:

These monitoring metrics enhance the visibility and operational insights needed for AI agent workflows, ensuring robust analysis and optimization. From there, continuously evaluate and monitor your agent in production with Azure AI Foundry Observability.

Seamless Integration

AgentOps integrates deeply into your existing workflows and tools, meeting developers where they are. With support for SDKs, portals, and third-party observability tools like Weights & Biases, you can start tracing and evaluating your agents with minimal setup. Whether you’re using Azure AI Foundry, OpenTelemetry, or custom pipelines, AgentOps in Foundry Agent Service works effortlessly across diverse AI ecosystems.

Why AgentOps Matters

AgentOps solves the most pressing challenges faced by AI developers today, including:

• Debugging Complexity: Simplify error detection and resolution with end-to-end execution visibility.
• Fine-Tuning Efficiency: Optimize agent performance by identifying bottlenecks and improving cost-effectiveness.
• Building Trust: Enhance the reliability and explainability of your agents with quality and safety evaluators.

What’s Next?

• Explore the documentation to get started with AgentOps in Azure AI Foundry Agent Service.
• Evaluate your AI agents locally with Azure AI Evaluation SDK.
• View Monitoring data reference for metrics created for Azure AI Foundry Agent Service.