Monitor AI Agents in Production with Zero Code

This content originally appeared on DEV Community and was authored by Elizabeth Fuentes L

Learn to monitor AI agents in production with Amazon Bedrock AgentCore Observability. Zero-code implementation, real-time traces, and production-ready dashboards. Step-by-step tutorial.

🇻🇪🇨🇱 Dev.to LinkedIn GitHub Twitter Instagram YouTube

You deployed your AI agent to production with AgentCore Runtime and added cross-session memory. Now users interact with your agent, but can you answer these questions: How do you trace agent decisions? What happens when behavior is inconsistent? How do you detect performance degradation?

Traditional monitoring shows infrastructure status. Agent observability reveals whether your AI makes effective decisions. Amazon Bedrock AgentCore Observability solves this problem. When you deploy your agent on AgentCore Runtime, observability works automatically—complete visibility without instrumentation code or configuration changes.

📊 AgentCore Observability by Services Overview

🧩 Understanding Sessions, Traces, and Spans

AgentCore Observability uses a three-tier hierarchy providing visibility at different granularity levels:

• 1️⃣ Sessions represent complete interaction context between you and your agent, from initialization to termination. Sessions provide high-level views of engagement patterns, monitor agent performance, and show how you interact with agents over time. Learn more in Sessions in AgentCore documentation.

• 2️⃣ Traces capture single request-response cycles within sessions—the complete execution path from agent invocation to response. Traces include request details, processing steps, tool invocations, resource utilization, error information, and response generation. This provides deep insights into internal agent workings for troubleshooting and optimization. Learn more in Traces in AgentCore documentation.

• 3️⃣ Spans represent discrete, measurable units of work within traces—fine-grained operations with defined start and end times. Spans capture operation names, timestamps, parent-child relationships, tags, events, and status information. Spans form hierarchical structures within traces; for example, a "process user query" span contains child spans for "parse input," "retrieve context," and "generate response." Learn more in Spans in AgentCore documentation.

🚀 The AgentCore Observability Advantage

• Zero Development - Automatic instrumentation for Runtime agents
• Complete Visibility - Sessions, traces, spans, and metrics in one place
• Pre-Built Dashboards - CloudWatch GenAI Observability ready to use
• Framework Agnostic - Works with Strands Agents, LangChain, CrewAI, or custom code
• OpenTelemetry Standard - Industry-standard telemetry format
• Production-Grade - Built for scale and enterprise reliability

📖 Learn more in AgentCore Observability Overview documentation.

Step-by-Step: Enable CloudWatch for AI Agent Monitoring

🔧 Console Setup (5-minute tutorial)

Important: This is a one-time setup per AWS account. After you enable it, all AgentCore agents automatically send observability data to CloudWatch.

Via Console:

1. Open the CloudWatch Console
2. Navigate to Application Signals → Transaction Search
3. Choose Enable Transaction Search
4. Select the checkbox to ingest spans as structured logs
5. Set indexing percentage to 1% (free tier) or adjust based on your requirements

Wait 10 minutes after enabling for spans to become available for search and analysis.

Learn more: Enable Transaction Search

✨ Verify Observability is Active

When you deploy your agent with agentcore launch, observability activates automatically. No additional configuration needed.

Use the AgentCore CLI to check your agent's status:

agentcore status

Your agent now sends session metrics, trace data with spans, performance metrics (latency, duration, tokens), and error tracking.

📖 Learn more in Get Started with AgentCore Observability documentation.

📊 Navigate the GenAI Observability Dashboard

Access your agent's observability data through the CloudWatch GenAI dashboard:

1. Open the CloudWatch Console
2. In the left navigation, select GenAI Observability
3. Click the Bedrock AgentCore tab

The main dashboard provides overview metrics: Total Agents, Sessions, Traces, Error Rate, and Throttle Rate. Adjust time filters to view specific periods (last hour, day, week, or custom range).

Yes, I don't have many agents on this account.

🤖 Analyzing AI Agent Performance: Dashboard Deep Dive

The dashboard offers three specialized views for detailed analysis:

Agents View shows all agents (Runtime and non-Runtime hosted), agent-specific performance metrics, session and trace counts per agent, error rates, and visualization graphs. Use this view to monitor operational health and compare performance across agents.

Sessions View displays all sessions across agents, session duration, request counts, session-level errors, user engagement patterns, and session timelines. Filter by Session ID, Agent, Time range, Duration, or Error status. Key insights reveal longest sessions, error locations, typical conversation length, and average requests per session.

Traces View provides detailed trace information, complete span breakdowns, execution timelines (waterfall visualization), tool invocation sequences, and error stack traces. Use this view to debug issues, optimize bottlenecks, and understand complete execution flow.

📖 Learn more in View Observability Data documentation.

🔍 Analyze Trace Details

Click any Trace ID in the Traces View to open detailed visualization. The trace shows a tree structure of operations (spans) with parent-child relationships. A waterfall chart displays the execution timeline, identifying sequential operations, parallel operations, bottlenecks, and error points.

Click any span to see detailed information in three tabs: Attributes (operation metadata, input parameters, output results), Events (significant occurrences with timestamps), and Duration and Status (exact start/end times, total duration, success/error status).

📖 Learn more in Understanding Traces and Spans documentation.

You achieved comprehensive production monitoring without writing instrumentation code. AgentCore Runtime handles this automatically.

📈 Expanding Observability to All AgentCore Services

AgentCore Runtime automatically creates CloudWatch log groups for service-provided logs. However, for Memory, Gateway, Identity, and Built-in Tools resources, you must configure log destinations manually. Follow the Enabling observability for AgentCore runtime, memory, gateway, built-in tools, and identity resources steps.

Since Memory, Gateway, Identity, and Built-in Tools don't appear in the GenAI Observability dashboard, access their metrics directly in CloudWatch:

AgentCore Memory tracks memory lifecycle operations: Latency, Invocations, System/User Errors, Throttles, and Creation Count. Learn more in AgentCore generate memory observability data documentation.

AgentCore Gateway monitors tool invocations: Invocations, Latency, Duration, TargetExecutionTime, Throttles, Errors, and TargetType. Learn more in AgentCore generated gateway observability data documentation.

AgentCore Identity tracks authentication operations: WorkloadAccessTokenFetch metrics, ResourceAccessTokenFetch metrics, and ApiKeyFetch metrics. Learn more in AgentCore generated identity observability data documentation.

Built-in Tools monitors Code Interpreter and Browser: Tool Invocations, Browser TakeOver events, and Resource Usage (CPU, Memory). Learn more in AgentCore Built-in Tools observability documentation

📈 Monitor Key Performance Metrics

AgentCore automatically tracks critical metrics for production monitoring. Learn more in https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-identity-metrics.html

🎉 What You've Learned

In this tutorial, you mastered production observability for AI agents without writing instrumentation code. You activated CloudWatch Transaction Search for automatic observability, understood the observability hierarchy (sessions, traces, spans), navigated the GenAI Observability Dashboard, analyzed trace details with waterfall visualizations, extended observability to Memory, Gateway, Identity, and Built-in Tools, and used OpenTelemetry standards for flexibility.

AgentCore Observability provides zero-code instrumentation for Runtime agents and comprehensive metrics for all services—giving you production-grade visibility into agent behavior, performance, and operational health.

🔗 Complete AgentCore Series

Master production AI agents with this series:

1️⃣ Deploy Production AI Agents with Amazon Bedrock AgentCore in 2 Commands

2️⃣ Bring AI Agents with Long-Term Memory into Production in Minutes

3️⃣ Monitor AI Agents in Production Without New Development ← You are here

❤️ If This Helped You

❤️ Heart it - helps others discover this tutorial

🦄 Unicorn it - if it blew your mind

🔖 Bookmark it - for when you need it later

📤 Share it - with your team or on social media

📚 Resources

• AgentCore Observability Overview
• Get Started with Observability
• Understanding Sessions, Traces, and Spans
• View Observability Data
• Configure Observability
• Amazon Bedrock AgentCore in CloudWatch
• CloudWatch GenAI Observability
• Amazon Bedrock AgentCore Samples
• CloudWatch GenAI Observability Samples
• Observability Best Practices

Happy building! 🚀

¡Gracias!

🇻🇪🇨🇱 Dev.to LinkedIn GitHub Twitter Instagram YouTube

This content originally appeared on DEV Community and was authored by Elizabeth Fuentes L