This content originally appeared on DEV Community and was authored by Devin Rosario
Your checkout API throws errors randomly. Users complain, logs show nothing useful, and every service claims it's working fine.
Welcome to distributed systems debugging hell.
AWS X-Ray traces every request across your entire application stack, showing exactly where milliseconds disappear and which service actually broke. The September 2025 update added adaptive sampling that automatically captures more traces during anomalies while keeping costs down during normal operations.
Debugging stops being guesswork. You see the complete story.
What X-Ray Actually Tracks in Your Application
X-Ray follows requests as they move through services, databases, external APIs, and queues. Each component adds timing data and metadata. The service stitches everything into traces—complete journeys from user click to final response.
Think of it as a GPS tracker for your requests. A user searches for products. The request hits API Gateway, routes to your search service, queries Elasticsearch, fetches product details from PostgreSQL, checks inventory via another microservice, and renders results. X-Ray captures every step with millisecond precision.
The service map generates automatically from trace data, showing boxes for services and lines for connections. Colors indicate health: green for normal operation, orange for elevated errors, red for failures. Your architecture becomes visible without manual diagramming.
X-Ray runs with under 2% performance overhead in production. The SDK instruments your code with minimal changes. The daemon forwards traces to AWS for storage and analysis.
September 2025 Game-Changer: Adaptive Sampling
AWS released adaptive sampling for X-Ray in September 2025, solving a problem that plagued DevOps teams for years: balancing trace capture with cost control.
Traditional fixed sampling captured a percentage of all requests. Set sampling at 10% and you caught 10% of errors. Miss a critical error because it fell in the unlucky 90%. Increase sampling to catch everything and watch your observability bills explode.
Adaptive sampling introduces two approaches working independently or combined:
Sampling Boost temporarily increases trace capture rates when anomalies appear. Your normal 5% sampling jumps to 50% during error spikes, capturing complete traces for root cause analysis. When conditions normalize, sampling drops back automatically.
Anomaly Span Capture ensures error-related spans always get captured even when full traces aren't sampled. You never miss critical failure data regardless of sampling rates.
Teams using adaptive sampling report 40-60% reduced mean time to resolution (MTTR) for production incidents while maintaining 30% lower observability costs compared to fixed high-rate sampling.
Setting Up X-Ray in Modern Applications
Integration varies by platform but stays simple across all AWS services.
Lambda Functions enable X-Ray with a checkbox in console or one line in CloudFormation. The X-Ray SDK initializes automatically. No daemon required. Traces show cold start times separately from execution duration, making performance optimization straightforward.
ECS and Fargate run the X-Ray daemon as a sidecar container. Your application container sends traces to localhost, the daemon forwards to AWS. Container orchestration handles everything.
EC2 Instances need daemon installation and IAM permissions. The daemon runs as a background service collecting traces from applications:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"xray:PutTraceSegments",
"xray:PutTelemetryRecords"
],
"Resource": "*"
}]
}
Node.js Application Setup:
const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));
const http = AWSXRay.captureHTTPs(require('https'));
const express = require('express');
const app = express();
// Automatic request tracing
app.use(AWSXRay.express.openSegment('OrderService'));
app.post('/api/orders', async (req, res) => {
// Custom subsegment for business logic
const segment = AWSXRay.getSegment();
const subsegment = segment.addNewSubsegment('ProcessOrder');
try {
const order = await processOrderLogic(req.body);
// Add searchable metadata
subsegment.addAnnotation('order_id', order.id);
subsegment.addAnnotation('customer_tier', order.customerTier);
subsegment.addMetadata('order_details', order);
res.json({ success: true, orderId: order.id });
} catch (error) {
subsegment.addError(error);
throw error;
} finally {
subsegment.close();
}
});
app.use(AWSXRay.express.closeSegment());
Python Flask Integration:
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware
from flask import Flask, jsonify
import boto3
# Capture AWS SDK calls automatically
from aws_xray_sdk.core import patch_all
patch_all()
app = Flask(__name__)
xray_recorder.configure(
service='PaymentService',
sampling=True,
context_missing='LOG_ERROR'
)
XRayMiddleware(app, xray_recorder)
@app.route('/api/process-payment', methods=['POST'])
def process_payment():
# Annotations for filtering traces
xray_recorder.put_annotation('payment_method', 'stripe')
xray_recorder.put_annotation('amount_usd', 99.00)
# Detailed metadata (not searchable but visible in trace)
xray_recorder.put_metadata('request_details', {
'currency': 'USD',
'customer_id': 'cus_123',
'items': 5
})
# This Stripe call gets traced automatically
result = process_stripe_payment()
return jsonify({'status': 'success'})
The SDK automatically captures HTTP calls, database queries, and AWS service interactions without additional instrumentation.
Understanding Trace Hierarchy and Segments
X-Ray organizes data in a three-level hierarchy:
Traces represent complete request journeys from entry to exit. Each trace gets a unique ID tracking the entire flow across services.
Segments represent individual services or components. Your API Gateway creates a segment. Your Lambda function creates another. Your DynamoDB query creates a third.
Subsegments break down what happens inside each service. Database calls, external API requests, business logic blocks—all become subsegments showing exact timing.
A typical e-commerce checkout trace structure:
Trace ID: 1-6733a2f0-4e8b9c7d2a1f3e5d6c8b7a9f
│
├─ API Gateway Segment (12ms)
│
├─ Checkout Lambda Segment (267ms)
│ │
│ ├─ ValidateCart Subsegment (18ms)
│ │ └─ DynamoDB Query (14ms)
│ │
│ ├─ CheckInventory Subsegment (95ms)
│ │ ├─ InventoryService HTTP Call (78ms)
│ │ └─ Cache Update (12ms)
│ │
│ ├─ ProcessPayment Subsegment (128ms)
│ │ ├─ Stripe API Call (110ms)
│ │ └─ PostgreSQL Insert (15ms)
│ │
│ └─ SendConfirmation Subsegment (26ms)
│ └─ SES API Call (23ms)
│
└─ CloudWatch Logs Upload (5ms)
Every subsegment includes precise timestamps, status codes, and error details. You see not just what failed, but exactly when and why.
Automatic Database and API Call Tracking
X-Ray SDKs instrument major databases automatically when using supported clients. PostgreSQL, MySQL, DynamoDB, MongoDB, Redis—all get tracked without explicit instrumentation.
Node.js with PostgreSQL:
const AWSXRay = require('aws-xray-sdk-core');
const pg = AWSXRay.capturePostgres(require('pg'));
const pool = new pg.Pool({
host: 'prod-db.example.com',
database: 'orders',
max: 20
});
// This query automatically appears in X-Ray traces
const result = await pool.query(
'SELECT * FROM orders WHERE user_id = $1 AND status = $2',
[userId, 'pending']
);
The trace shows query text (with parameters sanitized), execution time, and row counts. Slow queries become immediately visible.
Python with SQLAlchemy:
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.ext.sqlalchemy.query import XRaySessionMaker
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
engine = create_engine('postgresql://user:pass@host/db')
Session = XRaySessionMaker(sessionmaker(bind=engine))
# All database operations through this session get traced
session = Session()
orders = session.query(Order).filter(
Order.status == 'pending',
Order.created_at > datetime.now() - timedelta(hours=24)
).all()
External HTTP calls need explicit capture but the API stays simple:
const AWSXRay = require('aws-xray-sdk-core');
const https = AWSXRay.captureHTTPs(require('https'));
// Stripe API calls now traced automatically
https.request({
hostname: 'api.stripe.com',
path: '/v1/charges',
method: 'POST'
}, (response) => {
// Response handling
});
Teams working with mobile app development in Virginia often integrate multiple backend services. X-Ray traces show exactly how backend API performance impacts mobile user experience, making optimization priorities crystal clear.
Finding Performance Bottlenecks with Service Maps
The service map updates in real-time, showing your architecture as it runs. Deploy a new microservice and it appears automatically. Remove a dependency and the connection disappears.
Click any service to see aggregated metrics:
- Request volume: Requests per second, minute, or hour
- Response time distribution: p50, p90, p95, p99 latencies
- Error rates: Percentage of failed requests
- Connections: Upstream and downstream dependencies
Filter traces by response time to find slow requests: "Show all traces where response time > 3 seconds AND service = checkout." X-Ray returns matching traces sorted by slowest first.
Response time distribution tells the real performance story:
Payment Processing Service
────────────────────────────────
p50: 145ms (half of requests faster)
p75: 280ms (75% faster)
p90: 520ms (90% faster)
p95: 890ms (95% faster)
p99: 2,400ms (outliers affecting 1%)
Focus optimization on p95 and p99 latencies. These outliers frustrate users and indicate underlying issues. A service with p50 of 100ms but p99 of 5 seconds needs investigation—something breaks occasionally.
Custom subsegments isolate specific code blocks for analysis:
# Wrap expensive operations in subsegments
@xray_recorder.capture('RecommendationEngine')
def generate_recommendations(user_id):
# This entire function becomes a subsegment
recommendations = complex_ml_algorithm(user_id)
return recommendations
# Or manually create subsegments
subsegment = xray_recorder.begin_subsegment('ImageProcessing')
try:
processed_image = resize_and_optimize(image)
subsegment.put_annotation('image_size_mb', image.size / 1024 / 1024)
finally:
xray_recorder.end_subsegment()
The trace timeline shows subsegments visually, making performance hotspots obvious at a glance.
Error Detection and Root Cause Analysis
X-Ray captures exceptions automatically. When code throws an error, the corresponding segment marks as failed and includes the stack trace.
The service map shows error rates per service. A service displaying 12% errors needs immediate attention. Click through to see which specific errors occur most frequently.
The trace timeline reveals error location precisely:
CheckoutService (HTTP 500 - Internal Server Error)
├─ ValidateCart (OK - 18ms)
├─ CheckInventory (OK - 95ms)
├─ ProcessPayment (ERROR - stripe_api_timeout)
│ └─ Stripe API Call (timeout after 30s)
└─ SendConfirmation (skipped - previous error)
You see immediately that Stripe timed out. Check Stripe's status page for outages. Review timeout settings. Add retry logic with exponential backoff. The root cause becomes obvious.
Sampling rules control costs while capturing every error. Configure X-Ray to sample 100% of errors but only 5% of successful requests:
{
"version": 2,
"rules": [
{
"description": "Capture all errors",
"service_name": "*",
"http_method": "*",
"url_path": "*",
"fixed_target": 0,
"rate": 1.0,
"reservoir_size": 1,
"host": "*",
"service_type": "*",
"resource_arn": "*",
"attributes": {}
},
{
"description": "Sample 5% of successful requests",
"service_name": "*",
"http_method": "*",
"url_path": "*",
"fixed_target": 1,
"rate": 0.05
}
],
"default": {
"fixed_target": 1,
"rate": 0.05
}
}
Critical paths get 100% sampling, background jobs sample at 2%, administrative endpoints at 1%. You control exactly what X-Ray captures based on business importance.
AWS Service Integration: Beyond Custom Code
X-Ray traces requests across AWS services automatically when enabled:
API Gateway traces every request with X-Ray active tracing enabled. See how long API Gateway processing takes versus backend Lambda execution.
Lambda shows cold start times separately from warm execution, revealing optimization opportunities. A function taking 800ms with 600ms cold start needs provisioned concurrency.
Step Functions traces workflow execution across multiple Lambda invocations, showing parallel execution and state machine transitions.
SNS/SQS captures message publishing, queue wait times, and processing duration. Async workflows become visible—see when messages get published, how long they wait in queues, and processing time.
DynamoDB, S3, SES operations appear automatically when using AWS SDK. Storage access patterns and email delivery timing become transparent.
CloudWatch integration connects logs with traces. Click a trace to see related log entries filtered automatically. Debug errors by reading logs in the context of the complete request flow.
Cost Management and Sampling Strategies
X-Ray pricing runs $5 per million traces recorded plus $0.50 per million traces retrieved. A moderate application generating 10 million traces monthly costs $50 for recording plus retrieval fees.
Use sampling strategically:
Development environments: 100% sampling for complete visibility
Staging: 50% sampling for realistic testing
Production: 5-10% sampling for cost control, 100% for errors
The new adaptive sampling in September 2025 automatically adjusts rates within your defined limits. Set minimum 5% and maximum 50% sampling. During normal operations, X-Ray samples at 5%. When error rates spike, sampling increases to 50% automatically, capturing detailed traces for debugging. When conditions normalize, sampling drops back to 5%.
Teams report 30% lower costs using adaptive sampling versus fixed 20% sampling while catching more production issues.
Actionable Takeaways:
- Enable X-Ray active tracing in API Gateway and Lambda for instant distributed tracing
- Use annotations for searchable business data (user_id, customer_tier) and metadata for detailed context
- Configure adaptive sampling with 5% minimum, 50% maximum to balance cost and error detection
- Create custom subsegments around expensive operations to isolate performance bottlenecks
- Filter traces by "response time > 2s AND error = true" to find critical issues fast
- Instrument database clients with X-Ray wrappers for automatic query tracing
- Set sampling rules to capture 100% of checkout/payment flows, 5% of read-only endpoints
- Export trace data to S3 for long-term analysis and custom dashboards using X-Ray API
- Use service maps during incident response to identify which upstream dependency failed
Which distributed tracing challenges slow your debugging workflow most?
This content originally appeared on DEV Community and was authored by Devin Rosario
Devin Rosario | Sciencx (2025-10-25T09:08:21+00:00) Complete Guide to AWS X-Ray Tracing. Retrieved from https://www.scien.cx/2025/10/25/complete-guide-to-aws-x-ray-tracing/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.