?? Learning Goal: Understand how to implement comprehensive observability in .NET applications using OpenTelemetry, complete with metrics, tracing, and beautiful dashboards.

This project demonstrates production-ready observability for a .NET Web API with:

• ?? Metrics - Performance and usage data per API endpoint
• ?? Traces - Request flows and timing analysis
• ?? Dashboards - Beautiful Grafana visualizations
• ?? Easy Setup - Everything runs with one command using .NET Aspire

OpenTelemetry is the industry standard for collecting observability data from applications:

• Request rates - How many requests per second?
• Response times - How fast is each endpoint?
• Error rates - What percentage of requests are failing?
• Resource usage - CPU, memory, garbage collection

• Request flow - Path a request takes through your system
• Timing breakdown - How long each step takes
• Error tracking - Exactly where and when things fail
• Distributed tracing - Follow requests across multiple services

• Not implemented in this example, but easily added

???????????????????    ?? gRPC     ????????????????????????    ?? HTTP    ???????????????    ?? Queries    ???????????????
?   Your .NET API ? ???????????????? OpenTelemetry        ? ??????????????? Prometheus  ? ??????????????????   Grafana   ?
?                 ?                ? Collector            ?               ? (Metrics    ?                  ? (Beautiful  ?
? • Custom Spans  ?                ?                      ?               ?  Storage)   ?                  ?  Dashboard) ?
? • Route Metrics ?                ? • Receives Telemetry ?               ?             ?                  ?             ?
? • Error Tracking?                ? • Processes Data     ?               ???????????????                  ???????????????
???????????????????                ? • Exports to Backend?               
                                   ????????????????????????               

1. Your API generates telemetry data (metrics + traces)
2. OpenTelemetry Collector receives and processes the data
3. Prometheus stores metrics in a time-series database
4. Grafana creates beautiful dashboards and alerts

??? Program.cs                     # ?? Web API with comprehensive OpenTelemetry setup
??? AppHost/
?   ??? AppHost.cs                 # ???  Aspire orchestration (starts all containers)
?   ??? otel-config.yml           # ??  OpenTelemetry Collector configuration  
?   ??? prometheus.yml             # ?? Prometheus scraping configuration
?   ??? grafana-datasources.yml   # ?? Grafana data source configuration
?   ??? dashboard-provider.yml     # ?? Dashboard auto-loading configuration
?   ??? api-dashboard.json         # ?? Pre-built production dashboard
??? docker-compose.reference.yml   # ?? Production deployment reference
??? README.md                      # ?? This comprehensive guide

• ? .NET 10 SDK - Download here
• ? Docker Desktop - Download here

# Clone and navigate to the project
git clone <your-repo>
cd dotnet-examples

# Start the complete observability stack
cd AppHost
dotnet run

?? That's it! Wait 30-60 seconds for all containers to start.

• ?? Your API: Check the Aspire dashboard for the dynamic port
• ??? Aspire Dashboard: http://localhost:15888 (Container orchestration)
• ?? Prometheus: http://localhost:9090 (Raw metrics)
• ?? Grafana: http://localhost:3000 (Beautiful dashboards - login: admin/admin)

1. Go to Grafana: http://localhost:3000
2. Login: admin/admin (you'll be prompted to change this)
3. Find the dashboard: Look for ".NET API Dashboard" in the side menu

You'll see a production-ready dashboard with:

• Requests/sec - Real-time throughput
• P95 Response Time - 95th percentile latency
• Error Rate - Percentage of failed requests
• Active Requests - Current concurrent load

• Request Rate by Route - Traffic per API endpoint (/api/weather, /api/error, etc.)
• Status Code Distribution - Visual breakdown of HTTP response codes

• Total Requests by Route - Cumulative usage per endpoint
• Overall Response Time Percentiles - P95, P50, and Average latencies
• .NET GC Collections/sec - Runtime performance monitoring
• P95 Response Time by Route - Performance breakdown per endpoint

Hit different endpoints to see real-time data:

# Replace [port] with your API's actual port from Aspire dashboard
curl http://localhost:[port]/api/weather          # ? Normal business logic
curl http://localhost:[port]/api/test-telemetry   # ?? Custom tracing example
curl http://localhost:[port]/api/external-call    # ?? HTTP client tracing
curl http://localhost:[port]/api/error            # ? Error tracing
curl http://localhost:[port]/debug/telemetry      # ?? Debug information

?? Watch the dashboard update in real-time! Each endpoint will appear as separate lines in the charts.

1. Visit /api/test-telemetry multiple times
2. Watch the "Request Rate by Route" chart in Grafana
3. Observe how the /api/test-telemetry line increases

?? Learning: You're seeing custom metrics that track each endpoint individually.

1. Visit /api/external-call (this makes an HTTP call to an external service)
2. Go to the Aspire dashboard traces section
3. Find traces that show the full request chain

?? Learning: You'll see how your request flows through multiple services with timing for each step.

1. Visit /api/error several times (this throws exceptions)
2. Watch the "Error Rate" stat turn red in Grafana
3. Check the "Status Code Distribution" for 500 errors

?? Learning: See how errors are automatically captured and visualized.

1. Generate mixed traffic to different endpoints
2. Compare response times in "P95 Response Time by Route"
3. Identify which endpoints are fastest/slowest

?? Learning: Understand how to identify performance bottlenecks per endpoint.

Our middleware captures detailed metrics for each endpoint:

// This runs for every HTTP request
requestCounter.Add(1, new TagList
{
    { "route", "/api/weather" },      // Which endpoint
    { "method", "GET" },              // HTTP method  
    { "status_code", "200" }          // Response status
});

Result: Grafana can show separate lines for /api/weather, /api/error, etc.

Business logic gets detailed tracing:

using var activity = activitySource.StartActivity("GenerateWeatherForecast");
activity?.SetTag("operation.type", "weather_forecast");
activity?.SetTag("forecast.count", forecasts.Length);

Result: Distributed traces show your custom business operations with timing.

The Grafana dashboard uses PromQL queries like:

# Request rate per endpoint
sum by(route) (rate(http_requests_total[1m]))

# P95 response time
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[1m])) by (le))

Result: Rich visualizations that update in real-time.

// 1. Define a new metric
var businessMetric = customMeter.CreateCounter<long>("business_operations_total");

// 2. Record data
businessMetric.Add(1, new TagList { { "operation", "user_signup" } });

// 3. Query in Grafana: business_operations_total

// Create detailed tracing for any operation
using var activity = activitySource.StartActivity("ProcessPayment");
activity?.SetTag("payment.amount", 99.99);
activity?.SetTag("payment.method", "credit_card");
// Your payment logic here...

1. Go to Grafana ? Create ? Dashboard
2. Add a panel with a PromQL query
3. Example query: sum by(route) (rate(http_requests_total[1m]))
4. Customize visualization, thresholds, and alerts

For production, see docker-compose.reference.yml for a complete setup including:

• ?? Security - Proper authentication and TLS
• ?? Persistence - Data storage volumes
• ?? Scaling - Multiple collector instances
• ?? Alerting - Automated monitoring alerts

After completing this example, you'll understand:

? How to implement OpenTelemetry in .NET applications
? Custom metrics vs automatic metrics and when to use each
? Distributed tracing for request flow analysis
? Dashboard creation for production monitoring
? Performance monitoring techniques for APIs
? Error tracking and alerting best practices
? Production deployment considerations

# Check if all containers are running
docker ps

# Restart if needed
cd AppHost && dotnet run

1. Check /debug/collector endpoint
2. Verify OpenTelemetry Collector is reachable
3. Wait 30-60 seconds for data to appear

• URL: http://localhost:3000
• Login: admin/admin
• Wait for container to fully start (check Docker Desktop)

1. ?? Experiment with custom metrics for your business logic
2. ?? Create additional dashboard panels for specific KPIs
3. ?? Set up alerting rules for critical thresholds
4. ?? Explore distributed tracing across multiple services
5. ?? Read OpenTelemetry documentation for advanced concepts

?? Congratulations! You now have a complete, production-ready observability setup that you can adapt for any .NET application. Happy monitoring! ???