Monitoring and Alerts

Monitoring and alerting system for the Plus261 platform. The system uses Prometheus, Loki, and Tempo for comprehensive observability.

What You’ll Learn

This guide will help you understand:

✅ Monitoring architecture
✅ Metrics collection with Prometheus
✅ Log aggregation with Loki
✅ Distributed tracing with Tempo
✅ Alert configuration
✅ Dashboard visualization

Overview

Monitoring includes:

Component	Technology	Purpose
Metrics	Prometheus	Performance metrics
Logs	Loki	Structured log aggregation
Tracing	Tempo	Distributed request tracing
Alerts	Alertmanager	Configurable alerts
Dashboards	Grafana	Data visualization

Metrics

Prometheus metrics monitor performance:

Key Metrics

Metric Type	Examples	Purpose
Counter	Request count, error count	Track totals
Gauge	Active connections, queue size	Current state
Histogram	Request duration, response size	Distribution analysis
Summary	Percentiles, quantiles	Statistical summaries

Example Metrics

http_requests_total - Total HTTP requests
http_request_duration_seconds - Request duration
database_connections_active - Active database connections
graphql_queries_total - Total GraphQL queries

Learn more: API - Metrics

Logs

Structured logs facilitate analysis:

Log Levels

Level	Use Case	Example
DEBUG	Detailed debugging	Variable values, flow tracking
INFO	General information	Request processing, state changes
WARN	Warning conditions	Deprecated features, fallbacks
ERROR	Error conditions	Exceptions, failures

Log Structure

logger.info('User signed in', {
  userId: user.id,
  email: user.email,
  timestamp: new Date(),
  traceId: context.traceId,
});

Learn more: API - Logging

Tracing

Distributed tracing follows requests:

Trace Components

Spans - Individual operations
Traces - Complete request flows
Context Propagation - Request correlation

Example Trace

Request → API → Database → External Service
  ↓         ↓        ↓            ↓
Span 1   Span 2   Span 3      Span 4

Learn more: API - Tracing

Alerts

Alerts notify in case of problems:

Alert Types

Alert Type	Trigger	Example
Critical	System down	API unavailable
Warning	Degraded performance	High error rate
Info	Notable events	Deployment completed

Alert Configuration

groups:
  - name: api_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_errors_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High error rate detected'

Dashboards

Grafana dashboards visualize data:

Dashboard Types

System Overview - Overall system health
API Performance - Request metrics
Database Performance - Query performance
Error Analysis - Error tracking
User Activity - User behavior metrics

Learn more: API - Observability

Quick Reference

Topic	Documentation
📊 Metrics	API - Metrics
📝 Logging	API - Logging
🔍 Tracing	API - Tracing
📈 Observability	API - Observability

Next Steps

Continue learning:

📊 API - Observability - Complete observability guide
📈 API - Metrics - Metrics collection
📝 API - Logging - Logging system
🔍 API - Tracing - Distributed tracing

Observability - API observability
Deployment - Deployment strategies
Architecture Overview - System architecture
Scalability - Scalability considerations