09 - Metrics, Alerting & Structured Logging

This document defines the observability strategy for the Admin Service, covering business metrics, technical metrics, alert thresholds, and structured logging specifications.

1. Business Metrics

Metric Name	Description	Source	Aggregation
`admin_registrations_total`	Total number of admin registration completions	`AdminAuthService`	Counter (monotonic)
`admin_logins_total`	Total number of successful admin logins	`AdminAuthService`	Counter (monotonic)
`admin_verification_codes_sent`	Verification codes dispatched, labeled by type	`AdminAuthService`	Counter by `type` (register, login)
`admin_verification_failures`	Failed verification code attempts (wrong code, expired, not found)	`AdminAuthService`	Counter (monotonic)
`admin_projects_soft_deleted`	Total number of projects soft-deleted by admins	`AdminProjectService`	Counter (monotonic)
`admin_users_blocked`	Total number of users blocked by admins	`AdminUserService`	Counter (monotonic)
`admin_users_unblocked`	Total number of users unblocked by admins	`AdminUserService`	Counter (monotonic)
`admin_project_status_changes`	Project status transitions performed by admins	`AdminProjectService`	Counter by `from_status`, `to_status`

2. Technical Metrics

Metric Name	Type	Labels	Description
`http_request_duration_seconds`	Histogram	`method`, `route`, `status`	Latency distribution of inbound HTTP requests
`http_requests_total`	Counter	`method`, `route`, `status`	Total count of inbound HTTP requests
`redis_operation_duration_seconds`	Histogram	`operation` (get, set, del)	Latency distribution of Redis commands
`email_send_duration_seconds`	Histogram	`template`	Latency distribution of outbound SMTP email delivery
`email_send_failures_total`	Counter	`template`	Total failed email send attempts
`prisma_query_duration_seconds`	Histogram	`model`, `operation`	Latency distribution of Prisma ORM queries
`jwt_token_generation_duration_seconds`	Histogram		Latency distribution of JWT token signing operations

3. Metrics Source Diagram

graph LR
    AdminAuthService -->|admin_registrations_total| Prometheus
    AdminAuthService -->|admin_logins_total| Prometheus
    AdminAuthService -->|admin_verification_codes_sent| Prometheus
    AdminAuthService -->|admin_verification_failures| Prometheus
    AdminProjectService -->|admin_projects_soft_deleted| Prometheus
    AdminProjectService -->|admin_project_status_changes| Prometheus
    AdminUserService -->|admin_users_blocked| Prometheus
    AdminUserService -->|admin_users_unblocked| Prometheus
    RedisClientService -->|redis_operation_duration_seconds| Prometheus
    EmailSenderService -->|email_send_duration_seconds| Prometheus
    EmailSenderService -->|email_send_failures_total| Prometheus
    HttpInterceptor -->|http_request_duration_seconds| Prometheus
    HttpInterceptor -->|http_requests_total| Prometheus
    PrismaMiddleware -->|prisma_query_duration_seconds| Prometheus
    AdminAuthService -->|jwt_token_generation_duration_seconds| Prometheus

4. Alert Thresholds

Alert Name	Condition	Severity	Action
HighVerificationFailureRate	`rate(admin_verification_failures[5m]) / rate(admin_verification_codes_sent[5m]) > 0.4`	Warning	Investigate potential brute-force attempts; review rate-limiting and cooldown configuration
EmailSendFailureSpike	`rate(email_send_failures_total[5m]) > 5`	Critical	Check SMTP provider connectivity and credentials; verify email service health
SlowDBQueries	`histogram_quantile(0.95, prisma_query_duration_seconds) > 1.0`	Warning	Analyze slow query logs; review missing indexes and query plans
HighAdminAuthErrorRate	`rate(http_requests_total{route=~"/admin/auth/.", status=~"4.."}[5m]) / rate(http_requests_total{route=~"/admin/auth/."}[5m]) > 0.5`	Warning	Review authentication flow for misconfiguration; check for credential stuffing attacks
RedisConnectionFailure	`redis_up == 0` or `rate(redis_operation_duration_seconds_count[1m]) == 0`	Critical	Verify Redis instance health; check network connectivity and restart if necessary

5. Structured Logging Specification

Event	Log Level	Fields	When
`admin.register.initiated`	INFO	`{ email, domain }`	On incoming register request
`admin.register.domain_rejected`	WARN	`{ email, domain, allowedDomains }`	Email domain fails allowlist validation
`admin.register.completed`	INFO	`{ userId, email }`	After successful registration and verification
`admin.login.initiated`	INFO	`{ email }`	On incoming login request
`admin.login.blocked`	WARN	`{ userId, email }`	Blocked user attempts to log in
`admin.verification.failed`	WARN	`{ email, reason }`	Verification code is invalid, expired, or not found
`admin.project.soft_deleted`	INFO	`{ projectId, adminId }`	After a project is soft-deleted
`admin.user.blocked`	INFO	`{ targetUserId, adminId }`	After a user is blocked
`admin.user.unblocked`	INFO	`{ targetUserId, adminId }`	After a user is unblocked
`admin.user.block_admin_rejected`	WARN	`{ targetUserId, adminId }`	Attempt to block another admin is rejected

All log entries are emitted as structured JSON with the following base fields appended automatically: timestamp, correlationId, service: "admin-service".

6. Instrumentation Sequence Diagram

sequenceDiagram
    autonumber
    participant Client
    participant HttpInterceptor
    participant Controller
    participant Service
    participant PrismaMiddleware
    participant Database
    participant RedisClientService
    participant EmailSenderService
    participant Prometheus

    Client->>HttpInterceptor: HTTP Request
    HttpInterceptor->>HttpInterceptor: Start timer for http_request_duration_seconds
    HttpInterceptor->>Controller: Forward request
    Controller->>Controller: Validate DTO
    Controller->>Service: Call service method
    Service->>Service: Emit structured log (e.g. admin.login.initiated)
    Service->>PrismaMiddleware: Execute query
    PrismaMiddleware->>PrismaMiddleware: Start timer for prisma_query_duration_seconds
    PrismaMiddleware->>Database: SQL query
    Database-->>PrismaMiddleware: Query result
    PrismaMiddleware->>Prometheus: Record prisma_query_duration_seconds
    PrismaMiddleware-->>Service: Return data
    Service->>RedisClientService: Redis operation (get/set/del)
    RedisClientService->>RedisClientService: Start timer for redis_operation_duration_seconds
    RedisClientService-->>Service: Redis result
    RedisClientService->>Prometheus: Record redis_operation_duration_seconds
    Service->>EmailSenderService: Send verification email
    EmailSenderService->>EmailSenderService: Start timer for email_send_duration_seconds
    EmailSenderService-->>Service: Email sent / failure
    EmailSenderService->>Prometheus: Record email_send_duration_seconds or email_send_failures_total
    Service->>Prometheus: Increment business metric counter
    Service->>Service: Emit structured log (e.g. admin.login.completed)
    Service-->>Controller: Return result
    Controller-->>HttpInterceptor: Response
    HttpInterceptor->>Prometheus: Record http_request_duration_seconds and http_requests_total
    HttpInterceptor-->>Client: HTTP Response