Observability, Drift Detection & MLOps
Prometheus instrumentation, Wasserstein drift alerts, model registry gating, and CI/CD deployment
In This Component
A deployed model that isn’t monitored is a liability. This component covers the full observability stack: structured logging with correlation IDs, Prometheus metrics, statistical drift detection, and the CI/CD pipeline that gates deployments behind a model registry.
Structured Logging
Every request generates structured logs with a correlation_id propagated through contextvars.ContextVar. Log records include extra context fields serialized as JSON — making logs machine-parseable and traceable across the full request lifecycle.
Prometheus Metrics (GET /metrics)
| Metric | Type | What it measures |
|---|---|---|
quote_requests_total | Counter | Volume and outcomes (quoted, rejected, failed) |
quote_latency_seconds | Histogram | End-to-end DLA evaluation time |
event_ingestion_total | Counter | B2B/IoT events received and deduplicated |
outbox_events_total | Counter | CRM delivery: delivered, retried, dead-lettered |
drift_alerts_total | Counter | Wasserstein distance threshold crossings |
Wasserstein Drift Detection
Each incoming event distribution is compared against a seeded reference baseline using Wasserstein distance (Earth Mover’s Distance). When drift exceeds a configured threshold, a structured warning is logged and a counter increments — surfacing distribution shift before it silently degrades quote quality.
Model Registry Gating
The DLAPolicyEngine will not initialize in staging or production without a registry-approved ModelRelease. The release object carries five version vectors stamped into every DecisionRecord, creating a complete lineage chain from quote to training data to deployed model.
CI/CD: Commit → Cloud Run
git push → lint + typecheck + pytest → docker build (commit-SHA tag) → Cloud Run deploy (private, 1 vCPU, 1 GiB, max 1 instance) → readiness probe → human promotion gate → production. If readiness fails, the previous revision remains live.