Observability stack

AlphaSwarm's centralized telemetry plane lives in the alphaswarm-observability namespace. The canonical, CRD-free data flow (see observability/README.md) is VictoriaMetrics-based: vmagent is the only Prometheus-style scraper, VictoriaMetrics is the sole metrics store, vmalert owns recording/alert rules, and a single raw OTel gateway (no DaemonSet agent) fans traces and logs out to their destinations. The older kube-prometheus-stack/, opentelemetry-operator/, and opentelemetry-collector-gateway/ folders still exist but are explicitly unselected rollback/reference assets only.

Components

Component	Folder	Notes
VictoriaMetrics + vmagent + vmalert	observability/victoriametrics/, observability/vmagent/, observability/vmalert/	canonical metrics authority — replaces Prometheus/kube-prometheus-stack
OTel gateway	observability/otel-gateway/	canonical single OTLP ingress (no DaemonSet agent); otel-collector Service name kept as a compat alias
Jaeger + OpenSearch	observability/jaeger/, observability/opensearch/	trace storage/query (traces also land in Phoenix for AI spans)
Loki	observability/loki/	log storage
Phoenix	observability/phoenix/	AI/agent trace explorer
kube-prometheus-stack, OpenTelemetry Operator, opentelemetry-collector-gateway	(legacy folders, same parent dir)	unselected rollback/reference assets only

Routing rule (gateway)

The tail_sampling processor in otel-gateway/configmap.yaml inspects every span and keeps it when any policy matches:

• errors — status_code == ERROR.
• ai_traces — attributes["openinference.span.kind"] is set.
• llm_traces — attributes["llm.model_name"] is set.
• agent_traces — attributes["agent.name"] is set.
• probabilistic_baseline — a flat 1% of everything else.

There's a single traces pipeline (not split into traces/ai / traces/infra): every span that survives tail sampling is exported to both otlp/jaeger and otlp/phoenix.

DataMCP tools

Tool	Surface
data.observability.prometheus.query	Instant PromQL.
data.observability.prometheus.query_range	Range PromQL.
data.observability.prometheus.list_alerts	Active alerts.
data.observability.grafana.list_dashboards	Dashboard catalog.
data.observability.grafana.export_dashboard	Dashboard JSON.
data.observability.phoenix.list_projects	Phoenix projects.
data.observability.phoenix.get_trace	LLM / agent trace.
data.observability.phoenix.annotate_span	Write evaluator verdict.

Frontend

• /observability in alphaswarm_client (the active Vite operator UI) — observability overview page.