Skip to main content

ADR 032 — Graph as a first-class data-layer pillar

  • Status: Proposed (2026-08-08) — architecture change proposal (ACP) awaiting human ratification. Program suite: alphaswarm_internal/docs/architecture/graph-data-pillar/; execution tracker: alphaswarm_internal/plans/2026-08-08-graph-data-pillar.md. Sub-decisions are drafted as GDP-ADR-001..010 in the program suite's 10-adrs.md and ratify individually.
  • Authors: Platform team (agent-drafted from three external research reports plus a full workspace conformance survey; see Provenance)
  • Related: ADR 005 (import boundaries), ADR 014 (KB boundary), ADR 029 (catalog contracts, GraphEntityRef, graph-entity URN kind), ADR 018 (canonical domain aggregates — alphaswarm_core.domain.financial_kg), ADR-0021 (data/ingest split, recorded in alphaswarm_data/AGENTS.md), ADR-0018 (DevOps consolidation — service Helm charts live in alphaswarm_devops)

Context​

What exists today (evidence-stated per alphaswarm_internal/REFERENCE_STATE.md)​

The platform already runs a substantial graph estate. It is implemented, production-shaped, and — critically — quadruplicated:

  1. alphaswarm_graph — the SOKG service (the Graph Plane). A multi-tenant, bitemporal Self-Organizing Knowledge Graph on Neo4j: closed taxonomy (111 NodeKind / 75 EdgeKind), store-owned envelope (valid_from/valid_to/tx_from/tx_to, confidence, source_id, extractor, evidence_class, quality_status, payload_hash, run_id), assertion/event governance with quarantine, a rules materialization engine, Graph-RAG retrieval profiles, GNN export with an anti-look-ahead guard, seeders for FIBO/GLEIF/FIGI/GICS, an HTTP surface on :8011 and a 16-tool MCP surface. Backend: Neo4j only, with the Cypher builders (alphaswarm_graph/store/_cypher.py) forming a de-facto second half of the store contract.
  2. alphaswarm_kb — the KB boundary (ADR-014). Owns the cognitive memory layer: PermissionedDataPoint bitemporal envelope, layer composition, HierarchicalRAG + pgvector + Redis retrieval, a Graphiti memory engine, a graph_store adapter kind with its own Neo4jGraphStore (different node labels: :KBDataPoint), SOKG projection handoff models (SourceAssertion/DocumentAssertion → ASSERTS/NORMALIZED_TO/DERIVED_FROM), and a planned but unbuilt alphaswarm_kb.ontology module ("Financial KG node/edge taxonomy + identifier validators").
  3. alphaswarm_learning — a second graph-native service. Neo4j is its primary datastore. It re-implements the bitemporal Cypher envelope, the read-only query guard, and Leiden community analytics, each with a docstring saying it "mirrors alphaswarm_graph" — while importing nothing from it. It carries three uncoordinated graph schemas (graph/schema.py, scholar/graph/schema.py, plus ad-hoc Scholar* labels), two incompatible bitemporal vocabularies (valid_from/valid_to/tx_from/tx_to vs valid_at/invalid_at/created_at/expired_at), and ~30 "TODO: persist to Neo4j" stubs where its Episode → Insight → Reflection → Concept memory hierarchy should land.
  4. alphaswarm_data — the data layer. Ships a graph/ package whose NodeLabel/EdgeType enums are hand-maintained 1:1 string relabels of the SOKG taxonomy (the HTTP-only boundary forbids importing it), a three-backend GraphBackend seam (local JSON / Neo4j / SOKG-seed HTTP), and — the strongest prior art in the estate — a durable transactional outbox that projects the bitemporal securities master into the graph and KB (OutboxTarget.GRAPH | KB | MARKET_CATALOG) with audited repair.

Around these, alphaswarm_core already carries partial pillar contracts (declared): domain/financial_kg.py (FinancialKBAssertion, FinancialLifecycleProjection with referential-integrity validation), contracts/lineage.py (Lineage.sokg_node_id — "the write-back loop"), messaging/outbox.py (pure outbox protocols), observe.SpanKind.GRAPH, and connectivity.graph_url. alphaswarm_ingest already opens its lineage emission span as graph.update. alphaswarm_memory is an empty placeholder repo whose README scope ("monitoring user activity and context, persisting them, retrieval for humans and agents") overlaps alphaswarm_learning's pedagogy memory. alphaswarm_kb_federation is a scaffold; the only place cross-silo graph topology is modeled is SiloTopology.graph_endpoint in alphaswarm_kb.

Why change​

Three external research reports (a conceptual survey of graph data solutions for augmented agents, and two independent deep-research engineering blueprints) were commissioned and reviewed against this codebase. Their consensus, corrected for overclaims and adapted to what already exists here, is that graph storage earns first-class status exactly when a platform needs explicit relationships, changing facts, provenance, and point-in-time correctness — all four of which are existing AlphaSwarm invariants (bitemporal envelopes, evidence classes, look-ahead guards, hash-chained audit). The reports' strongest operational warnings also all apply here:

  • Duplication is the failure mode we are already in. Four parallel taxonomies/envelopes/guards drift independently today. The :KBDataPoint vs :SokgNode label fork inside alphaswarm_kb (its writer and its graph-retrieval channel target different node vocabularies) is the canary.
  • A graph database must not become the tick store, the event bus, or the vector index. AlphaSwarm's polyglot tiering (QuestDB/Iceberg for observations, NATS JetStream for events, pgvector/Redis for embeddings, Postgres for aggregates) is correct and must be named as policy so the pillar cannot absorb it.
  • Backend claims must be benchmarked, not believed (LDBC FinBench for transactional financial graph workloads; tabular baselines before GNNs — the Elliptic dataset's Random Forest result is the canonical caution).
  • Agent access must be typed and bounded, never raw query shells — a posture alphaswarm_graph (cypher guard, write-approval MCP tool) and alphaswarm_kb (read-only guard, LIMIT caps, server-side tenant injection) already implement twice, separately.

Provenance​

This ADR is the human/agent-facing summary of an Architecture Change Proposal assembled from:

  • three uploaded research artifacts, preserved verbatim in alphaswarm_internal/docs/architecture/graph-data-pillar/reference/;
  • a five-track structural survey of alphaswarm_graph, alphaswarm_kb, alphaswarm_kb_federation, alphaswarm_learning, alphaswarm_memory, alphaswarm_core, alphaswarm_data, alphaswarm_index, and alphaswarm_ingest (recorded in the suite's 02-repository-conformance.md);
  • currency verification of external claims (Graphiti backend matrix, Kuzu end-of-life, FalkorDB operations posture, Neo4j GDS licensing tiers, PostgreSQL SQL/PGQ status, LDBC FinBench) in the suite's 01-research-addendum.md.

Decision​

Adopt the Graph Data Pillar program: graph becomes a named, first-class pillar of the data layer — peer to relational, timeseries, object, and vector storage — with one set of contracts, one authoritative service, and one sanctioned write path. Ten sub-decisions (GDP-ADR-001..010, drafted in the program suite) implement it; in summary:

  1. One authority. alphaswarm_graph is the authoritative graph plane service. Other repos consume it over HTTP (services) or via shared contracts (libraries); none re-implements its invariants.
  2. Shared contracts move to alphaswarm_core.graph. Taxonomy identifiers (the SOKG NodeKind/EdgeKind vocabulary), the bitemporal envelope, seed/projection wire shapes (SokgSeedBatch), the citation shape, and the look-ahead guard become a dependency-light contracts package in alphaswarm_core — joining domain/financial_kg.py where partial contracts already live. HTTP-only service seams (ADR-005, ADR-0021) are unchanged; what changes is that the string vocabularies those seams exchange stop being hand-copied.
  3. One bitemporal vocabulary. valid_from/valid_to/tx_from/tx_to with half-open intervals is canonical platform-wide; legacy Graphiti-style fields (valid_at/invalid_at/created_at/expired_at) survive only inside adapters. Research-time reads enforce tx_from <= as_of (no look-ahead) as a contract, not a convention.
  4. A backend-portable store seam. The GraphStore contract splits into a semantic port and a query-dialect adapter so the 755-line Cypher module stops being an implicit half of the contract. Neo4j remains the reference backend everywhere; any second backend (FalkorDB is the leading candidate for low-latency agent memory) must pass the shared contract suite and a FinBench-derived benchmark gate first. RedisGraph (EOL 2025) and Kuzu (upstream archived 2025-10) are rejected.
  5. One physical envelope, reconciled labels. The :SokgNode shared label + store-owned envelope becomes the one physical convention; alphaswarm_kb's :KBDataPoint and alphaswarm_learning's :LearningNode become plane labels layered onto it, closing the label-schema fork.
  6. The outbox is the only cross-service graph write path. The security-master outbox (alphaswarm_data) generalizes into the sanctioned projection backbone (alphaswarm_core.messaging.outbox protocols + idempotent seed batches + staged-assertion governance); direct cross-service writes into the graph are prohibited.
  7. alphaswarm_learning adopts the pillar. It replaces its mirrored envelope/guard/analytics with pillar contracts, converges its three schemas and two temporal vocabularies, and persists the scholar memory hierarchy through the pillar write path — unblocking ~30 TODOs.
  8. alphaswarm_memory is chartered on the pillar. The empty repo becomes the graph-backed user-activity/context memory service (episodes → facts with provenance), delegating storage to the pillar rather than growing a fifth graph stack; its boundary with alphaswarm_learning pedagogy memory is decided before code lands.
  9. Unified agent access governance. Graph MCP tools across alphaswarm_graph and alphaswarm_learning share one tool taxonomy (read / additive-write / privileged-write / admin), bounded traversal and result caps, no raw-query tools for general agents, and shared audit span semantics (SpanKind.GRAPH).
  10. Benchmark-before-believe. Backend changes, GNN adoption, and retrieval-architecture changes gate on a FinBench-derived + domain-query benchmark harness with tabular baselines mandatory for any graph-ML claim.

Consequences​

Positive.

  • Ends four-way drift: one taxonomy, one envelope, one guard, tested for string-level equality where HTTP seams exchange them.
  • The scholar memory hierarchy, KB ontology module, memory service, and federation graph topology all land on one substrate instead of four.
  • Point-in-time correctness (the platform's defense against backtest look-ahead bias) becomes an importable, testable contract.
  • Backend portability becomes real instead of aspirational — today a second backend is blocked by Cypher-locked internals in three repos.
  • Agent surfaces stop diverging: one tool governance model, one citation shape carried by every retrieval channel.

Negative / accepted costs.

  • alphaswarm_core grows a new contracts package that four-plus repos pin; contract changes need cross-repo sequencing discipline (mitigated by additive-only evolution and parity tests, the ADR-029 pattern).
  • Label reconciliation requires a Neo4j data migration in alphaswarm_kb deployments (mitigated: additive dual-labeling first, cutover later).
  • alphaswarm_learning's frozen import ratchet and report-only CI baseline must be re-baselined as re-implementations are deleted.
  • The program touches the SOKG governance surface, which has a sole owning reviewer (alphaswarm_index/subagents/alphaswarm-graph-expert.md) — review capacity is a scheduling constraint, not a veto to bypass.

Explicitly out of scope.

  • No change to the money plane; graph metrics remain advisory signals and the existing money-plane gate stands.
  • No new datastore products (AGENTS rule in alphaswarm_learning stands); the pillar names existing stores.
  • No autonomous-trading authority for any graph-derived signal.

Alternatives considered​

  1. Status quo (four parallel graph stacks). Rejected: drift is already observable (label fork, duplicated guards, two temporal vocabularies); every new consumer copies rather than imports.
  2. Consolidate all graph code into one repo (merge kb/learning graph layers into alphaswarm_graph). Rejected: violates ADR-014's KB boundary and the services' independent deploy cadence; the platform's pattern (ADR-029) is shared contracts + independent services, not mega-repos. Re-open if contract drift proves unmanageable after GD2.
  3. Adopt a different reference backend now (FalkorDB / Memgraph / ArangoDB / Neptune). Rejected for now: Neo4j is live in three services with tested invariants; no measured requirement justifies a migration. The store seam plus benchmark harness keeps this option open; re-open on FinBench-derived evidence of cost/latency need.
  4. Postgres-native graph (Apache AGE today, SQL/PGQ in PostgreSQL 19). Attractive long-term because the platform already operates Postgres/pgvector everywhere, and SQL/PGQ landed in PostgreSQL 19devel (2026-03) — but 19 is not GA and AGE would be a fifth graph dialect. Tracked in the research addendum; re-open when PostgreSQL 19 is GA and a workload fits.
  5. A federated "virtual graph" (query-time composition, no shared contracts). Rejected: does not fix the write-side duplication or temporal-vocabulary drift; Neo4j Composite (federated_read) already covers read-side federation for silos.
  6. Put shared graph contracts in alphaswarm_catalog instead of alphaswarm_core. Rejected: catalog owns data-object identity (ADR-029's GraphEntityRef stays); the graph envelope/taxonomy is runtime domain vocabulary, and alphaswarm_core.domain.financial_kg (ADR 018) already set the precedent. Boundary detailed in GDP-ADR-002.

Implementation status​

Nothing is implemented under this ADR yet; it is a proposal. The program suite (alphaswarm_internal/docs/architecture/graph-data-pillar/) carries:

  • 00-unified-blueprint.md — the combined engineering blueprint;
  • 01-research-addendum.md — currency verification and corrections;
  • 02-repository-conformance.md — the design→code map and open decisions;
  • 10-adrs.md — GDP-ADR-001..010 (Proposed, ratify individually);
  • 20-implementation-guide.md — exhaustive file-level mapping;

and the execution tracker alphaswarm_internal/plans/2026-08-08-graph-data-pillar.md defines phases GD0–GD6 with gates G0–G6 ("gates, not dates"), a verification matrix, and rejected alternatives with re-open conditions.