Saltar al contenido principal

ADR 029 — Catalog-first data object model

  • Status: Accepted (2026-07-19) — first slice implemented (contracts + monolith projection + first MCP consumer; see Implementation status). Roadmap: infra program §11.5 Workstream 5B ("Extract the catalog aggregate, E7/S2") and alphaswarm_internal/TODO.md §23.1.
  • Authors: Platform team
  • Related: ADR 019 (one component registry / URN index), infra program §11.5 5A (strangler artifacts, /metadata/catalog CANDIDATE route), TODO §23.2 (engine-noun internalization), TODO §23.3 (one physical-registration status)

Context​

Catalog concerns are triple-homed:

  1. the monolith relational catalog — DatasetCatalog/DatasetVersion ORM (70 importing files) + MetadataCatalogService + the /metadata/catalog/*, /datasets/* routes and data.catalog.* MCP tools;
  2. the satellite in-memory alphaswarm_data.DataCatalog (DatasetDescriptor/InstrumentDescriptor provider index) — isolated, zero shared shapes with the monolith;
  3. the physical catalogs (Polaris / Gravitino / Iceberg-REST / Trino), leaking engine nouns through customer-visible surfaces (iceberg_identifier fields, /data/polaris/catalogs, …).

There was no dependency-light, typed object model that UI / CLI / MCP / agents / satellites could share — every consumer either imported the ORM directly or defined its own dataset shape, and "is this dataset registered?" had per-engine answers instead of one.

Decision​

Populate the data half of alphaswarm_catalog (alphaswarm_catalog.data) as contracts ONLY, mirroring the dependency posture of alphaswarm_core:

  1. Typed frozen wire contracts — Dataset, DatasetVersion, DataSource (alias Feed), Instrument (field-compatible with the satellite's InstrumentDescriptor, canonical open identifiers included), FeatureSet, Corpus, GraphEntityRef, ArtifactRef, ColumnDoc. All frozen=True + extra="forbid"; additive evolution only.
  2. One URN scheme, second kind namespace — data objects share the PCG's urn:alphaswarm:<kind>:<environment>:<local-id> format (same URN_PREFIX, never forked) with their own closed kind set (DATA_URN_KINDS: dataset, dataset-version, feed, instrument, feature-set, corpus, graph-entity, artifact — disjoint from PCG_URN_KINDS, enforced by test). Both namespaces register into the ADR-019 component registry (register_data_catalog()).
  3. Never engine nouns — the object surface carries no iceberg/polaris/gravitino fields. Physical placement surfaces only through the single PhysicalRegistration aspect (§23.3): one registered|pending|drifted|orphaned state + a per-engine evidence map, maintained by one reconciler on the implementation side.
  4. Pinned enums — MedallionLayer (bronze/silver/gold, migration 0027), CurationState (uncurated/certified/deprecated) and SupportStatus (supported/experimental/retired) (migration 0108), RecordType (bar/quote/trade/reference/fundamental/macro — satellite vocabulary). Values are contract-locked to the shipped immutable migrations.
  5. DatasetRef resolution semantics — exactly one addressing form per ref (urn → dataset_id → provider-qualified name), fail-closed on empty/mixed refs and on name ambiguity. The resolver implementation stays with the monolith service; the contract pins only the semantics.
  6. Ownership stays put — the monolith keeps catalog persistence + service implementation and projects rows onto the contract (alphaswarm/services/catalog_contract.py, guarded import so the monolith boots without the sibling); the satellite DataCatalog becomes a client/projection of the same shapes; physical catalogs remain implementation detail behind the aspect. No persistence or governance moves into the contract package.

Consequences​

Positive

  • UI/CLI/MCP/satellites can import one catalog contract; the "every consumer forks the dataset shape" class of drift ends at the contract boundary.
  • Engine-noun exposure becomes a projection concern: the §23.2 route internalization and the eventual /metadata/catalog strangler flip (5A CANDIDATE → extracted) now have a stable target shape.
  • One registration verdict (§23.3) with per-engine evidence replaces per-engine status fields.

Negative / risks

  • Dual shapes exist during the strangler window (legacy ORM-shaped payloads + additive contract fields); consumers must not mix them. Mitigated by additive-only enrichment (existing fields byte-identical) and projection tests pinning both.
  • The PhysicalRegistration derivation is heuristic until the one reconciler lands; states beyond registered|pending require it.

Implementation status​

  • 2026-07-19 (first slice, §11.5 5B): alphaswarm_catalog.data landed with contract tests (enum pinning, URN round-trip + disjointness, frozen/forbid, DatasetRef fail-closed, ADR-019 idempotent registration); monolith projection alphaswarm/services/catalog_contract.py (dataset_from_row / dataset_payload_from_row / dataset_urn_for_row / physical_registration_from_row, typed CatalogContractUnavailableError when the package is absent); first consumer data.catalog.browse MCP tool emits additive urn + physical_registration fields.
  • 2026-07-19 (second slice): /metadata/catalog/* route payloads carry the contract — MetadataDataset + MetadataDatasetResponse gained additive urn + physical_registration fields populated by the guarded _contract_enrichment in MetadataCatalogService._row_to_dataset (null when the package is absent). Satellite projection landed: alphaswarm_data/catalog/contract_projection.py (optional-convergence pattern; dataset_to_contract / instrument_to_contract / catalog_to_contract_datasets; provider ids with : map to ~ URN local-ids with the verbatim id in metadata["provider_dataset_id"]; catalog extra on the package).
  • 2026-07-19 (third slice): the ONE physical-registration reconciler landed — migration 0139 (physical_registration_json + _checked_at on dataset_catalogs), alphaswarm/data/catalog/registration_reconciler.py (pure verdicts: registered|pending|drifted, per-engine evidence, fail-honest on listing outages, orphan sweep in the task summary), beat task catalog-registration-reconcile (ALPHASWARM_CATALOG_REGISTRATION_RECONCILE_* knobs), and reads prefer the persisted verdict. §23.2 shadow phase: the 11 engine-noun read routes carry shadow logging + Deprecation/Link headers; operator mirrors live at /data/_diagnostics/* behind admin:iceberg + step-up, schema-excluded. Consumer cutover extended: data.catalog.discover_datasets + data.catalog.get_schema MCP tools and the Vite client types (catalog.ts: PhysicalRegistration, MetadataDataset.urn).
  • 2026-07-19 (fourth slice): /datasets/tables payloads carry the contract (TableSummary.urn + .physical_registration via CatalogRowSnapshot in-session enrichment; Iceberg-only entries stay null). The per-engine polaris_registration_status/gravitino_registration_status response fields are RETIRED (zero consumers verified across monolith/client/ui) — MarketDataMaterializeResponse now carries urn + the single physical_registration aspect instead (§23.3 acceptance: per-engine statuses are evidence, not API surface). MCP re-keying started: data.catalog.get_schema + data.iceberg.run_query accept the catalog URN (_resolve_dataset_ref: URN → row-id lookup under the same tenancy filter, physical identifier read off the row; malformed URNs and unregistered rows fail closed; legacy identifier path byte-identical).
  • Open: legacy engine-noun alias deletion after the shadow window; folding the read-time orphan merges (_iceberg_only_rows, DiscoveryService) into the reconciler (auto-registering orphan rows needs a tenancy decision — orphan tables carry no workspace context and NULL-workspace rows are visible to all tenants); CLI cutover (blocked on the CLI's own GET /datasets/ target, which does not exist on the monolith — a CLI defect, not a payload gap anymore); remaining engine-noun MCP tool re-keying (data.iceberg.read_slice/snapshot_history/time_travel_read, hudi/questdb/streaming writes); strangler extracted_ref + flip once a serving extraction target exists.