Skip to main content

Connector Control Plane & Data Mesh

The Connector Control Plane expands the AlphaSwarm data fabric into a distributed data mesh: customers bring their own data sources and credentials (BYOB / BYODB / BYOK), AlphaSwarm ships curated source templates and a public-data catalog, and a federated query layer lets analytics span the lakehouse, customer sources, and public datasets through one SQL surface. It is governed by hard rules 61-63 and .cursor/rules/connector-control-plane.mdc.

Capabilities​

  • BYOB (Bring-Your-Own-Bucket) - read a customer-owned S3 / GCS / Azure Blob / S3-compatible bucket via the byob_object dataset kind.
  • BYODB (Bring-Your-Own-Database) - federated read of a customer Postgres / MySQL / SQL Server / Snowflake / Redshift via the byodb_sql dataset kind (no copy).
  • BYOK (Bring-Your-Own-Key) - per-tenant AWS KMS CMK wraps the DEKs that encrypt connector credentials; disabling / deleting the CMK crypto-shreds the tenant's data (the kill-switch).
  • Source templates + public data - the alphaswarm_ingest marketplace catalog (AWS Open Data, SEC EDGAR, FRED, Hugging Face / Croissant, World Bank, IMF SDMX 3.0, Nasdaq) with license + attribution.
  • Federated query - Apache Gravitino (catalog-of-catalogs, Metalake = tenant boundary) + Trino + Trino Gateway + OPA.
  • Outbound zero-copy sharing - the read-only alphaswarm_data_federation broker (OpenFGA + signed share tokens).

Architecture​

Registering a customer source​

POST /me/data-sources (step-up MFA). For a vaulted credential the secret payload is accepted once, envelope-encrypted, and dropped. For cross-account AWS, the platform derives an ExternalId; deploy the returned trust policy (GET /me/data-sources/{id}/trust-policy) in your account, then the registration validator runs the AssumeRole-without-ExternalId negative test. A validated source appears in the discovery browser and is selectable as a hosted dataset.

Security​

  • Cross-account: sts:AssumeRole with a platform-generated, non-guessable ExternalId (confused-deputy guard); short-lived STS creds are never persisted.
  • BYOK: per-tenant CMK provisioned through TerraformRuntime (tenant_byok_cmk module) with kms:ViaService / kms:CallerAccount conditions + rotation; crypto-shred = disable / schedule-delete.
  • SSRF: every customer-supplied URL is validated (scheme allow-list, DNS re-resolution per redirect, link-local / 169.254.169.254 blocked); IMDSv2 is enforced fleet-wide as defense-in-depth.
  • Tenancy: customer_data_sources is RLS-protected by workspace_id and envelope-encrypted on top; secrets never appear in any response / log / audit.

Federated query​

data.federation.query runs read-only SQL through Trino. Authz is enforced server-side by the OPA plugin (catalog/schema/table + row filters scoped to the caller's workspace); the client never makes a local allow/deny decision. Gravitino federates the Iceberg lakehouse, customer sources, and public datasets behind one Iceberg REST surface.

Outbound sharing​

Cross-tenant zero-copy sharing goes through the standalone alphaswarm_data_federation gateway: OpenFGA check -> mint an aud-bound share token (RS256, jti, TTL <= 600s, never a bearer passthrough) -> upstream recall -> Redis result cache. Shares record a SHARED_WITH ownership-graph edge. The broker is read-only by design.