ADR 025 — Portable dual-engine orchestration authority

• Status: Accepted (2026-08-03, gated) via the ARP review — alphaswarm_orchestration becomes the Workflow Runtime context's task-graph home; TaskDefinition compilation stays activation-gated; precondition: the repo is cataloged and evidence-graded (ARP OQ-2). See alphaswarm_internal docs/architecture/agent-first-research-platform/02-context-map-and-ownership.md §6 (binding disposition table) and 10-adrs.md. Previously: Proposed (2026-07-11); activation remains gated by representative parity and rollback evidence
• Authors: Platform, data, agentic, security, and SRE teams
• Related: ADR 005, ADR 015, ADR 016, canonical guide, and rollout runbook

Context

AlphaSwarm currently uses several execution mechanisms for different jobs: Celery for compatibility tasks and schedules, Dagster for data assets and checks, Argo for cluster batch infrastructure, and WorkflowRuntime for agent semantics. This is operationally useful but does not provide one generic task definition, one cross-engine run model, or a consistent context and reconciliation contract.

Dagster and Prefect solve overlapping orchestration problems but expose different native strengths. Dagster is the established data-asset authority. Prefect provides a flexible flow/deployment/event model and Kubernetes work pools suited to agent workflows. Replacing one with the other would discard useful native behavior and create a large migration blast radius.

The platform also requires tenant-aware customer APIs and authorization that neither internal native UI should own. Reading framework internal tables would couple AlphaSwarm to undocumented schemas and bypass supported behavior.

Decision

Create the private alphaswarm_orchestration package as the dependency-light portable orchestration authority.

1. A generic TaskDefinition compiles to Dagster or Prefect. Immutable versions are content-hashed.
2. Each logical run, retry policy, concurrency policy, and active trigger is owned by exactly one selected engine.
3. extensions.dagster and extensions.prefect preserve native features. A required extension that cannot compile produces an error diagnostic.
4. EngineAdapter is the only generic control interface. It includes explicit restart rehydration for persisted registrations and runs.
5. Dagster and Prefect retain their own databases and native metadata. Adapters use supported SDK, HTTP, and version-gated GraphQL contracts only.
6. The monolith owns the tenant-scoped normalized ledger, customer API, authorization, Alembic migration, audit, SSE compatibility projection, and reconciliation cursor.
7. Large native payloads live in MinIO behind checksummed references. Queryable summaries remain in PostgreSQL.
8. ExecutionContextV1 is the signed cross-process context contract. Native metadata is never an authorization source.
9. alphaswarm_worker remains the heavy-compute plane, and agents.WorkflowRuntime remains the agent-workflow semantic layer.
10. Celery remains a compatibility transport during migration. Argo remains cluster-batch infrastructure, not a third logical-run authority.
11. The monolith is the durable session-resource manager authority. It embeds the provider-neutral coordinator, owns forced-RLS state and idempotent outcomes, and performs restart reconciliation. A session resource never becomes a second run scheduler or an authorization source.
12. A dependency-light HTTP provider calls a versioned, bearer-authenticated controller API. The controller alone mutates fenced Kubernetes resources; it does not derive tenant authorization from request bodies, leases, labels, or native resource metadata.

The renderer-neutral semantic model and generated diagrams in the canonical guide are part of this decision's review evidence.

Why two engines

Concern	Dagster	Prefect
Existing strength	Assets, dbt, partitions, checks, sensors, backfills	Flows, deployments, events, automations, Kubernetes work pools
Representative default	data.pipeline_manifest_materialization	agents.workflow_runtime
Generic compilation	Ops and job, optional assets	Flow and tasks
Native control API	Public client plus version-gated GraphQL	PrefectClient and events APIs
Heavy compute	WorkRequest delegation	WorkRequest delegation
Customer UI	AlphaSwarm only	AlphaSwarm only

Co-equal does not mean simultaneous. A definition may be registered with both engines for portability tests, but a logical run and active trigger are never jointly owned.

Metadata authority

The normalized ledger records definitions, immutable versions, registrations, trigger ownership, logical runs, attempts, append-only events, artifact references, logical/native bindings, and reconciliation cursors. It retains canonical and native state together.

Every table uses existing project/workspace scope columns and forced PostgreSQL RLS. Unique engine/native IDs prevent duplicate projections. A partial unique trigger-owner index prevents concurrent schedule ownership. A partial active-run index supports operational queries. Initial events remain a single append-only table; partitioning requires measured evidence near the existing 100-million-row threshold.

This projection is not a replacement for native metadata. Native assets, partitions, checks, causal events, flow artifacts, and engine diagnostics remain queryable through supported APIs and are summarized or referenced by the ledger.

The session-execution projection is separately additive: execution sessions, resources, reservations, immutable lease authority, reference-counted consumers, append-only events, durable operation outcomes, and scoped cache references. Provider fencing and consumer compare and swap are durable state, not process-local locks. This allows restart repair without accepting a stale renew/release or exposing a native endpoint as authority.

Security decision

The gateway remains the authorization policy enforcement point. Mutations require authenticated scopes, tenant and object ownership, idempotency, audit, and step-up for destructive actions. PostgreSQL RLS is a second boundary.

The monolith-to-controller lifecycle API is a distinct machine trust boundary. The controller authenticates a resolved service credential with the manage:infrastructure scope before accepting a versioned provision, renew, release, or abandoned-cleanup command. Signed context projections and fencing data constrain the operation but never replace machine authentication or gateway authorization.

The signed ExecutionContextV1 envelope protects integrity and expiry while carrying identity, tenancy, runtime, session, experiment, and trace state. It contains references to config and secrets, not secret values. Dagster tags, Prefect parameters, Kubernetes labels, and worker environment variables carry only the reference, digest, trace carrier, and safe searchable identifiers.

Alternatives considered

Dagster only

Rejected as the universal authority. It would preserve the data asset system but force agent workflows and dynamic Kubernetes flow behavior into a less natural model. It would also make portability testing impossible.

Prefect only

Rejected as the universal authority. It would require replacing established Dagster assets, dbt integration, checks, partitions, sensors, and backfills.

Keep Celery as the primary generic authority

Rejected as the long-term target. Celery remains useful as a compatibility transport, but does not supply the desired definition compiler, native metadata, trigger semantics, and run reconciliation model.

Add a third custom scheduler

Rejected. AlphaSwarm normalizes and governs supported control planes; it does not recreate scheduling, retries, or distributed execution inside the monolith.

Read Dagster and Prefect databases directly

Rejected. Internal schemas are not integration contracts, upgrades could break the projection, and direct reads would blur native ownership.

Mirror one logical run into both engines

Rejected. Dual ownership makes cancellation, retry, concurrency, schedule deduplication, and incident recovery ambiguous.

Consequences

Positive

• One generic definition and normalized run model can execute on either engine.
• Native framework value is retained instead of reduced to a lowest-common denominator.
• Restart-safe rehydration and reconciliation make the projection repairable.
• Tenant authorization remains in AlphaSwarm and is testable independently of native UIs.
• Representative migration is reversible without deleting additive metadata.

Costs and risks

• Two self-hosted control planes, databases, API contracts, upgrade paths, and runbooks require operations ownership.
• Compiler parity must be tested continually; native extensions can reduce portability by design.
• Version-gated Dagster GraphQL documents require contract tests on every upgrade.
• The normalized ledger is eventually consistent with native engines and must expose freshness and reconciliation health.
• Schedule handoff is a high-risk operation unless the single-owner invariant is checked before and after the flag change.

Acceptance before status changes to Accepted

1. system.context_echo succeeds on both engines with identical normalized context and output.
2. data.pipeline_manifest_materialization passes Dagster default and Prefect parity runs against the same manifest contract.
3. agents.workflow_runtime passes Prefect default and valid Dagster compilation.
4. Signed user, session, environment, and W3C trace context survives every process boundary.
5. Cancellation, retry ownership, restart rehydration, event pagination, reconciliation, and rollback drills pass.
6. A schedule-owner query proves no migrated schedule has more or less than one active owner during cutover.
7. Compatibility tests pass with rollout flags off.
8. Compose and k3d smoke tests prove both control planes, databases, Redis, workers, OTel, representative runs, cancellation, and restart repair.
9. Ray, Dask, and Spark provider fixtures prove create/readiness/renew/release, namespace and NetworkPolicy enforcement, fencing, tombstones, reference counting, cancellation compensation, and restart reconciliation.
10. A bound WorkRequest proves signed context, lease authority, runtime compatibility, point-in-time data binding, deadline, cancellation, and trace continuity across the driver and remote worker boundary.
11. The embedded manager survives process restart using forced-RLS state, and its dependency-light HTTP provider proves authenticated controller health, versioned lifecycle calls, response-authority validation, and secret redaction.
12. SSE reconnect tests prove Last-Event-ID overrides a stale after query cursor and neither refresh nor reconnect drops or duplicates events.

Rollback

Rollback is flag-first and additive:

1. stop new registrations and launches;
2. disable the selected native trigger;
3. restore the corresponding Celery Beat entry and confirm it is the only owner;
4. allow active native runs to finish or cancel them through the authorized gateway; and
5. retain normalized definitions, events, artifacts, and cursors as readable audit history.

No rollback step reads or edits native framework tables directly.