Saltar al contenido principal

Agentic pipeline

Doc map: intro · Sequence diagrams: flows · Spec-pattern primer: agentic-development · Multi-agent topologies: multi-agent-patterns · Orchestration adapters: workflow-studio · Worked tutorial: tutorials/first-agent-workflow.

This page walks through the AlphaSwarm agentic-trading lifecycle: pick a model, register a data source, snapshot the spec, dispatch through the workflow runtime, and review the run. Every action has a REST + CLI surface so you can script the same flow; every action also has an alphaswarm_client (Vite UI) route at alpha-swarm.ai so a human can drive it.

The pipeline is five stages. The new stage since the prior version of this doc is Spec snapshot — every spec-driven run now hash-locks into an immutable *_spec_versions row before any work happens.

1 — Models and providers​

Open /models in the operator UI (alphaswarm_client). The page lives at alphaswarm_client/src/routes/models/ and exposes three tabs:

  • Ollama (host) — type a model tag in Pull a model (e.g. nemotron, llama3.2, qwen2:7b) and click Pull. A Celery task streams progress over the canonical /chat/stream/{task_id} envelope so the page shows a real-time download bar.
  • vLLM — every YAML under configs/llm/ becomes a profile card showing compose status, served models, and Start / Stop buttons. Starting a profile auto-saves its base_url as the active vLLM endpoint.
  • SERA-32B — opt-in Ai2 Open Coding model for the codebase MCP elaborator (see sera). Configure ALPHASWARM_SERA_ENABLED=true + ALPHASWARM_SERA_ENDPOINT in your env.

Every model call routes through router_complete (AGENTS rule 2). Provider selection is declared in AgentSpec.model; the runtime drives the call — never call router_complete directly from inside an agent body (AGENTS rule 12).

REST equivalents (each returns TaskAccepted for streaming endpoints):

curl -X POST localhost:8000/agentic/models/pull \
    -H 'content-type: application/json' \
    -d '{"name":"llama3.2"}'

curl -X DELETE localhost:8000/agentic/models/llama3.2
curl -X GET    localhost:8000/agentic/models/running
curl -X GET    localhost:8000/agentic/vllm/profiles
curl -X POST   localhost:8000/agentic/vllm/start \
    -H 'content-type: application/json' \
    -d '{"profile":"vllm_nemotron"}'

2 — Data sources​

Open /data/hub in the operator UI. This is the active replacement for the legacy Solara explorer pages.

The Hub exposes the four data-plane tiers (see data-plane):

  • Discovery browser — unified ingested / pending / orphan / external_only entries; filter chips drive the DiscoveryService.
  • Iceberg Editor — namespace browser + parquet preview + column profiling.
  • Airbyte builder — schema-driven connector editor at alphaswarm_client/src/components/airbyte/builder/. Emits either Airbyte YAML or an AlphaSwarm-native Fetcher stub. No free-text credential fields — every secret resolves through <EntityPicker kind="credentials" /> (AGENTS rule 31).
  • Dagster sandbox — ephemeral per-session Dagster + Airbyte environment (AGENTS rule 32).

REST surface:

curl -X GET  http://localhost:8000/discovery/entries
curl -X POST http://localhost:8000/sources/alpha_vantage/probe
curl -X POST http://localhost:8000/discovery/entries/<id>/promote
curl -X POST http://localhost:8000/dagster/sandbox/sessions

Or invoke the data MCP tools directly:

curl -X POST http://localhost:8000/mcp/data/tools/data.discovery.browse/invoke \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(alphaswarm-cli auth token)" \
    -d '{"namespace_prefix":"alphaswarm_silver_yfinance"}'

3 — Spec snapshot​

Every spec-driven run hash-locks the spec into a *_spec_versions row before any work happens. The same content always returns the same version_id; any field change creates a new row; old rows stay forever for replay. This is the invariant that makes the entire agentic pipeline auditable.

Five hash-locked spec types ship today:

SpecRuntimeVersions tableAGENTS rule
AgentSpecAgentRuntimeagent_spec_versions12-13
BotSpecBotRuntimebot_versions14-15
RLExperimentSpecRLRuntimerl_experiment_versions16-17
AnalysisSpecAnalysisRuntimeanalysis_spec_versions23-24
WorkflowSpecWorkflowRuntimeworkflow_spec_versions40-41

Plus two additive ones from the management engine:

SpecRuntimeVersions tableAGENTS rule
TerraformStackSpecTerraformRuntimeterraform_stack_spec_versions42-43
(workload ops)WorkloadRuntimeworkload_runs (write-only ledger)45

REST:

# AgentSpec
curl -X POST http://localhost:8000/agents/specs \
    -H "Content-Type: application/json" \
    -d @configs/agents/research_lite.yaml

# WorkflowSpec
curl -X POST http://localhost:8000/workflows/specs \
    -H "Content-Type: application/json" \
    -d @configs/workflows/my-research-loop.yaml

4 — Workflow dispatch​

WorkflowRuntime is the additive control plane that composes every spec runtime into multi-node DAGs. It ships with seven OrchestrationAdapter kinds (AGENTS rule 40):

  • graph — LangGraph state machine
  • crew — CrewAI manager-pattern crew
  • debate — bounded debate with N participants
  • fusion — fan-out / fan-in
  • execution — wraps an RLRuntime / BotRuntime / AnalysisRuntime as a single node
  • schedule — Cron-triggered, idempotent
  • studio — Operator-driven UI wiring at /workflows

Dispatch:

curl -X POST http://localhost:8000/workflows/<name>/run \
    -H "Content-Type: application/json" \
    -d '{"inputs": {...}}'

The runtime:

  1. Re-hash-locks every referenced spec (idempotent).
  2. Opens a workflow_runs row with status=pending.
  3. Builds the adapter DAG.
  4. Walks nodes; for each, opens an agent_runs_v2 row and delegates to the relevant runtime.
  5. Emits canonical progress frames at every transition.
  6. Calls should_halt() before every step — the topbar KillSwitch reaches every node within ~250ms.
  7. Enforces cost_caps (per_node_max_tokens, per_run_max_usd) per AGENTS rule 12.

Replay:

curl -X POST http://localhost:8000/workflows/runs/<run_id>/replay

Replay reuses the same workflow_spec_versions row + every referenced *_spec_versions row; a new workflow_runs row lands with a parent_run_id pointer.

5 — Review​

Three review surfaces, each consuming the same canonical ledger:

WebSocket stream​

The frame envelope is {task_id, stage, message, timestamp, **extras} per AGENTS rule 4. Subscribe from any client:

const ws = new WebSocket(`ws://localhost:8000/chat/stream/${task_id}`);
ws.onmessage = (e) => {
  const f = JSON.parse(e.data);
  console.log(f.stage, f.message, f.extras);
};

agent_runs_v2 + workflow_runs ledger​

Agent-safe reads via DataMCP:

curl -X POST http://localhost:8000/mcp/data/tools/data.workflows.describe/invoke \
    -H "Content-Type: application/json" \
    -d '{"workflow_run_id": "<id>"}'

curl -X POST http://localhost:8000/mcp/data/tools/data.agents.list_runs/invoke \
    -H "Content-Type: application/json" \
    -d '{"workflow_run_id": "<id>", "limit": 20}'

Each row carries experiment_id + test_id (AGENTS rule 34), total_tokens, total_cost_usd, and a full per-step breakdown under agent_run_steps.

Inkeep AI assistant + docs MCP server​

Two new surfaces in 2026-05:

  • Inkeep widget in-product. The "Ask AI" button in alphaswarm_client routes to an Inkeep agent that has the entire docs corpus + every public AlphaSwarm API spec ingested. It cites by URL and never invents references.
  • Docs MCP server at docs.alpha-swarm.ai/mcp. An RFC 9728 + 8707 compliant Cloudflare Worker (AGENTS rule 49). Cursor / Claude / Continue / custom scripts connect to it for search, fetch_page, and list_pages over the same corpus. In-platform agents reach it through the bridged data.docs.* MCP tools.

Both surfaces compose with the workflow runtime: a workflow node can call Inkeep / the docs MCP server as an external tool, and the agent_runs_v2 row records the call.

Worked example: build a research workflow​

Goal: snapshot an AgentSpec + WorkflowSpec, dispatch the workflow, tail progress, inspect the ledger — all from this page.

Step 1 — snapshot an AgentSpec​

Re-running with identical content returns the same (spec_id, version_id) — the runtime treats it as a no-op.

Step 2 — snapshot a WorkflowSpec that references it​

Step 3 — dispatch​

Step 4 — tail progress​

curl -N http://localhost:8000/chat/stream/<task_id>

You will see frames in the canonical envelope. Expected stages: workflow.started → node.research.started → agent.token (×N) → node.research.completed → workflow.completed.

Step 5 — inspect the ledger​

Demonstrate the analysis pattern with a small inline sample of what the MCP describe call returns:

Step 6 — verify​

  • agent_spec_versions row exists with the recorded spec_hash.
  • workflow_spec_versions row exists; its content references the agent_spec_versions row from Step 1.
  • One workflow_runs row + one agent_runs_v2 row (one node).
  • total_cost_usd is under the workflow's per_run_max_usd cap.
  • Re-dispatching by triggering Step 3 again creates a NEW workflow_runs row but reuses ALL the same *_spec_versions rows.

What next​

The four-runtime story​

This pipeline is one of four overlapping execution surfaces. Each has its own concept doc but they all share the same hash-lock invariant, the same canonical progress frame, the same kill-switch fan-out, and the same experiment_id audit chain.

RuntimeLifecycle surfaceWorked tutorialConcept doc
AgentRuntimeSingle agent, single spec(covered here)agents
BotRuntimeBot = universe + strategy + ML + agents + RAG + risktutorials/first-botbots
RLRuntimeTrain / evaluate / paper / replay / walk-forwardtutorials/first-rl-experimentconcepts/rl/rl-framework
WorkflowRuntimeComposition layer over the other threetutorials/first-agent-workflowworkflow-studio

Hard rules (agentic-pipeline scope)​

The full set is in AGENTS.md. The agentic-pipeline subset:

  • Rules 12-13 — All spec-driven agent runs go through AgentRuntime; agent_spec_versions rows are immutable.
  • Rule 22 — Agents never read Postgres / Iceberg directly; every read through a DataMCPTool.
  • Rule 40 — All workflow lifecycle actions go through WorkflowRuntime.
  • Rule 41 — workflow_spec_versions rows are immutable hash-locked snapshots.
  • Rule 34 — Every run-producing flow populates experiment_id.
  • Rule 49 — Every MCP server is RFC 9728 + 8707 conformant.
  • Rule 54 — Delegated agent tokens for HTTP MCP calls go through TokenExchangeBroker (RFC 8693 + Auth0 Custom Token Exchange Profile alphaswarm-agent-delegation).

Deeper reads​