Auto-Spawn Chain Protocol — Hands-Free Pipeline Execution

Type: Operational pattern
Related: kelly-handbook-ch7-multi-agent, beads-adoption, operator-control-patterns

The Problem: Manual Step Chaining

Without auto-spawn, every pipeline step required the Router to:

  1. Wait for the current subagent to finish
  2. Read the completion event
  3. Manually run spawn-subphase.sh for the next step
  4. Copy-paste the generated sessions_spawn command
  5. Execute it
  6. Call sessions_yield to wait again

For a 20-step pipeline, that's 20 manual interventions. The operator had to babysit every transition. Pipelines stalled whenever the Router was in another session or the operator was away.

The Solution: --auto-spawn Flag (PIP-68)

The --auto-spawn flag on spawn-subphase.sh appends structured metadata after the sessions_spawn output. This metadata tells the Router what to do when the current step completes — no human input needed.

Usage

factory/scripts/spawn-subphase.sh --auto-spawn <project> <step-id> <pipeline>
# e.g. factory/scripts/spawn-subphase.sh --auto-spawn voice-ai 2.1 web-build

Without --auto-spawn, the script behaves as before — no metadata block, manual chaining required.

The Metadata Format

When --auto-spawn is used, the script appends this block after the sessions_spawn output:

# === AUTO_SPAWN_PIP68 ===
NEXT_STEP=2.2
PROJECT=voice-ai
PIPELINE=web-build
PIPELINE_DONE=0
CLOSING_BEAD=workspace-abc
# On wake-up (after sessions_yield returns), Router MUST:
#   1. bd close <CLOSING_BEAD> router  # close this bead
#   2. Run: spawn-subphase.sh --auto-spawn <PROJECT> <NEXT_STEP> <PIPELINE>
#   3. sessions_yield  # wait for child completion
#   4. Repeat until pipeline complete (NEXT_STEP empty or PIPELINE_DONE=1)
# === END_AUTO_SPAWN_PIP68 ===

Field Reference

Field Meaning
NEXT_STEP The step ID to spawn next (e.g., 2.2). Empty if this is the last step or the step isn't in the formula.
PROJECT The project identifier (e.g., voice-ai).
PIPELINE The pipeline name (e.g., web-build).
PIPELINE_DONE 1 if this is the last step in the pipeline. 0 if more steps follow.
CLOSING_BEAD The bead ID to close when this step completes.

Router Wake-Up Behavior

After sessions_yield returns (the child subagent completed), the Router follows this exact sequence:

1. Close the Completed Bead

bd close <CLOSING_BEAD> router

This marks the step as done in the Beads tracking system.

2. Check NEXT_STEP

  • If NEXT_STEP is non-empty: Run spawn-subphase.sh --auto-spawn <PROJECT> <NEXT_STEP> <PIPELINE>, then sessions_yield to wait for the next child.
  • If NEXT_STEP is empty and PIPELINE_DONE=1: Pipeline is complete. Stop chaining.
  • If NEXT_STEP is empty and PIPELINE_DONE=0: The step is not in the formula (molecule sub-step or manual-spawn-only step). Stop chaining — operator intervention required.

3. Repeat

The loop continues: spawn → yield → wake → close bead → spawn next → yield → ... until the pipeline completes or hits a non-formula step.

Execution Timeouts

Each step type gets an appropriate runTimeoutSeconds:

Step Type Timeout
BUILD 900s (15 min)
TEST 600s (10 min)
QA 720s (12 min)
DEPLOY 600s (10 min)

If a step times out, RALPH takes over (retry → escalate). The auto-spawn chain pauses during RALPH handling and resumes when the step completes successfully.

What Happens on Failure

Auto-spawn doesn't bypass error handling. When a step fails:

  1. The Router detects the failure (timeout, STATUS: FAILED, or missing DONE marker)
  2. RALPH protocol kicks in — up to 3 retry attempts
  3. If retries exhaust, the task is marked BLOCKED and escalated
  4. The auto-spawn chain does not advance past a failed step

This is critical: auto-spawn is for happy-path chaining. Error handling is still the Router's responsibility.

Interaction with Operator Hold

Auto-spawn respects Operator Hold (PIP-17). If a hold is set after step 2.3:

set-hold.sh myproject 2.3

The auto-spawn chain will complete step 2.3, then stop. It won't spawn step 2.4 until the operator clears the hold:

continue myproject

This gives the operator a way to inspect intermediate state without canceling the pipeline.

Metadata Parsing

The Router extracts metadata by scanning for the sentinel comments:

# Pseudocode for metadata extraction
if output contains "=== AUTO_SPAWN_PIP68 ===":
    extract NEXT_STEP, PROJECT, PIPELINE, PIPELINE_DONE, CLOSING_BEAD
    store for wake-up handling

The metadata is always in the same format — key=value pairs between sentinel comments. No YAML parsing, no JSON — just plain text that's easy to grep.

Before and After

Aspect Before PIP-68 After PIP-68
Pipeline execution Manual step-by-step Hands-free from 1.1 to completion
Operator involvement Every step transition Only on failures or holds
Overnight runs Not possible Runs unattended
Error handling Same (RALPH) Same (RALPH) — unaffected
Hold/resume N/A (manual already) Respects set-hold.sh

Example: Full Pipeline Run

# Start the pipeline — spawn step 1.1 with auto-spawn
spawn-subphase.sh --auto-spawn myproject 1.1 web-build
# → outputs sessions_spawn command + AUTO_SPAWN_PIP68 metadata
# → NEXT_STEP=1.2, CLOSING_BEAD=workspace-123

# Execute the sessions_spawn command
# → sessions_yield — wait for child

# Child completes — Router wakes up
# 1. bd close workspace-123 router
# 2. NEXT_STEP=1.2 → spawn-subphase.sh --auto-spawn myproject 1.2 web-build
# 3. sessions_yield — wait for next child

# ... repeats through 7.4 (DEPLOY) ...
# PIPELINE_DONE=1, NEXT_STEP="" → pipeline complete

Key Takeaway

Auto-spawn turns a pipeline from a sequence of manual handoffs into a self-driving chain. The operator sets it in motion and walks away. Failures are caught by RALPH. Holds give inspection points. The metadata protocol is dead simple — five key=value pairs between sentinel comments — and that simplicity is why it works.

PIP-68 was implemented after running the first multi-step pipeline manually, where the Router had to intervene at every transition. The motivation: overnight pipeline runs shouldn't require an awake operator.