Beads Adoption — Formula-Driven Pipeline Tracking

Type: Operational pattern
Related: kelly-handbook-ch7-multi-agent, kelly-gas-town-gap-analysis, steve-yegge-beads

The Problem: Four Siloed Mechanisms

Before Beads, the factory tracked pipeline state through four separate, disconnected mechanisms:

  1. Pipeline state JSON — a machine-readable file tracking which steps were complete
  2. DONE marker files — filesystem flags (done_dir/DONE) indicating artifact output
  3. TEA audit logs — time-event-action records for post-hoc analysis
  4. Memory files — daily markdown notes capturing what happened and why

Each served a different purpose. None talked to the others. A step could be marked DONE in the filesystem but never recorded in pipeline state. Memory files might reference work that the audit log missed. Debugging required reading all four.

The Decision: Option B — Formula-Driven Beads

The factory chose "Option B" for Beads integration: formula TOML files define the pipeline structure, and Beads tracks execution against that structure.

A formula TOML file declares:
- Step IDs — e.g., 1.1, 2.1, 3.2
- Dependencies — which steps must complete before another can start (needs = ["1.1"])
- Agent assignment — which agent handles each step (agent = "carson")
- Workflow paths — the markdown workflow file to follow (workflow = "factory/workflows/research/1.1-intake.md")
- Output paths — where DONE markers land (output_path = "projects/{{project}}/scaffold/1.1-intake/")

Formula files use {{variable}} substitution (like {{project}}) so a single formula can drive many projects. This is the canonical approach — hand-written formulas with variable substitution, not auto-generated ones.

Example Formula Structure

# .beads/formulas/kelly-research.formula.toml

step
id = "1.1"
title = "Intake: {{project}}"
agent = "carson"
workflow = "factory/workflows/research/1.1-intake.md"
output_path = "projects/{{project}}/scaffold/1.1-intake/"

step
id = "2.1"
title = "Publish: {{project}}"
agent = "carson"
needs = ["1.1"]
workflow = "factory/workflows/research/2.1-publish.md"
output_path = "projects/{{project}}/publish/2.1-publish/"

Why Not bd mol distill?

bd mol distill extracts a formula from a completed molecule, but it hardcodes the project name into step IDs and titles — "title": "Publish: beads-full-test" with no variable substitution. Distill is useful for capturing ad-hoc work to replay exactly, not for creating generic templates. For reusable pipelines, hand-write the formula.

How It Works: Molecule Pour and Bead Resolution

Step 1: Pour a Molecule

When a project starts, the Router pours a molecule from the formula:

BEADS_DIR=.beads bd mol pour kelly-research --var project=my-project

This creates all step beads wired by dependencies — a complete graph of the pipeline in one command.

Step 2: spawn-subphase.sh Resolves Beads

When spawn-subphase.sh runs for a given step, it resolves the bead:

  1. Look for existing molecule step bead — searches for a bead whose title matches the step pattern (e.g., "Research: my-project")
  2. If found → reuse it — sets BEAD_SOURCE=molecule, uses the existing bead ID
  3. If not found → create new bead — sets BEAD_SOURCE=create, calls bd create

The bead ID is embedded in the task description as an HTML comment:

<!-- BEAD: workspace-mol-abc -->

Agents extract it with: grep -o '<!-- BEAD: \([a-z0-9-]*\) -->'

Step 3: Agent Closes Bead on Completion

When the agent finishes, it closes the bead using the bddone wrapper:

bddone <bead-id> <agent-name>
# e.g. bddone workspace-mol-abc carson

bddone is installed at ~/.local/bin/bddone. It auto-detects BEADS_DIR from the workspace and calls bd update <id> --status closed with a timestamp and agent note.

The Dual-Write Requirement

Every subphase must complete both:

  1. DONE marker — written to filesystem at done_dir/DONE
  2. Bead closure — via bddone <bead-id> <agent>

Neither alone is sufficient:
- Beads track work state — open, in_progress, closed. Is the work happening?
- DONE markers track artifact output — did the agent actually produce files?

A bead can be closed without artifacts being written (agent bugs out, marks complete anyway). A DONE marker can exist without a bead (manual work, pre-Beads artifacts). The dual-write catches both failure modes.

Key Commands

# See all open beads for a project
BEADS_DIR=.beads bd list --state open | grep <project>

# See what a specific agent is working on
BEADS_DIR=.beads bd list --state in_progress --owner <agent>

# View the dependency graph
BEADS_DIR=.beads bd graph <project>

# Inspect a molecule's structure
BEADS_DIR=.beads bd mol show <mol-id>

# Check bead change history
BEADS_DIR=.beads bd history <bead-id>

# Close a bead (the wrapper)
bddone <bead-id> [agent-name]

Disabling Beads

Set BEADS_ENABLED=0 to disable all Beads tracking without editing spawn-subphase.sh. This is useful for testing or when running outside the factory workspace.

What Beads Unified

Before (4 silos) After (Beads)
Pipeline state JSON — machine-readable Bead list + graph — machine-readable AND human-readable
DONE markers — filesystem flags DONE markers still used (dual-write)
TEA audit logs — post-hoc Bead history — built-in audit trail
Memory files — narrative Memory files still used (complementary)

The key improvement: a single bd list --state open shows everything that's in-flight across all pipelines. No more cross-referencing four files to understand system state.

Lessons Learned

  1. Formula paths must match reality — output_path in the formula must exactly match where DONE files actually go. A mismatch causes prerequisite checks to fail silently. (See: formula validation bug, Apr 27.)
  2. Test a formula before committing — pour the molecule, spawn one step, verify the DONE marker lands where expected.
  3. {{variable}} substitution is critical — without it, every project needs its own formula file. With it, one formula drives unlimited projects.
  4. Bead IDs must be stable — the <!-- BEAD: ... --> token in the task description is the link between the spawn system and the tracking system. If it's missing or wrong, the bead never closes.

This pattern was discovered and refined during the April 2026 two-pipeline validation run (test-web-run + factory-dashboard-rebuild), where 34 subphases completed successfully with dual-write tracking.