Q: How does Kelly's memory system actually work across its 5 layers?

Short answer: Kelly's memory system has two parts: the foundational 5-layer narrative hierarchy (SOUL → MEMORY → daily logs → project context → structured data) and the semantic search acceleration layer on top (sub-20ms lookups over the narrative files). Together they solve the problem thatLLM recall is associative, not indexed — so the system uses narrative at the high layers for compounding, and structured tables only at the low layers for lookup.

The 5-Layer System

The five layers are defined in five-layer-memory and operationalized in kelly-factory-overview. Each layer has a distinct purpose and format:

Layer 1 — SOUL.md (Identity)

Purpose: Who the agent is — identity, principles, communication style, domain expertise, recurring rules.

Format: Narrative. Written once, updated rarely. Read at every session start before anything else.

What it contains:
- Tone and communication style
- Core principles and operating rules
- Domain expertise and known capabilities
- Memory-writing rules (explicit triggers for when to update MEMORY.md)

Key property: SOUL.md is the agent's identity. It shapes everything. If an agent's SOUL.md says it's a "sharp, calvinist operator," it will be blunt and critical; if it says "warm and supportive," it will be encouraging. This layer is foundational — it runs before any other context loads.

Layer 2 — MEMORY.md (Curated Long-Term)

Purpose: Persistent cross-session facts — decisions, operator preferences, system configuration, known issues, escalation patterns, learnings.

Format: Narrative, curated, human-editable. The agent writes to it after significant discoveries; the operator edits it when context changes.

What it contains:
- Routing decisions and why they were made
- Operator preferences (how they like to be updated, what they care about)
- System configuration and known failure modes
- Escalation patterns that have worked
- Learnings distilled from daily logs

Key property: This is the agent's "curated long-term memory" — not everything that happened, but what matters. Entries here persist across sessions and inform behavior without requiring re-learning. The agent writes to it when it learns something worth remembering; it reads from it to avoid repeating mistakes.

Load behavior: MEMORY.md loads at every session start (alongside SOUL.md) as part of the workspace boot sequence.

Layer 3 — Daily Memory Logs (memory/YYYY-MM-DD.md)

Purpose: Raw chronological operational record — projects initiated and completed, failures and resolutions, gate passes and fails, operator decisions, notable findings.

Format: Append-only per session day. Narrative, chronological.

What it contains:
- What was worked on today
- What succeeded and what failed
- Gate decisions (PASS/FAIL and why)
- Operator interactions and decisions
- Findings worth noting for future reference

Key property: These are raw logs, not curated summaries. They capture everything in case it's needed later. The daily format means you can reconstruct any day's work by reading that day's file. They're append-only — once written, they form a permanent record that compounds over time.

Load behavior: Loaded on demand (not at session start — too much volume). The agent reads specific days when relevant to the current work.

Layer 4 — Project Context Files (projects/{id}/context.md)

Purpose: Long-running project state that survives across gaps between active work periods.

Format: Narrative per project. Updated as projects evolve. Sub-agents spawned months later can read the context file and resume work without re-doing earlier steps.

What it contains:
- Current project state and what's been done
- What's pending and what's blocked
- Key decisions made during this project
- Relevant artifacts and where to find them
- What's been tried and what worked

Key property: Enables continuity across time gaps. A project might be actively worked on for a week, then idle for two months. When the agent returns, projects/{id}/context.md has the full state — no re-reading prior work required. This is critical for long-running projects.

Load behavior: Loaded on demand when entering a project scope. The Router loads it when routing work to a project, not at session start.

Layer 5 — Structured Data (data/.json, CSVs)

Purpose: Machine-readable data for programmatic manipulation without text parsing.

Format: JSON, CSV, structured files. Read and written by processes, not directly by agents.

What it contains:
- Token usage logs and budgets
- Agent performance metrics
- System configuration in machine-readable form
- Any structured data that would be cumbersome in narrative format

Key property: Agents don't directly parse this layer — it's for processes and scripts. The agent reads SOUL.md and MEMORY.md (narrative, human-readable); it reads daily logs and project context (narrative, human-readable); it does not read structured data files directly.

The Semantic Search Acceleration Layer

memory-system describes an additional layer that sits above the 5-layer narrative system:

The Dual-Layer Architecture

Layer 1 (Narrative Files)   ←  Source of truth
Layer 2 (Semantic Index)    ←  Acceleration structure

The Memory System adds a searchable semantic index overlaid on the narrative files. When content is written to memory, it's also indexed in a semantic search engine (typically a local vector store or lightweight JSON-based index). Queries go through the index for speed.

Sub-20ms lookup time is a deliberate engineering constraint. At the speed the factory operates — spawning multiple agents in parallel, running overnight builds, processing dozens of opportunities — memory retrieval can't be a bottleneck. 20ms means the memory system adds negligible latency to any operation that needs to look up prior context.

Why Two Layers?

The semantic index is not the memory itself — it's an acceleration structure. The source of truth is the narrative files. The index can be rebuilt from source files at any time. This separation means:
- If the semantic index is unavailable, the narrative files still work (humans can read them directly)
- If the narrative files need to be migrated, the semantic index can be rebuilt from them
- The index is optimized for speed; the files are optimized for meaning

What It Enables

At 85+ iOS apps built, the memory corpus was too large to search manually and too slow for real-time agent operations. The semantic layer solved both problems: fast retrieval via the index, while preserving narrative richness in the source files.

The learning loop (referenced in memory-system) depends on this: when a new app build surfaces a pattern worth capturing, the Memory System can quickly identify related prior patterns across dozens of previous builds — "we've seen this competitive gap before; here's what we tried and what worked."

Design Principle: Narrative > Tables at the Top

The fundamental insight (attributed to Yuki AI CEO's experiments, cited in kelly-factory-overview):

LLM recall is associative, not indexed. Tables are for lookup; narrative is for association. More knowledge can compound without expanding the attention footprint via progressive disclosure.

This explains the layer design:
- Layers 1–3 (SOUL, MEMORY, daily logs): Pure narrative. The agent reads these to understand context, make connections, and compound learnings. Narrative compounds because LLM recall works through association — a story about a decision connects to other similar decisions more naturally than a table row connects to other table rows.
- Layer 4 (project context): Narrative with project-specific structure. Still narrative — project histories are stories.
- Layer 5 (structured data): Tables only here, and only for machine consumption. Humans don't read this layer directly.

The "Map, Not Encyclopedia" Principle

kelly-factory-overview cites Yuki AI CEO's CLAUDE.md shrinking 36% while the repo doubled in size — validated by the "map, not encyclopedia" principle. The semantic index acts as the "map" — it tells the agent where to find relevant information without containing all the information itself. The narrative files are the "encyclopedia" — rich with detail, loaded on demand.