Agentic Coding

Spec-driven agentic coding is a systematic approach to building software with AI coding agents like Hermes. Instead of asking the agent to “just write it” and hoping for the best, you follow a disciplined process:

1. Define what to build — chat with the agent to refine requirements from the end-user perspective

2. Lock in the spec — turn the conversation into a concrete implementation plan

3. Set up the workspace — create a repository, install dependencies, configure tests and logging

4. Agent implements and self-corrects — the agent writes code, runs tests, reads logs, finds errors, and fixes them until everything passes

This approach turns the agent from a code generator into an autonomous developer that can build, test, and debug on its own.

Why Spec-Driven?¶

Most people use coding agents reactively — they describe a problem, get code back, and manually fix issues. Spec-driven coding is proactive: you invest time upfront to define clear requirements and acceptance criteria, then the agent works toward that target with measurable feedback.

Benefits:

• Predictable results — the agent has a concrete target, not a vague prompt

• Self-healing — automated tests catch regressions; the agent reads logs and fixes its own errors

• Traceable — every change ties back to a spec item, so you can audit why code exists

• Scalable — works for a single utility script or a multi-file project with dependencies

Without a spec and automated feedback, the agent is just guessing what you want. With them, it’s building to a blueprint.

Phase 1: Define the Spec¶

Start From the End-User’s Perspective¶

The most common mistake is describing what you want in technical terms before understanding what the user actually needs. Instead, start by describing the problem from the end-user’s point of view:

Bad spec: “Create a Flask API with /bmi endpoint that takes JSON input and returns a classification.”

Good spec: “A health app user types their weight (kg) and height (cm) and sees their BMI score with a category label like ‘normal’ or ‘overweight’. The result should be accurate to one decimal place.”

The Conversation Process¶

Work through these questions with the agent before writing any code:

1. Who is the user? — A beginner student? An experienced developer? End consumers?

2. What is the input? — CLI arguments? File? API request? Interactive form?

3. What is the output? — A number? A report? A chart? A downloadable file?

4. What are the edge cases? — Invalid input, missing data, extreme values, network failures

5. What does success look like? — Concrete acceptance criteria the agent can verify

Example conversation:

You: I want to build a tool that helps students check their BMI.

Agent: How will students use it?

You: They'll type their weight in kg and height in cm on the command line.

Agent: What should it show them?

You: Their BMI rounded to one decimal, plus a WHO category. If they enter
invalid data, it should explain the error instead of crashing.

Agent: Should it work as a script or a package they can import?

You: Both — a command-line script and a library they can import.

Lock It In¶

Once the conversation clarifies what you need, ask the agent to write a specification document that captures:

• Purpose — one-sentence description of the tool

• User stories — “As a [user], I want to [action] so that [benefit]”

• Input/output — exact formats, types, and constraints

• Acceptance criteria — testable conditions that define “done”

• Non-goals — what this version explicitly does NOT do

Save this as SPEC.md. It becomes the contract between you and the agent.

Live Documentation: Let the Agent Write Docs as It Goes¶

Beyond a static spec, treat documentation as a living artifact that the agent updates alongside the code. Instead of writing docs after the project is done, have the agent maintain them throughout:

Ask the agent to create and maintain:

• README.md — High-level overview, installation, quick start. The agent writes the first draft during scaffolding and updates it as features are added.

• Developer guide (docs/DEVELOPER.md or docs/developer.myst) — Architecture, module structure, API reference, how to run tests locally.

• User guide (docs/USER.md or docs/user.myst) — End-user-facing walkthroughs, examples, and FAQ.

These can be generated with Sphinx (using sphinx-quickstart and autodoc) or with MyST-MD directly (using myst build for static sites). Either way, serve them via GitHub Pages or a similar platform so docs stay up to date.

Prompt to start:

Create a docs/ folder with a README.md in the root that explains what
this project does, how to install it, and a quick example.
Also create docs/DEVELOPER.md for internal notes and docs/USER.md
for end-user guides. Update all three as we add features.

The agent will generate the initial structure. As you add features, remind it:

Update README.md and docs/USER.md to reflect the new feature we just added.

This way your documentation evolves naturally with the codebase rather than becoming a last-minute chore.

Generated documentation is a strong starting point, not the final product. The agent describes what the code does based on its understanding of the conversation and implementation — but that understanding is imperfect. Before publishing or sharing:

• Read through them to understand the current state of development — auto-generated docs can lag behind code changes, include outdated descriptions, or describe features that were planned but never implemented

• Remove temporary work — scratch implementations, debug scaffolding, and interim approaches should be stripped out

• Strip internal details — workarounds, hacks, and experimental features not meant for the audience should be removed

• Verify accuracy — confirm the documentation matches what the code actually does, not what the agent thought it does

Briefing the Agent: Start With Context, Not Commands¶

Before giving the agent implementation tasks, brief it so it understands the full context. An unbriefed agent picks random files, misses constraints, and creates conflicts. A briefed agent knows where to look and what to respect.

Step 1: Orient the agent to the workspace

You are going to help me build a Python package in this repository.
First, explore the current directory structure and tell me what you see.

The agent lists files and confirms its understanding. If it’s wrong, correct it before continuing.

Step 2: Confirm target files

The files you will modify are:
- pybmi/__init__.py (main module)
- tests/test_bmi.py (tests)
- README.md (documentation)

Do not modify any other files unless I ask. Confirm you see them.

This prevents the agent from editing unrelated files or creating duplicates.

Step 3: Share the spec and ask for questions

Here is our spec in SPEC.md. Read it and ask any clarifying questions
before you start implementing. Do not write code yet.

A well-briefed agent will ask about edge cases, naming conventions, or dependencies you haven’t considered — saving rework later.

Step 4: Assign the first task with constraints

Implement the bmi() function as described in SPEC.md.
Rules:
- Only modify pybmi/__init__.py
- After writing, run pytest and report which tests pass/fail
- Do not commit until I say so

Each subsequent task follows the same pattern: brief, confirm, execute, report.

Test Plans in the Spec, Not in Code¶

Writing test code is often the hardest part of a spec — and the one where agents excel most. Instead of hand-writing every assert, include a test plan in your SPEC.md and let the agent generate the test code from it.

Example test plan in SPEC.md:

## Test Plan

- Normal BMI: weight=70, height=175, expected bmi≈22.9, category="normal"
- Underweight: weight=45, height=170, expected bmi≈15.6, category="underweight"
- Obese: weight=100, height=170, expected bmi≈34.6, category="obese"
- Invalid weight: weight=-5, height=175, expected ValueError raised
- Zero height: weight=70, height=0, expected ValueError raised

Then prompt:

Read the Test Plan in SPEC.md and write the corresponding pytest
functions in tests/test_bmi.py. Use pytest.approx() for float
comparisons and pytest.raises() for expected exceptions.

The agent translates the plan into proper test code. You review the plan (which is readable), not the implementation details.

Testing Before Committing: The Agent’s Responsibility¶

Make testing a hard requirement in your briefing. Add this to every implementation prompt:

Before you propose a commit:
1. Run the full test suite: python -m pytest tests/ -v
2. Ensure ALL tests pass — not just the ones for your new feature
3. Show me the test output before committing
4. Do NOT commit until I confirm

This ensures the agent:

• Tests thoroughly, not just its own changes

• Catches regressions in previously working code

• Lets you verify the output before any commit lands

Always treat the agent’s test run as a pre-check, not a final gate. Before pushing to a remote repo or publishing, run the tests yourself, review git diff, and spot-check the output:

# Your final gate before pushing
python -m pytest tests/ -v
git diff HEAD
# Review, then push manually

Phase 2: Set Up the Workspace¶

Create the Repository¶

Before the agent writes any code, set up a clean workspace with structure, version control, and safety nets:

mkdir ~/projects/pybmi && cd ~/projects/pybmi
git init
git config user.name "Your Name"
git config user.email "you@yourdomain.edu"

Ask the agent to scaffold:

Create a Python package structure:
1. .gitignore for Python
2. pyproject.toml with setuptools and pytest as test dependency
3. pybmi/__init__.py with a docstring
4. tests/test_bmi.py with a placeholder test
5. SPEC.md with our spec

Commit everything:

git add -A && git commit -m "Initial spec and scaffold"

Golden rule: commit before every agent prompt so you can always revert.

Isolate the Environment¶

Create a conda environment so the agent’s package installs don’t pollute your kernel:

conda create -n pybmi python=3.12 pip pytest -y
conda activate pybmi

Without isolation, the agent might install packages that conflict with your Jupyter kernel. A dedicated environment lets it experiment freely.

Configure Automated Testing¶

The agent needs automated feedback to know if it’s succeeding. Write tests matching the spec before the agent implements anything:

In tests/test_bmi.py, add tests matching the spec:
1. test_bmi_calculation — bmi(70, 175) returns approx 22.9
2. test_classify_normal — classify(22.9) returns "normal"
3. test_classify_underweight — classify(15.7) returns "underweight"
4. test_invalid_input — negative weight raises ValueError

Run to confirm they fail (code doesn’t exist yet):

python -m pytest tests/ -v
# Expected: all FAIL — this is your baseline

This is the “Red” phase of TDD. The tests are now the agent’s target.

Phase 3: Agent Implements and Self-Corrects¶

The Implementation Prompt¶

Now give the agent a single, comprehensive prompt that references the spec, tells it to implement one piece at a time, and requires it to run tests after every change:

Read SPEC.md and implement the pybmi package.

Follow this process for each function:
1. Write the minimal implementation in pybmi/__init__.py
2. Run pytest to see which tests pass and which fail
3. If tests fail, read the error output and fix the code
4. Repeat until all tests for that function pass
5. Commit before moving to the next function

Stop if you hit an error you cannot resolve after 3 attempts.

Incremental Improvement: The Agent’s “Learning Rate”¶

Think of the agent’s work like gradient descent. Each iteration is a step toward the goal, and the step size determines whether it converges or diverges.

Too big a step — ask the agent to implement the entire spec in one shot, and it may produce incomplete code, skip edge cases, or generate tests that don’t match the implementation. It looks like progress but fails on inspection.

Too small a step — ask the agent to implement one line at a time, and it takes dozens of iterations to reach a working result. Each round of prompt, execute, review eats time.

Just right — break the work into functions or modules, have the agent implement one per iteration, run tests, and move on. This is the sweet spot where each step is meaningful but verifiable.

Key: specify this loop explicitly in your prompt so the agent runs autonomously rather than waiting for you to approve each step. A common mistake is writing:

Implement the code and run the tests.

The agent will do one pass and stop, waiting for your feedback. Instead, write:

Implement the functions in SPEC.md one at a time:
1. Write the next unimplemented function
2. Run pytest tests/ -v
3. If tests fail, fix and re-run (up to 3 attempts)
4. When all tests pass, move to the next function
5. Repeat until every function in SPEC.md is done

Do this loop autonomously — do not wait for me between functions.
Report your progress after each function, and stop only when all are done
or you hit an error you cannot resolve.

The phrase “do this autonomously — do not wait for me” is critical. Without it, the agent will implement one function, show you the result, and pause for approval — which adds a human delay to every iteration. With it, the agent keeps the loop running and delivers the full result in one go.

How to choose step size:

• Small project (single module): One function per step

• Medium project (package with multiple modules): One module per step, run all tests after each

• Large project (multiple packages): One package or feature group per step, commit between steps

If the agent is producing bugs consistently, reduce the step size — break functions into smaller pieces or add more tests between steps. If it’s taking too many rounds, increase the step size — group related functions together.

How the Agent Works¶

Here’s what happens behind the scenes. The agent follows a loop:

1. Read the spec — loads SPEC.md to understand requirements

2. Implement — writes code based on one spec item

3. Run tests — executes pytest and reads the output

4. Analyze failures — reads traceback, identifies root cause

5. Fix and repeat — patches code, re-runs tests

6. Commit — saves working code before moving on

The agent sees the same test output you would. If a test fails with ValueError: height must be positive, it reads that message and adds input validation. This is the key difference from casual prompting — the agent has measurable feedback rather than guessing.

Watching the Agent in Action¶

In VSCode (ACP Client), you see the agent’s tool calls in real time:

[Agent] Reading SPEC.md...
[Agent] Writing pybmi/__init__.py (bmi function)...
[Agent] Running: pytest tests/test_bmi.py -v
[Tool] test_bmi_calculation PASSED
[Tool] test_classify_normal FAILED — NameError: classify not defined
[Agent] classify() not implemented yet, moving on...
[Agent] Writing classify() function...
[Agent] Running: pytest tests/test_bmi.py -v
[Tool] All 3 tests PASSED
[Agent] Committing: git commit -m "feat: implement bmi and classify"

In JupyterLab, the chat panel shows the same progression — the agent reports each step as it goes.

The Agent Hits a Real Bug¶

Sometimes the agent makes mistakes it can fix itself. Example:

[Agent] Running: pytest tests/test_bmi.py -v
[Tool] test_invalid_input FAILED
[Tool] AssertionError: ValueError not raised
[Tool] +  where 0.0 = bmi(-5, 175)
[Agent] The bmi() function doesn't validate negative input.
[Agent] Adding validation:
[Agent]   if weight <= 0: raise ValueError("weight must be positive")
[Agent]   if height <= 0: raise ValueError("height must be positive")
[Agent] Running: pytest tests/test_bmi.py -v
[Tool] All 4 tests PASSED

The agent found its own bug, read the test output, and fixed it — without you needing to intervene. This is the self-healing loop in action.

Phase 4: Review and Iterate¶

Verify the Results¶

After the agent finishes, verify independently:

# Run all tests
python -m pytest tests/ -v

# Try the package manually
python -c "from pybmi import bmi, classify; print(bmi(70, 175), classify(bmi(70, 175)))"

# Review the agent's changes
git log --oneline
git diff HEAD~3

Quote and Explain Critical Code¶

Refactor and Verify From a Clean State¶

After stepwise development, the agent should refactor and run the entire project from a clean state before you declare it done. Here’s why:

When an agent divides a task into small steps, each step modifies files independently. Sometimes a later step overwrites or bypasses changes made in an earlier step — creating inconsistencies that only surface when the full code is evaluated together.

Think of it like a Jupyter notebook. Running individual cells out of order can produce results that look correct but depend on variables left over from previous runs. The only way to be sure is to restart the kernel and run everything from top to bottom. Similarly, the agent must treat the completed project as a whole — starting fresh and exercising the full workflow — rather than assuming each step’s isolated success implies overall correctness.

What to ask the agent:

Now that all features are implemented:

1. **Refactor** — clean up code structure, remove duplicates, consolidate
   imports, and ensure consistent style across all files.

2. **Clean-slate test** — delete any __pycache__ or build artifacts,
   reinstall the package (`pip install -e .`), and run the full test
   suite from scratch.

3. **End-to-end verification** — run the package's main entry point
   (CLI, API, or script) as an end user would, and confirm the output
   matches the spec.

4. **Report** — show me the test results and any changes made during
   refactoring.

This clean-slate step catches:

• Accidental overwrites — later steps that silently replaced earlier work

• Missing imports — functions added but never exported in __init__.py

• Stale state — tests that passed because of cached .pyc files

• Integration issues — individual modules working, but failing when combined

Add Logging for Real-World Debugging¶

For more complex projects, add logging so the agent (and you) can trace execution when things go wrong:

# pybmi/__init__.py
import logging
logger = logging.getLogger(__name__)

def bmi(weight, height):
    logger.debug(f"Calculating BMI: weight={weight}kg, height={height}cm")
    if weight <= 0:
        logger.error(f"Invalid weight: {weight}")
        raise ValueError("weight must be positive")
    if height <= 0:
        logger.error(f"Invalid height: {height}")
        raise ValueError("height must be positive")
    height_m = height / 100
    result = weight / (height_m ** 2)
    logger.info(f"BMI = {result:.1f}")
    return round(result, 1)

When the agent runs tests with pytest -v -s, it sees the log output alongside test results. This helps it debug issues that aren’t obvious from the traceback alone.

python -m pytest tests/ -v -s --log-cli-level=DEBUG

Teach the Agent to Read Logs¶

Include this in your prompt:

When tests fail:
1. Read the full pytest output including any print/log statements
2. Identify the root cause (not just the symptom)
3. Check git diff to see what you changed last
4. Fix the root cause, not just the failing line
5. Re-run the full test suite, not just the failing test

This trains the agent to debug systematically rather than guessing.

Phase 5: The Human-Agent Outer Loop¶

The agent’s inner loop — write, test, fix, repeat — is powerful, but it operates in a bubble. The agent sees its own tests passing and declares success, but those tests were written by the same reasoning that wrote the code. A shared misconception can propagate through both, creating false confidence.

This is where you enter as the critical human layer. Phase 5 introduces two roles you alternate between: the skeptical student who learns the code, and the evidence-driven teacher who diagnoses failures and instructs the agent to fix them. Together, these roles form an outer loop between human and agent.

Play the Skeptical Student¶

When the agent says it’s done, don’t accept “all tests passed” at face value. Instead, put on your student hat and try to genuinely understand what the code does — as if you’re encountering it for the first time.

Questions to ask as a student:

• What does this function actually do, step by step? — Trace through the logic with a concrete example. Don’t rely on docstrings; read the code.

• What happens if I give it unusual input? — Empty lists, extreme values, None, strings where numbers are expected. The agent’s test cases cover the spec, but not necessarily the world.

• Does the output make sense? — Run it interactively and eyeball results. A function that always returns a plausible-looking number isn’t necessarily correct.

• What side effects might I be missing? — File writes, network calls, global state changes that no test checks for.

Concrete exercises:

# Instead of just running pytest, try the code yourself:
from pybmi import bmi, classify

# What does this do?
print(bmi(70, 175))       # 22.9 — looks right
print(bmi(0.001, 175))    # What happens with nearly-zero weight?
print(bmi(70, 1))         # Height of 1 cm — should this be caught?
print(bmi(-70, -175))     # Both negative — does validation catch this?
print(classify(24.9))     # Boundary case — which category?
print(classify(25.0))     # Adjacent boundary — does it flip correctly?

The goal isn’t to find bugs specifically — it’s to learn the code well enough that when it does break, you understand why rather than treating it as a black box.

When Things Break: Be the Teacher, Not the Bystander¶

Eventually your skeptical exploration will find something wrong. An off-by-one error. A boundary case that crashes. A function that silently returns None instead of raising.

The mistake most people make: immediately telling the agent “fix it” and letting the agent’s inner loop run again. This is the equivalent of a student who doesn’t understand why a math answer is wrong, just erases it and guesses again.

The better approach: play the role of a teacher. Before asking the agent to fix anything, invest time to understand the failure yourself.

Step 1: Reproduce the failure cleanly

Don’t describe the bug vaguely (“it sometimes crashes”). Write a minimal reproduction:

# Minimal failing case
from pybmi import classify
result = classify(18.5)
print(f"classify(18.5) = {result!r}")
# Expected: "normal" (WHO says 18.5 is the lower bound of normal)
# Got: "underweight" — the comparison is wrong

Step 2: Gather evidence

Collect everything that helps pinpoint the root cause:

• Logs — Run with pytest -v -s --log-cli-level=DEBUG to see the agent’s debug output

• Diff — Run git diff to see what changed. The bug may have been introduced in a specific commit.

• Traceback — Copy the full stack trace, not just the last line

• Input/output — Record exactly what you fed in and what came out

Step 3: Form a hypothesis

Before opening a chat with the agent, articulate what you think went wrong:

Hypothesis: classify() uses <= instead of < for the underweight
boundary. The WHO defines underweight as BMI < 18.5, but the code
appears to treat 18.5 itself as underweight, suggesting the comparison
operator is off by one direction.

Evidence:
- classify(18.4) returns "underweight" (correct)
- classify(18.5) returns "underweight" (incorrect — should be "normal")
- classify(18.6) returns "normal" (correct)
- The threshold at 18.5 is the only one that flips wrong

This is not a formality. Formulating a hypothesis forces you to understand the code rather than treating the agent as a magic box that fixes things if you ask nicely.

Step 4: Write a clear bug report for the agent

Now — and only now — hand the diagnosis to the agent. Structure it like a teaching moment, not a complaint:

Bug found: classify() uses wrong comparison for BMI=18.5 boundary.

What I expect: classify(18.5) should return "normal"
What I got: "underweight"

Evidence:
- classify(18.4) = "underweight" (correct)
- classify(18.5) = "underweight" (WRONG — boundary is inclusive)
- classify(18.6) = "normal" (correct)

Suspected root cause: The condition `if bmi < 18.5` should be
`if bmi <= 18.5` — or more likely, the 18.5 threshold comparison
uses the wrong operator direction.

My hypothesis: The code uses `<=` where it should use `<`, because
WHO defines underweight strictly as BMI < 18.5.

Please:
1. Quote the classify() function so I can verify the fix
2. Add a test case for classify(18.5) that asserts "normal"
3. Run the full test suite after the fix

Notice the structure:

• Observation — what you actually saw

• Evidence — concrete data points

• Hypothesis — your reasoning about the cause

• Requested action — specific steps for the agent

This turns the agent from a guesser into a taught implementer. It knows exactly what went wrong, why you think it went wrong, and what to do about it.

Diagnose the Root Cause, Not Just the Symptom¶

Even when you play the teacher role well — reproducing the failure, gathering evidence, writing a clear bug report — there is a common trap. The agent will often fix only the symptom rather than the true root cause. This happens because the agent’s training optimizes for “make the test pass,” not “understand the underlying design flaw.”

Why this is costly:

• Wasted time and computation — The agent applies a patch, the test passes, but the same class of bug reappears elsewhere. You go through another outer loop cycle only to discover the first fix was superficial.

• Technical debt accumulates — Each symptomatic fix adds special-case logic that masks the real problem. Later fixes build on earlier patches, creating a fragile web of workarounds.

• Reverting becomes harder — Once the agent has woven a symptomatic fix into multiple files, undoing it requires untangling changes across a broader codebase than if you had caught the design flaw on the first pass.

The habit to build: before accepting the agent’s fix, ask whether it addresses the pattern that caused the bug, not just the instance.

Concrete example — a flawed data validation design:

Suppose you’re building a student grade management system. The agent implements a function to calculate the final grade from a list of scores:

def calculate_final(grades: list) -> float:
    return sum(grades) / len(grades)

As the skeptical student, you discover:

calculate_final([])  # ZeroDivisionError — empty list crashes

You report it to the agent. First fix (symptom):

def calculate_final(grades: list) -> float:
    if len(grades) == 0:
        return 0  # ← patch for the crash
    return sum(grades) / len(grades)

Test passes. You move on. Weeks later, a student reports: “My grade shows 0, but I have scores entered.” Tracing reveals that somewhere else in the code, a filtering step accidentally produced an empty list, and the function silently returned 0 — masking a data flow bug upstream. The symptomatic fix (return 0) turned a loud error into a silent miscomputation.

Now the agent adds another patch:

def calculate_final(grades: list) -> float:
    if len(grades) == 0:
        logger.warning("No grades provided, defaulting to 0")
        return 0  # ← now with a warning, but still wrong
    # ... special case for all-None values ...
    # ... special case for mixed types ...
    return sum(g for g in grades if g is not None) / count

Each ad-hoc fix handles one symptom. The function grows with special cases. Meanwhile, the real design flaw was never addressed: there is no single source of truth for what constitutes valid grade data. Different parts of the system pass [], [None], or malformed lists because nothing enforces a consistent data contract.

The root-cause fix would be different:

class GradeRecord:
    """A validated collection of grades with a clear contract."""
    def __init__(self, student_id: str, scores: list[float]):
        self.student_id = student_id
        if not scores:
            raise ValueError(f"No scores for student {student_id}")
        self.scores = [s for s in scores if s is not None]
        if not self.scores:
            raise ValueError(f"All scores None for student {student_id}")

    @property
    def final_grade(self) -> float:
        return sum(self.scores) / len(self.scores)

Now calculate_final([]) is impossible — the data is validated at the boundary, not patched at the calculation site. Every downstream function receives a guaranteed-valid GradeRecord. One design decision eliminated all the special cases.

How to push for the root-cause fix as the teacher:

The empty-list crash in calculate_final() is not just a missing
guard clause — it reveals that invalid data is flowing through
the system without validation at the source.

Before patching the symptom, investigate:
1. Where does the empty list originate? Is it a data loading bug,
   a filtering error, or a missing input?
2. Should validation happen at the data entry point rather than
   inside the calculation function?
3. Would a data class or schema (e.g., pydantic BaseModel) enforce
   the grade contract so the calculation function never receives
   invalid data?

Propose a fix that prevents invalid data from reaching the
calculation, not just a guard clause that returns a default value.

What this means for the outer loop:

The teacher’s job isn’t just to diagnose what broke — it’s to diagnose why the design allowed it to break. A teacher who only patches symptoms produces a student who can pass tests but doesn’t understand the material. A teacher who addresses root causes produces code that is simpler and more robust, not just correct on today’s test cases.

When Both You and the Agent Are Stuck: Instrument the Code¶

Sometimes even the teacher doesn’t know what’s wrong. You’ve reproduced the failure, gathered evidence, and formed a hypothesis — but the hypothesis is wrong, and the agent’s fix makes things worse. Or worse yet, you can’t even form a hypothesis because the failure is intermittent or the code path is opaque.

Dead end pattern:

You: The function returns the wrong result but I can't tell why.
Agent: Fixed the comparison operator. (still wrong)
You: Still broken. Fix it.
Agent: Refactored the if/elif chain. (still wrong, now more confusing)

The agent is guessing in the dark, and so are you. At this point, stop trying to fix and start trying to see.

Step 1: Ask the agent to add extensive tracing

Instead of another blind fix, instruct the agent to temporarily instrument the code so it prints exactly what’s happening at every decision point:

We can't find the root cause of the classify() bug. Instead of
fixing it, add detailed tracing so we can see exactly what
happens inside the function.

Add print statements or logger.debug() calls at:
1. Function entry — log all input arguments and their types
2. Every comparison — log the value being compared and the result
   (e.g., "Comparing 24.9 < 25.0: True")
3. Every branch taken — log which if/elif/else path was chosen
4. Before returning — log the final result

Do NOT fix the bug yet. Only add tracing. Then run:
  python -c "from pybmi import classify; classify(24.9)"
  python -c "from pybmi import classify; classify(25.0)"
and show me the trace output for both.

Step 2: Read the trace as a student

The trace output is like watching a debugger step through the code line by line. Read it carefully:

>>> classify(24.9)
[DEBUG] classify() called with bmi=24.9 (float)
[DEBUG] Comparing 24.9 < 18.5: False
[DEBUG] Comparing 24.9 < 25.0: True
[DEBUG] Branch: returning "normal"
[DEBUG] Returning: "normal"
Result: 'normal'  ← correct

>>> classify(25.0)
[DEBUG] classify() called with bmi=25.0 (float)
[DEBUG] Comparing 25.0 < 18.5: False
[DEBUG] Comparing 25.0 < 25.0: False  ← here's the problem
[DEBUG] Comparing 25.0 < 30.0: True
[DEBUG] Branch: returning "overweight"
[DEBUG] Returning: "overweight"
Result: 'overweight'  ← also correct, but...

>>> classify(25.0000001)
[DEBUG] classify() called with bmi=25.0000001 (float)
[DEBUG] Comparing 25.0000001 < 18.5: False
[DEBUG] Comparing 25.0000001 < 25.0: False
[DEBUG] Comparing 25.0000001 < 30.0: True
[DEBUG] Branch: returning "overweight"
[DEBUG] Returning: "overweight"
Result: 'overweight'  ← now the boundary issue is visible

The trace reveals something the source code alone didn’t: the comparison logic for 25.0 depends on floating-point precision, and the boundary behavior is sensitive to tiny rounding differences. The bug isn’t the comparison operator — it’s that the spec uses thresholds that conflict with floating-point arithmetic.

Step 3: Now the teacher has evidence

With the trace, you can form a real hypothesis:

Root cause identified via trace: The classify() thresholds use
exact floating-point comparisons (e.g., bmi < 25.0), but BMI
values are computed from weight/height and can have floating-point
representation errors. A BMI of exactly 25.0 might be stored as
24.99999999999 or 25.00000000001, causing inconsistent boundary
behavior.

Fix: Round the BMI value to 1 decimal place before classification,
or use a small epsilon in comparisons. The spec says BMI should be
"accurate to one decimal place" — we should classify the rounded
value, not the raw float.

Step 4: Clean up and fix properly

Now instruct the agent to apply the real fix and remove the tracing:

Root cause: floating-point boundary sensitivity in classify().

Fix:
1. Round the BMI value to 1 decimal place before classification
   (matches spec: "accurate to one decimal place")
2. Add test cases for values at each WHO boundary ± small epsilon:
   classify(24.99), classify(25.00), classify(25.01)
3. Remove all the debug print statements
4. Run the full test suite
5. Quote the fixed classify() function so I can verify

When to use this approach:

• The agent has tried 2+ fixes and the test still fails

• You can’t form a hypothesis from the test output alone

• The bug is intermittent or depends on floating-point / ordering / timing

• The code path is complex (multiple functions, data transformations)

• You suspect the bug is in data flow rather than logic (e.g., wrong value arriving at a correct comparison)

What NOT to do when stuck:

Tracing turns a guessing game into a detective investigation. The agent isn’t smarter with a trace — but it has information it didn’t have before. Information beats intelligence every time.

The Outer Loop: Alternating Teacher and Student¶

Put it together, and you get this pattern:

┌───────────────────────────────────────────────────────┐
│                    OUTER LOOP                         │
│                                                       │
│  Human (Student) ──► Examines agent's code            │
│  ──► Finds a bug or gap                               │
│                                                       │
│  Human (Teacher) ──► Diagnoses the issue              │
│  ──► Writes clear bug report with evidence            │
│  ──► Hands it to the agent                            │
│                                                       │
│  ┌── INNER LOOP (agent) ─────────────────────┐        │
│  │  Agent writes code ──► Tests ──►          │        │
│  │  Fixes ──► Commits ──► Reports            │        │
│  └───────────────────────────────────────────┘        │
│         │                                             │
│         ├─► Works? ──► Back to Human (Student)        │
│         │          ──► Repeat                         │
│         │                                             │
│         └─► Still broken? ──► Instrument the code     │
│                               (add tracing/logs)      │
│                           ──► Read trace              │
│                           ──► New hypothesis          │
│                           ──► Retry inner loop        │
└───────────────────────────────────────────────────────┘

Each cycle of the outer loop has two halves:

Half 1 — You are the student. Learn the code. Explore it. Break it. Don’t trust passing tests. Treat the agent’s output as a first draft, not a finished product. This is active learning, not passive acceptance.

Half 2 — You are the teacher. You found something wrong. Before speaking to the agent, diagnose it yourself. Gather evidence. Form a hypothesis. Write a clear report. You are teaching the agent what to fix and why.

When both are stuck — you are the detective. The hypothesis was wrong. The agent guessed and failed. Stop the cycle: instrument the code, read the trace, form a real hypothesis backed by runtime evidence, then restart the inner loop. This is the emergency exit from a dead-end guessing spiral.

Why this matters:

• Without the student half, you can’t spot problems. You’re accepting the agent’s work blindly, and subtle bugs accumulate.

• Without the teacher half, you can’t communicate problems effectively. You tell the agent “it’s broken” and get the same buggy output back, because the agent has no direction.

• Without the detective half, you and the agent waste cycles guessing in the dark. Tracing breaks the loop by replacing guesses with data.

All three roles are essential. The student finds the gap. The teacher closes it. The detective illuminates the dark corners where neither of them can see.

What a Full Outer Loop Looks Like¶

Round 1 — Agent delivers, human examines:

Agent: All tests passed. Implementation complete.
You: Let me check... BMI formula looks correct.
     But classify(24.9) = "normal" and classify(25.0) = "overweight"...
     Wait, classify(29.9) = "overweight" but classify(30.0) = "overweight" too?
     That's wrong — 30.0 should be "obese".

Round 2 — Human diagnoses, teaches agent:

Bug: classify(30.0) returns "overweight" instead of "obese".

Evidence:
- classify(29.9) = "overweight" (correct)
- classify(30.0) = "overweight" (WRONG)
- classify(30.1) = "obese" (correct)

Hypothesis: The obese threshold uses `> 30` instead of `>= 30`.
The WHO defines obese as BMI >= 30, so 30.0 exactly should
trigger "obese", but the strict greater-than comparison misses it.

Fix: Change `if bmi > 30` to `if bmi >= 30` (or restructure
the if/elif chain so the final else catches it).

Round 3 — Agent fixes, human verifies:

Agent: Fixed the comparison. Added test case for 30.0. All tests pass.
You: Checking... classify(30.0) now returns "obese". Good.
     But wait — did it fix the 18.5 issue too? Let me check...
     classify(18.5) still returns "underweight". That's the same
     class of bug. I'll open another ticket for that.

Each round is a teaching moment. The human learns the code. The agent learns from the human. Both get better.

Phase 6: Publish Your Package¶

When your package is solid — tests pass, docs are reviewed, and you’re satisfied — consider publishing it so others (and other AI agents) can use and extend it.

Choose a Meaningful Name¶

The name is the first thing people see. A good name communicates purpose and memorability in one glance.

Option 1: A name with symbolic meaning — Pick something whose etymology reflects what your tool does. For example, Hermes is the Greek god who moves freely across boundaries (heaven, earth, underworld) — and Hermes Agent can be deployed across different environments: JupyterHub, standalone servers, educational quiz platforms, and more. The name tells a story about the tool’s core capability.

Option 2: Credit the ecosystem you’re building on — When your package integrates two existing tools, include both names. This makes it discoverable and gives proper attribution. Examples:

• jupyter-ai-hermes — integrates Hermes Agent into Jupyter AI

• jupyter-hermes-proxy — uses jupyter-server-proxy to run the Hermes dashboard

• jupyterhub-litellm — integrates LiteLLM proxy into JupyterHub

These names tell you exactly what they do before you read the description.

Document Your Package: Four Must-Haves¶

When publishing, remember to include these four elements in your documentation:

a) Motivation with concrete example use case(s)

Don’t just describe what your package does — explain why it exists. Give real scenarios:

• What problem were you solving?

• Who faces this problem?

• What was the painful workaround before your package?

Example:

“Setting up an LLM proxy for a JupyterHub cluster with 200 students means managing API keys, rate limits, and model routing manually. jupyterhub-litellm deploys LiteLLM as a single-user-spawned proxy so each student gets their own model gateway — zero hub admin configuration needed.”

b) User guide — solve the problems from the use cases above

Walk users through solving the exact problems you described in your motivation:

• Installation instructions (pip, conda, etc.)

• Quick start — the 5-minute example that works

• Feature walkthroughs matching each use case

• Configuration options and defaults

• Troubleshooting common issues

c) Developer guide — help other AI agents pick up the project

Enable others — human or agent — to continue spec-driven development on your project:

• Architecture overview — module structure, data flow, key design decisions

• How to run tests locally (pytest, mypy, linting)

• How the spec-driven workflow was used (include SPEC.md or link to it)

• How to add new features following the established pattern

• CI/CD setup (GitHub Actions, etc.)

• Coding conventions and style guide

When your developer guide is thorough, another AI agent can clone the repo, read the docs, and start implementing new features using the same spec-driven approach — no human hand-holding needed.

d) Demo — let users try it before installing

For pure Python packages, a Jupyter notebook in a JupyterLite site is ideal — it runs entirely in the browser, requires no server setup, and can be hosted as a static site on GitHub Pages:

1. Ask the agent to create a Jupyter notebook that demonstrates your package’s key features

2. Build a JupyterLite site: pip install jupyterlite then jupyter lite build

3. Deploy the _build directory to GitHub Pages

Now anyone can interact with your package in their browser — no Python installation, no API keys, no dependencies to resolve.

For packages that need real servers or databases, consider a live demo on platforms like Binder, GitHub Codespaces, or a deployed sandbox.

Check Your Publish Readiness¶

Before publishing, ask the agent to run through this checklist:

Before I publish, verify:
1. README.md explains the motivation with concrete use cases
2. docs/USER.md solves the problems from those use cases
3. docs/DEVELOPER.md helps contributors (human or AI) extend the project
4. A demo notebook exists and builds successfully
5. pyproject.toml has proper metadata (name, version, description, author, license)
6. All tests pass and coverage is above 80%
7. git diff shows no debug statements or TODOs left in code
8. CHANGELOG.md describes what's new

Putting It All Together¶

The Complete Workflow¶

Here’s the full spec-driven workflow in one view:

1. Chat — discuss requirements with the agent, focus on the end-user experience

2. Spec — lock requirements into SPEC.md with acceptance criteria

3. Scaffold — create repo, pyproject.toml, environment, and failing tests

4. Implement — agent writes code, runs tests, reads errors, fixes bugs

5. Examine — play the skeptical student: learn the code, find gaps, break things

6. Teach — diagnose failures yourself, gather evidence, write clear bug reports

7. Iterate — hand the diagnosis to the agent, run another inner loop, repeat from step 5

The inner loop (agent: write → test → fix) handles known problems against known tests. The outer loop (human: examine → diagnose → teach) discovers and resolves the unknown problems that slip through. Both are necessary for production-quality code.

Comparison with Casual Prompting¶

• Input — Casual: “Write me a BMI calculator”. Spec-driven: SPEC.md with acceptance criteria.

• Feedback — Casual: You read the code manually. Spec-driven: Automated tests + logs.

• Errors — Casual: You fix them yourself. Spec-driven: Agent reads test output and fixes.

• Tracking — Casual: Copy-paste or manual saves. Spec-driven: Git commits after each change.

• Quality — Casual: Depends on prompt luck. Spec-driven: Measured against spec criteria.

• Review — Casual: Trust that it works. Spec-driven: Skeptical student examines, evidence-driven teacher diagnoses.

• Scale — Casual: Works for small scripts. Spec-driven: Works for multi-file projects.

Spec-driven doesn’t mean you can’t ask casual questions — it means you have a safety net when building anything real. And the outer loop ensures that safety net catches the bugs the agent’s own tests miss.