Build log · Month 3 of 12

Verification Is the Product

not getting fooled. When an agent writes the code and grades its own homework, the only thing between you and a green-checkmark disaster is a gate that actually runs.

Day 61–90 · 30 posts · Building in public as @caelum0x42

Day 61
This month: the least glamorous, most important thing I do — verification. In an agent workflow, the model writing the code is the same one telling you it works. You need a second, dumber, incorruptible judge: the machine actually running the test.
Day 62
"False green" is my #1 enemy. An agent reports "40 tests pass" and: (a) it ran 6, (b) 12 were skipped, (c) the assertions were expect(true).toBe(true). Now I parse the runner's output — run, skipped, assertions — not the summary. Read the receipt, not the story.
Day 63
Caught a beauty today: a test that mocked the exact function it claimed to test. 100% green, tested nothing but the mock. Added a lint rule flagging when the unit under test is also mocked in its own file. The tightest lies are self-consistent.
Day 64
My verifier's first job is to distrust the happy path. Any agent can make the demo work. I make it prove the error path: bad input, network down, empty list, the 30-decimal token. A feature isn't done when it works — it's done when it fails correctly.
Day 65
🧵 Thread: how I keep AI-written code from lying to me.
- Never trust "tests pass." Run the command yourself, read the exit code.
- Parse counts: X run, Y skipped, Z assertions. 0 assertions is a red flag, not a pass.
- Ban self-mocking: if the unit under test is mocked in its own test, it's theater.
- Diff the coverage. New feature, unchanged coverage = no real tests were added.
- Keep a "poison input" suite the agent didn't write. It can't teach to a test it's never seen.
Day 66
A gate is only real if it can fail. I periodically break something on purpose and confirm the pipeline goes red. An untested test suite is a smoke alarm with no battery — silent, reassuring, useless the one night it matters.
Day 67
Sandbox-mode testing changed my life: I run the API layer against an in-memory fake, no real DB, no network. Tests run in seconds and can't be flaky from a slow connection. Flaky tests get muted, and muted tests protect nothing. Determinism first.
Day 68
Opinion: coverage percentage is a vanity metric alone. 90% with weak assertions is worse than 60% with sharp ones, because it feels safe. I look at what tests would catch, not what lines they touch. A test that can't fail isn't coverage.
Day 69
Built a "claims ledger": every time the agent asserts something's done, that claim gets checked against reality — file exists? test runs? endpoint returns 200? Unverified claims don't get to be true just because they were stated confidently.
Day 70
The bug that made me build all this: an agent confidently "fixed" a payment rounding issue in a file that wasn't even imported. Dead code, green vibes. Now nothing is "fixed" until the failing test that proves the bug goes green.
Day 71
Regression tests are memory. Every bug gets a test that reproduces it first (red), then the fix (green). The bug can never silently come back. My suite is a scar collection — each one a mistake I refuse to make twice.
Day 72
Verification applies to these very tweets: bracket every metric until it's true. "Crossed [N] users" stays bracketed until the dashboard says so. If I'll lie to followers about a number, I'll lie to myself about a test. Same discipline.
Day 73
Set up a nightly full-suite run across all twelve apps, not just the changed one. Shared core means a change in packages/core can break an app I didn't touch. The blast radius of a shared dependency is the whole fleet. The nightly run is my radar.
Day 74
A reviewer agent that only says "looks good" is worse than none — it launders bad code with a stamp. I prompt mine to find the worst thing in the diff and argue for rejection. An adversarial reviewer catches things; an agreeable one adds latency.
Day 75
Type checking is verification for free, so I max it. Strict mode, no implicit any, no unchecked nulls. Every any is a place I told the compiler "trust me" — and I've established this month that "trust me" is exactly what I shouldn't do.
Day 76
Found a false-green in the wild: CI passed because the test command exited 0 even when zero tests were found. A typo in the glob matched nothing and "passed." Now the pipeline fails if the test count is 0. No tests found is a failure, not a pass.
Day 77
The meta-lesson: I don't verify to catch the machine being dumb. I verify to catch it being confident and wrong, which is worse. A visibly broken build is a gift. The silent, plausible, green-checkmark bug is the one that reaches your users.
Day 78
🧵 Thread: the three gates every change passes before it's live.
- Pre-commit: format + lint + type-check. Catches the dumb stuff in seconds.
- Pre-push: the fast test suite. Fail on my laptop, not in CI.
- CI: full suite + build across all twelve apps. No PR merges red, ever.
- Escalating cost: cheap checks early, expensive checks late.
- The rule that makes it work: the gate is never negotiable, not even at midnight.
Day 79
Honest cost of this rigor: it's slower. Some days the gates make me redo work three times. But I've never shipped a money bug to a user, and for a fleet with a wallet, that trade isn't close. Slow and correct beats fast and sorry.
Day 80
People think verification is about tests. It's about trust boundaries: what am I willing to believe without checking? For an agent workflow the answer is "almost nothing" — and that's how you move fast without moving off a cliff.
Day 81
Retro on false-greens caught this month: [N]. Every one was code that said it worked and, when actually run, didn't. If I'd trusted the prose I'd have shipped all of them. The verifier isn't overhead. It's the product's spine.
Day 82
Applied verification to a real CalDesk win: the AI suggested a food swap with wrong macros. I now validate every AI nutrition claim against the food database before showing it. An LLM that hallucinates your calories is worse than no app.
Day 83
Rule crystallized: the LLM proposes, the deterministic check disposes. AI for the fuzzy, creative, generative part; hard code for the "is this actually true" part. Never let the thing that can hallucinate also be the thing that confirms it didn't.
Day 84
Verification is quietly a distribution asset. "This wallet has a poison-input suite and never trusts optimistic UI for money" is a real trust story. Boring rigor, told honestly, beats any growth-hack thread. Still ~0 users though.
Day 85
Someone asked if all this gating kills velocity. Opposite. It lets me merge agent-written code without reading every line, because the gates read it for me. Trust the process, not the prose. Verification is what makes the speed safe to use.
Day 86
A test I love: property-based fuzzing on the cost-basis calc. Throw thousands of random trade sequences at it, assert invariants (total in = total out + fees, no negative holdings). Found two overflow bugs a human test writer would never dream up.
Day 87
The uncomfortable truth about self-grading systems, human or AI: they inflate. The fix isn't better intentions, it's separation of powers. Writer and judge must be different processes with different incentives. True for my agents, true for me.
Day 88
Month 3 done. Verification isn't a phase, it's a posture: assume the happy report is wrong until the machine says otherwise. It's why I can run a 12-app fleet solo without it collapsing into confident, broken green checkmarks.
Day 89
Next month: Cipher, the daily code-breaking game. Verification kept the fleet honest; now I need something to keep users coming back. Games are a retention lab you're allowed to make fun. Time to study the loop instead of just the code.
Day 90
Quarter one done. A build machine, a wallet I trust, a verification spine, and still basically no users. Exactly where an honest early builder should be at day 90: the product is real, the audience isn't yet. Q2 I start hunting for the loop.

← All build-log months