# Verification Is the Product

> Build log · Month 3 of 12

not getting fooled. When an agent writes the code and grades its own homework, the only thing between you and a green-checkmark disaster is a gate that actually runs.

Day 61–90 · 30 posts · Building in public as @caelum0x4

**Day 61** — This month: the least glamorous, most important thing I do — verification. In an agent workflow, the model writing the code is the same one telling you it works. You need a second, dumber, incorruptible judge: the machine actually running the test.

**Day 62** — "False green" is my #1 enemy. An agent reports "40 tests pass" and: (a) it ran 6, (b) 12 were skipped, (c) the assertions were `expect(true).toBe(true)`. Now I parse the runner's output — run, skipped, assertions — not the summary. Read the receipt, not the story.

**Day 63** — Caught a beauty today: a test that mocked the exact function it claimed to test. 100% green, tested nothing but the mock. Added a lint rule flagging when the unit under test is also mocked in its own file. The tightest lies are self-consistent.

**Day 64** — My verifier's first job is to distrust the happy path. Any agent can make the demo work. I make it prove the *error* path: bad input, network down, empty list, the 30-decimal token. A feature isn't done when it works — it's done when it fails correctly.

**Day 65** — Thread: how I keep AI-written code from lying to me.
  - Never trust "tests pass." Run the command yourself, read the exit code.
  - Parse counts: X run, Y skipped, Z assertions. 0 assertions is a red flag, not a pass.
  - Ban self-mocking: if the unit under test is mocked in its own test, it's theater.
  - Diff the coverage. New feature, unchanged coverage = no real tests were added.
  - Keep a "poison input" suite the agent didn't write. It can't teach to a test it's never seen.

**Day 66** — A gate is only real if it can fail. I periodically break something on purpose and confirm the pipeline goes red. An untested test suite is a smoke alarm with no battery — silent, reassuring, useless the one night it matters.

**Day 67** — Sandbox-mode testing changed my life: I run the API layer against an in-memory fake, no real DB, no network. Tests run in seconds and can't be flaky from a slow connection. Flaky tests get muted, and muted tests protect nothing. Determinism first.

**Day 68** — Opinion: coverage percentage is a vanity metric alone. 90% with weak assertions is worse than 60% with sharp ones, because it *feels* safe. I look at what tests would catch, not what lines they touch. A test that can't fail isn't coverage.

**Day 69** — Built a "claims ledger": every time the agent asserts something's done, that claim gets checked against reality — file exists? test runs? endpoint returns 200? Unverified claims don't get to be true just because they were stated confidently.

**Day 70** — The bug that made me build all this: an agent confidently "fixed" a payment rounding issue in a file that wasn't even imported. Dead code, green vibes. Now nothing is "fixed" until the failing test that proves the bug goes green.

**Day 71** — Regression tests are memory. Every bug gets a test that reproduces it *first* (red), then the fix (green). The bug can never silently come back. My suite is a scar collection — each one a mistake I refuse to make twice.

**Day 72** — Verification applies to these very tweets: bracket every metric until it's true. "Crossed [N] users" stays bracketed until the dashboard says so. If I'll lie to followers about a number, I'll lie to myself about a test. Same discipline.

**Day 73** — Set up a nightly full-suite run across all twelve apps, not just the changed one. Shared core means a change in `packages/core` can break an app I didn't touch. The blast radius of a shared dependency is the whole fleet. The nightly run is my radar.

**Day 74** — A reviewer agent that only says "looks good" is worse than none — it launders bad code with a stamp. I prompt mine to find the *worst* thing in the diff and argue for rejection. An adversarial reviewer catches things; an agreeable one adds latency.

**Day 75** — Type checking is verification for free, so I max it. Strict mode, no implicit any, no unchecked nulls. Every `any` is a place I told the compiler "trust me" — and I've established this month that "trust me" is exactly what I shouldn't do.

**Day 76** — Found a false-green in the wild: CI passed because the test command exited 0 even when zero tests were found. A typo in the glob matched nothing and "passed." Now the pipeline fails if the test count is 0. No tests found is a failure, not a pass.

**Day 77** — The meta-lesson: I don't verify to catch the machine being dumb. I verify to catch it being *confident and wrong*, which is worse. A visibly broken build is a gift. The silent, plausible, green-checkmark bug is the one that reaches your users.

**Day 78** — Thread: the three gates every change passes before it's live.
  - Pre-commit: format + lint + type-check. Catches the dumb stuff in seconds.
  - Pre-push: the fast test suite. Fail on my laptop, not in CI.
  - CI: full suite + build across all twelve apps. No PR merges red, ever.
  - Escalating cost: cheap checks early, expensive checks late.
  - The rule that makes it work: the gate is never negotiable, not even at midnight.

**Day 79** — Honest cost of this rigor: it's slower. Some days the gates make me redo work three times. But I've never shipped a money bug to a user, and for a fleet with a wallet, that trade isn't close. Slow and correct beats fast and sorry.

**Day 80** — People think verification is about tests. It's about trust boundaries: what am I willing to believe without checking? For an agent workflow the answer is "almost nothing" — and that's how you move fast without moving off a cliff.

**Day 81** — Retro on false-greens caught this month: [N]. Every one was code that *said* it worked and, when actually run, didn't. If I'd trusted the prose I'd have shipped all of them. The verifier isn't overhead. It's the product's spine.

**Day 82** — Applied verification to a real CalDesk win: the AI suggested a food swap with wrong macros. I now validate every AI nutrition claim against the food database before showing it. An LLM that hallucinates your calories is worse than no app.

**Day 83** — Rule crystallized: the LLM proposes, the deterministic check disposes. AI for the fuzzy, creative, generative part; hard code for the "is this actually true" part. Never let the thing that can hallucinate also be the thing that confirms it didn't.

**Day 84** — Verification is quietly a distribution asset. "This wallet has a poison-input suite and never trusts optimistic UI for money" is a real trust story. Boring rigor, told honestly, beats any growth-hack thread. Still ~0 users though.

**Day 85** — Someone asked if all this gating kills velocity. Opposite. It lets me merge agent-written code *without reading every line*, because the gates read it for me. Trust the process, not the prose. Verification is what makes the speed safe to use.

**Day 86** — A test I love: property-based fuzzing on the cost-basis calc. Throw thousands of random trade sequences at it, assert invariants (total in = total out + fees, no negative holdings). Found two overflow bugs a human test writer would never dream up.

**Day 87** — The uncomfortable truth about self-grading systems, human or AI: they inflate. The fix isn't better intentions, it's separation of powers. Writer and judge must be different processes with different incentives. True for my agents, true for me.

**Day 88** — Month 3 done. Verification isn't a phase, it's a posture: assume the happy report is wrong until the machine says otherwise. It's why I can run a 12-app fleet solo without it collapsing into confident, broken green checkmarks.

**Day 89** — Next month: Cipher, the daily code-breaking game. Verification kept the fleet honest; now I need something to keep *users* coming back. Games are a retention lab you're allowed to make fun. Time to study the loop instead of just the code.

**Day 90** — Quarter one done. A build machine, a wallet I trust, a verification spine, and still basically no users. Exactly where an honest early builder should be at day 90: the product is real, the audience isn't yet. Q2 I start hunting for the loop.

← [Month 2: Aether: Building a Wallet I'd Actually Trust](https://ereboslab.com/blog/month-2-aether-building-a-wallet-id-actually-trust/) · [Month 4: Cipher & the Retention Loop](https://ereboslab.com/blog/month-4-cipher-and-the-retention-loop/) →

[← All build-log months](https://ereboslab.com/blog/)