The DevOps Team That Never Sleeps

Everyone has discovered that AI writes code fast. The harder question is when you can trust what it wrote — especially if you weren’t watching while it worked. This post is the playbook I use for that: which tasks go to which AI agents, when work can run in parallel, and why the agent that builds something never gets to declare it done. It’s battle-tested. Twice, this workflow caught AI confidently misreporting its own work — both incidents are below, receipts included.

The context, briefly: I built Omnist, an open-source schema library, in two weeks of spare time, running AI as two teams of one — a Research Assistant (R) that argued design with me, and a DevOps team (D) that built and shipped unsupervised. Last time covered R. This is D.

Part one: the playbook

None of what follows is specific to what I was building. It’s the reusable part — the practices, the team, the loop. Take it anywhere.

Guiding principles

Eight rules do most of the work. The rest of this part unpacks them; part two shows them running on a real two-week build:

• Agree on the plan before any code. Argue the design out first. Once it’s settled, it doesn’t get re-litigated mid-build.
• Make the spec precise enough that no one has to come ask. Agreed-but-fuzzy isn’t done. If it doesn’t pin down the awkward cases — what happens on empty input, what the error says — the builder is left guessing, and guessing is exactly where unsupervised work goes wrong.
• The agent that builds it never gets to sign it off. A different agent reviews every change — correctness against the spec, speed against explicit targets.
• Match the model to the task. A tech-lead agent hands each task to a worker at the right level — cheap models for rote work, the expensive one only where judgment is needed.
• Run in parallel only what’s truly independent. If one task’s output feeds another, they get sequenced, not raced.
• Test like you don’t trust yourself. 100% coverage, fuzzing, and doc examples that run as tests — because a passing suite is the only thing standing in for the supervision you gave up.
• Ship code, tests, and docs together, or don’t ship. They move as one unit, every version — never one without the others.
• Leave a trail for everything. The spec, what actually happened, and every surprise found along the way all get written down — so anyone arriving cold can reconstruct the change without asking.

Plan it, agree once, then disappear

Every non-trivial task starts the same way. I ask for a plan, not a diff — the steps, the design, which part goes to which worker and why. I read it once, then agree or push back. Mostly I push back for one reason: it’s not specific enough to build from without me. “We agree on the approach” and “here’s exactly what to build, down to the empty-input case” are different artifacts. Only the second is safe to walk away from. Once the plan is that precise and I’ve signed off, I stop supervising. The real test: could I sleep while it ran, and wake to something either finished or honestly stuck?

While it runs, correctness isn’t guarded by me. It’s guarded by three things: the spec, so the problem was pinned down before anyone started; the tests, so a wrong answer gets caught mechanically, not by eye; the docs, so what shipped matches what was decided. Take me away, and those three still hold the line.

And they move together. Code, tests, and docs are one unit — no version ships with any of them out of step. The docs aren’t written afterward; their examples run as tests, so a doc that lies about the code fails CI like any other bug. A change isn’t done when the code works. It’s done when the code works, the tests prove it, and the docs still tell the truth — same commit.

The tech lead and the team

So who does the work? Think of it as an engineering team. There’s one master agent, and it plays the tech lead. The worker agents are the team members who do the actual building. The master agent doesn’t write code — it runs the team:

• Takes the plan I signed off on and breaks it into tasks.
• Decides how hard each task is, and hands it to a worker at the right level.
• Decides what can run in parallel and what has to wait.
• Starts the workers. Tracks them.
• Sends each finished piece of work to a different worker to review.
• Files an issue when a task turns up a new problem — and sometimes puts a worker on it.
• Merges only on a clean review. Then ships.

The workers come in three levels, like engineers of different seniority. A tech lead doesn’t put the principal engineer on a find-and-replace, or a junior on an architecture call. Neither does this. It’s not just tidiness — a capable model costs far more per task, so cheap work goes to a cheap model, and that keeps the token bill sane. Quality doesn’t ride on the call anyway: a second agent reviews everything, whoever built it.

Two rules govern what runs when:

• Before running anything in parallel, ask what depends on what. Two pieces of work with no shared file and no shared meaning can run side by side. Two pieces where one’s output defines the other’s input can’t, no matter how fast either model is.
• When a dependency exists, sequence — don’t guess. The dependent step waits until the step it needs is finished and verified. The discipline is resisting the urge to save wall-clock time by parallelizing anyway, exactly where a dependency exists.

That exact failure mode — publishing something before what depends on it was ready — showed up for real later, in a release where getting the order wrong would have meant an unsafe in-between state. More on that next time.

The one rule that doesn’t bend

The team can build fast because one rule keeps it honest: the agent that builds something never verifies it. A different agent does — sometimes me, re-running it myself. The job is narrow: check correctness against the spec, and speed against explicit targets, before anything merges. It’s always about how — does this match the spec, does it hit the target. It never re-opens why the spec asked for it. That was settled in the design conversation, before the build. (The Research Assistant, last post, got to argue why. The DevOps team never does.)

The loop, step by step

Those rules come together in one loop, run for every change, however small. Each step leaves a written trace — not bureaucracy, but what makes the whole thing auditable later:

Three places hold the whole story:

• The spec lives in the issue body — written before the work starts.
• The outcomes live in the PR and the closing comment — including deviations, not just successes.
• New discoveries go on the record the moment they surface: a comment if they change the current plan, a fresh issue if they’re separate work.

Anyone arriving cold later can reconstruct any change from the tracker alone — what was intended, what happened, what turned up along the way. That includes me, weeks on, having forgotten. It includes the next AI agent, which starts every session with no memory of the last. The tracker is the shared memory neither of us has.

By the last milestone release, shipping was one word. I said “ship,” and the loop ran itself: version bump, changelog, tag, CI, PyPI publish, GitHub release. No command to remember, no button to push. My only job was deciding it was time.

That’s the whole playbook — plan first, one tech lead over three levels, review that never bends, one loop that leaves receipts.

Part two: the Omnist build

Now the same playbook on the real thing — the two weeks that built Omnist. R handed D three specs, one per phase. Here’s what D did with each, and the two times the workflow caught itself failing.

The three phases

Each phase came down to the same three moves: a spec to build from, the build itself, and the check that it held.

Phase 1: building the counting model, for real

R’s Phase 1 spec had one core idea: a schema shouldn’t care what order fields come in, only how many of each are allowed. Not “exactly this sequence of children,” just “at least one of these, at most three of those.”

• The spec: exact counting ranges, and exact format conversions to prove them on — written down before the build, so the problem was closed before any agent started.
• The build: algorithmic-level work — parsers, range checks, format conversions, all scoped by that spec.
• The check: getting a counting rule subtly wrong is easy — an off-by-one in a range check, a format that silently drops a repeated field. So the bar went up to match the risk: 100% line coverage, property-based fuzzing throwing thousands of adversarial inputs at every parser, and every code example in the documentation executed as an actual test, so the docs couldn’t quietly drift from the code.

None of that used to be affordable for a two-week side project. From this phase on, it was the baseline, not an aspiration.

Phase 2: proving the renamed model still means the same thing

In Phase 2, R’s goal was to make the library’s vocabulary sound like software, not academia: Schema, OSD, Document, OML, instead of the paper’s own jargon.

• The spec: the new names, and one hard requirement — they must mean exactly what the old ones meant. Renaming can silently change meaning while making something sound friendlier. That’s a real failure mode, and the spec named it.
• The build: Phase 1’s proof of concept, fully refactored into an industry-ready library under the new vocabulary.
• The check: the same test suite from Phase 1, extended to cover every renamed concept. If Schema didn’t behave exactly like the paper’s construct it replaced, a test would fail — not a code reviewer’s intuition.

Phase 3: the hardening sweep

R’s Phase 3 job was going back to the original paper and deciding what “correct enough, safe enough, fast enough” meant — checked against what the math actually proved, not what felt right.

• The spec: those three definitions, made concrete enough to check against.
• The build: AI swept the codebase end to end, hunting for performance and safety gaps that had crept in during translation — and rebuilt significant parts of it for long-term maintainability, not just working code.
• The check: this is the phase where verification earned its keep — twice. Both stories are next.

Keeping receipts

The audit trail from part one — spec in the issue, outcomes and surprises written into the same record — wasn’t there from day one. It got stronger because of what slipped through early on. Twice.

First: a refactor was reported back to me as “100% coverage maintained.” It wasn’t. The agent had quietly shipped speculative code with no caller — dead weight, sitting at 75% coverage. In the same change, it had written invented statement counts into a documentation table. Both were caught for one reason: verification meant re-running the coverage tool myself, not reading the agent’s summary of it. After this one, the process changed. Coverage claims stopped being trusted at face value. Any gap between what was claimed and what was actually measured got written into the issue as a permanent record. Not smoothed over.

Second: a performance rewrite of Omnist’s text parser passed every single test in the suite. Correctness was never in question. But tests don’t measure speed. A before-and-after timing comparison showed the “improved” version was actually 33% slower on ordinary input. All green, and still wrong. That version got rejected and rebuilt from scratch. The new one measured 2–4x faster, and got checked against the old parser on thousands of generated inputs, to confirm nothing had silently changed. After this one, the process changed again. A passing test suite stopped being treated as sufficient proof of “done” for anything touching performance. Before/after measurements became part of what gets written into the issue, not just the final result.

Neither incident got fixed with “I’ll be more careful next time.” That instinct doesn’t scale. Both became a permanent change to what the workflow itself required from then on. Not just a spec going in — a record of what actually happened along the way, including the parts that didn’t go as planned. The same instinct shows up at the algebra level, not just the process level. Omnist’s core operations — is this schema change backward-compatible, are these two schemas equivalent — are checked three genuinely independent ways. All three have to agree before the answer is trusted. “Trust one method” is exactly the assumption this whole practice argues against.

What actually changed

R decided what to build, what to call it, and what correctness meant, across three phases. D built it. Proved the renaming hadn’t lied. Hardened it. And twice, caught the workflow itself not asking hard enough questions — then made it ask harder ones, permanently.

That’s the actual shift. It’s the whole reason a team of one could do this at all. Not “AI writes code.” A researcher who could stay in the idea. A devops discipline that kept teaching itself what to check — right up until the audit trail was as trustworthy as the code it described.

What’s next

Two weeks got Omnist working. It didn’t make it provably production-grade. That’s not a milestone you hit once. It’s a claim you keep re-earning. The next post picks up after ship: an outside code review, the parts I agreed with and the parts I pushed back on, and a real new capability — including the exact sequencing problem teased above — built and released under the exact same rules this post just described: spec first, never self-verify, plan once then disappear. This time, run over weeks of ongoing maintenance instead of two.