The System That Found Its Own Worst Bug

A system that stops working gets noticed the same day. A system that keeps working, keeps saying so, and loses most of what it was built to keep can run for months. The second one is the one worth writing about, because it was ours.

What it was supposed to do

CHIP exists to keep what a business learns. Every working session ends with a short record: what was decided, what got done, and what we learned the hard way. That record goes into a long-term log, so the next session starts from where the last one stopped instead of from zero.

The obvious way for that to fail is for it to stop recording. We watched for that. Every close-out printed a line saying the save was complete, and every close-out printed it.

What it was actually doing

When we finally read the log back against what the sessions had written, the numbers didn't match. Across our first few hundred sessions, 56% of everything recorded at close-out never reached the long-term log. For lessons, the part that's meant to stop us repeating a mistake, it was 94%.

Nothing had crashed. The save step ran, wrote what it wrote, and reported success. It was simply writing a fraction of what it was handed.

Why it happened

The cause was small and ordinary. The save step had a cap on how many items it'd write in one go, a sensible guard against a runaway session. The list it capped was built in a fixed order: decisions first, then what got done, then lessons. So whenever a session produced more than the cap, the cut fell on whatever came last. Lessons always came last.

Every part of that made sense on its own. Together they meant the system was most likely to forget exactly the thing it existed to remember, and least likely to say so.

What we changed

Three things, and none of them was clever.

• The part you can't replace goes first. Lessons are now built at the front of the list, so if anything gets cut, it isn't them.
• A cut gets counted out loud. If anything is held back, the run report says how many and which ones. A trimmed list that reports itself as complete is the actual bug, more than the trim.
• "Done" means it's in the log. A close-out now reads back what landed and compares it with what it was given. A success message on its own doesn't count as proof of anything.

The part that applies to your business

You don't need software for this to happen to you. It's the shape of most quiet failures in a small team: a process that runs, reports that it ran, and drops the same kind of thing every time without anyone deciding it should.

The new hire who files the paperwork but never passes on why a client gets a different rate. The handover document that covers every task and none of the exceptions. The weekly meeting where the decisions get made and the reasons don't get written down. Each one says "done." Each one is losing the most valuable part, and it's always the same part, because it's always the part that comes last.

The fix is the same one we used. Decide which part you can't afford to lose and put it first. Count what gets dropped instead of trusting the summary. And check what actually landed, not what the process says it did.

It's not glamorous. It's also the difference between a system that remembers and one that only says it does.