It started as a performance problem, which is how these things usually arrive.
The year was 2002. We were moving a call centre system off distributed client-server and onto one of the early online databases. Before, the branches exchanged data on physical media. Disks moved. Data arrived in batches, late, sometimes out of order, sometimes twice, occasionally not at all, and everything downstream had to cope with that.
Now it was one database, always on, and everybody wrote to it directly.
The new system was slow. Not catastrophically, but wrong-slow, in the way that tells you something is happening that should not be happening at all.
Three hours of not understanding
I went into the code and found reconciliation routines.
Checks for records in inconsistent states. Logic for two versions of the same thing arriving out of sequence. Repair paths for data that had gone stale in transit. Careful, thorough, expensive, and firing on every single operation.
In the old system they had run once, on import, asynchronously, out of everyone’s way.
I read them for about three hours and could not work out why they were there.
Not because they were badly written. They were good. Every check was checking for something that could genuinely go wrong, and the code knew exactly what to do about it. That was the confusing part. This was not the work of somebody who did not understand the system. It was the work of somebody who understood it extremely well.
Then it landed.
He was protecting us against disks.
There were no disks. There was no transit. There was no window in which data could go stale, because the window was the thing we had just deleted. Two people writing to the same record at the same time was not an unsolved problem requiring careful defensive reconciliation. It was a record lock, and the database had been doing it for us, correctly, for free, the entire time.
He had rebuilt the entire apparatus of a world that had ended, inside a system that had no use for it.
Legacy
There is something absurd in it, and I want to be careful about where the absurdity sits, because it was not on him.
Every one of those routines was correct once. Each one existed because something had gone wrong, in production, with real customers, and he had been the person who fixed it. Somebody’s weekend is in each of those checks. He had spent years learning, at cost, precisely what happens when data arrives late and out of order, and that knowledge was the most valuable thing he owned.
Then we changed the ground under him and asked him to keep building.
He did not sit down and decide to copy the old system. He reasoned, carefully, from what he knew, and what he knew was a set of constraints that had been true for a decade and had stopped being true a few months earlier. His logic was flawless. His premises were extinct.
That is the thing I keep running into, and I have run into it enough times now that I say it out loud to boards, usually to blank faces.
Legacy is not in the code. Legacy is in people.
Not because they are bad. Because the workarounds have become invisible to the people who built them. Every strange thing in an old system has a reason, and the person who knows the reason cannot see it as strange. The constraints that formed over years inside his head, technical and commercial, had become simply how the world was.
What I did
I took him off the core of the rewrite.
I want to be honest about how that looks from the inside, because it looks like a punishment. He had done nothing wrong. He was one of the strongest people I had. And from anywhere in the room, taking your best man off the most important part of the work reads as a judgment on him.
It was not. And saying so does not help, so I did not spend much time saying it.
What I did instead was give him the two things nobody else could do.
He wrote the edge cases. All of them. Every failure mode he had ever seen, every ugly thing that had ever happened to that data in ten years, went into the test suite. The exact knowledge that had made him wrong for the architecture was the only place in the company where that knowledge existed at all, and in the test suite it was not a constraint. It was the truth.
And he reviewed, but not the code. He reviewed business behaviour. Not “is this built well,” which other people could judge. “Is this still doing the thing it is supposed to do.” He caught things that were technically perfect and quietly wrong, because he was the only person who knew what right looked like.
The right person for the task
There is an old saying about putting the right person on the right job.
It is not wrong. It is just too big. A job is a large object, and it contains many things, and a person can be perfect for four of them and the wrong fit for the fifth.
What I actually needed was smaller. The right person on the right task.
The same man, on the same project, in the same week, was the wrong choice to design the architecture and the single best choice for the edge cases and the business review. Both of those things were true at once and neither one cancelled the other.
I have done this more than once since. The second time I saw it in minutes rather than hours, which is progress of a kind.
What has not changed is the argument I have to win to make it happen. Every time, somebody senior asks why the person who knows the system best is not the one building the new one. It is the most reasonable-sounding wrong question in the room, and it assumes expertise points in every direction at once.
It does not. Expertise has a direction. The man who knows where all the bodies are buried is exactly who you want guarding the ground, and exactly who you do not want deciding where to dig.
All these years on, that is still the hardest thing I have to explain in that room.