Hidden Systems Aren't Complex, They're Just Hidden

I love diving into hidden systems. They've always been a fun puzzle for me to solve. Undocumented, mysterious, black boxes sitting in the corner of the office with the good old, "Don't touch," sticky note. Forensics and biology meet in a place with significantly fewer fluids. That's the dream.

When I was at DV, one of the first things I got ahold of was a half-done trading system that was left to me by a former colleague. It was largely a solo effort and a necessary but abrupt departure meant I was on my own. About 3 days in and the thing is dead, now I get to resurrect it. It's cool, not like all of the trade desks were looking at me while I worked or anything...

No docs, no prior experience, and I'm not even sure I have all of the permissions I need but I do have root access to the servers it runs on and bash history. Digging through different users' bash histories, running processes, systemctl configs, anything I can find to grok some more information about this beast. A few hours and a Red Bull led to a properly started trading system along with all of its supporting infrastructure and I felt like a hero.

After a couple of weeks of dissecting, reconfiguring, minor refactoring, and automation, we had a reliable trading system. I thought I felt like a hero before, our lead trader told me I did a good job. Talk about hero.

Being an "infrastructure" guy, I wound up in this position a few times later in my career too. CI/CD pipelines that do some genuinely heinous things, click-ops AWS accounts that were caught somewhere between Terraform and CloudFormation, and teams that didn't have time (or patience) to really dig in. All of these things get described as complex but rarely has that turned out to be true.

One of my big misses was a build system that I refactored earlier in my career. It started out pretty small and honestly reasonable. Over time it grew, and my docs got out dated, and I didn't have an LLM to write them for me, and I got swamped so I just gave up on the docs, and I was the only one who could work on the build system so I got even more swamped, and you see what I'm getting at, right? It worked for a time until it didn't and then I became the bottleneck.

What's worse, I made the system far more complex than necessary while I hacked away at my own echo chamber. While I was in solo mode I got comfortable not feeling the pressure to explain myself. It was a working system but built for just my thought process. I added workflows and abstractions that made sense to me (and only me).

This shows up everywhere, not just in the code. Promotion criteria that live in a manager's head, performance review expectations that are never written down, closed door budget decisions, strategy shifts with no context, 'cultural fit' that just means "already knows how we do things." All of these are black boxes with the same problem of storing inaccessible information. Even if it's available upon request, how often do you know to ask?

When systems are hidden, you create dependencies on individuals. The person who "just knows" how things work becomes a single point of failure. That's pressure most people don't want to live with, and it breeds exactly the resentment and risk you're trying to avoid. Systems become fragile, people burn out, and everyone else feels powerless. Or, well, at least I do.

If you're the person who knows how the system works, I'd argue that you have the responsibility to make it knowable. If you're the person stuck diving into it, then you share that responsibility too.

When you're digging through bash history, write down what you find. Turn a debugging session into a run book. Whatever it is, don't wait for it to be perfect, just write it down. A messy README beats no README every time and you can deal with staleness later. Don't just announce the decision, explain the thinking. Don't just push the code, document why you chose this approach. The "why" is what turns black boxes into shared knowledge.

There's an uncomfortable truth here too, a weird incentive to stay hidden. Job security, ego boost, immediate productivity. Writing docs feels like busywork when you could be shipping features.

I get it. I felt it too.

But being irreplaceable isn't job security, it's a trap. You can't grow if you can't be replaced. You can't take vacation if you're the only one who knows. You can't work on new problems if you're stuck maintaining your monopoly.

Now when I build something, I ask, "Could someone else fix this if I disappeared tomorrow?"

If the answer is no, I'm not done building.