A little while ago a coworker pointed to the problem that he doesn’t understand how I come up with decisions, what my criteria are. There’s probably a million answers to that, but since the topic was system design, it got me thinking. What are my guardrails, my thought processes when it comes to designing systems of any size?
I went through a lot of specific examples in my head and reflected on the decisions I made or the beliefs I held in those situations. It was an interesting exercise, and it led me to learn two things about myself. The first is that I’m designing systems mostly based on a very small set of convictions – and that those convictions changed over time, at least to some extent. You could call also call those convictions my “system design core beliefs”. There will probably be more than one post on those. Today it’s about strong systems, and weak systems.
Systems, at least the more interesting ones, consist of multiple components. Sometimes two, sometimes a hundred. Each of those components has a different reason for why it exists. Some are databases, others are queues, some are backends, some are CDNs – they are different, and through the virtue of how they are built and ultimately connected, they form a system.
One of the mental models I apply to group components inside a system is to think of weak things and strong things. Let’s dive into strong things.
You want to build your system around some central ideas. That helps you both to have the right conversations early on – and it helps to create tons of clarity. It’s as much a vehicle for driving organisational progress as it is a technical driver. A central idea might be to store all data in one big Postgres, or a central idea might be that most inter-service communication is happening through Kafka.
Whatever you need to execute on your central ideas are things that need to be robust, strong, reliable, resilient. When all of your business data resides in a Postgres, this is where you don’t want to let the intern setup a Postgres server yourself. You want to make sure that those systems are as stable as they can possibly be. They are central, and being cheap on the foundation of your design is just not a particularly smart choice.
On the other hand, your weak things are everything that is non-critical. Things that are allowed to fail intermittently. If your site isn’t reachable for 3 minutes it’s not awesome, but it happens, and your business will likely not even notice the interruption. One of the fundamental freedoms of weak things is that they are just far less critical. They are the “break things” in “move fast and break things”. Everything that is not part of the central ideas is a weak thing.
Good systems are very intentional on what their central ideas are, and they make sure that two things are true at the same time: you’re strict about the strong things. Governance, great observability, on-call rotations, painful decision making processes and a rather slow pace of change - you name it. But you’re also really liberal about the weak things – they are the components where you can move fast, iterate and make progress.
So when you’re looking at what you’ve built, ask yourself: what was your central idea?