blog.moritzhaarmann.de

Matter of fact, I wasn't someone that I'd put in charge of a software project, like, 10 years ago. I'd gladly put me in that project as an IC, but certainly not in charge. Why? I think I got all the wrong ideas what actually matters overall.

When you start, it's about working on things that other people deemed important to be worked on – building a feature, building a feature in a certain way and so on. You're basically building experience by building small blocks, and then moving on to bigger blocks. That's a fairly normal progression – but your focus is on getting good at certain activities. Those activities vary, some are writing code, others are working on documentation, tests and so on – but all of them need to be done. But do they really?

There's probably three types of activities that regularly get done in most engineering groups – necessary things, interesting things, and valuable things. I don't want to spend a lot of time talking about necessary things – they are stuff like performing updates of critical dependencies, ensuring that whatever system exists is actually operational and so on – it's things you cannot put off for too long without having some bad effects.

Now, interesting things are something else. I recently was part of a discussion that focused on choosing a technical solution for a message broker. There were a few alternatives we discussed, one of them being Kafka, an industry standard for just that, and then some others. One of the options was writing one from scratch – and that's what I'd define as an interesting thing, and interesting activity.

If I could spend a few months and just write an enterprise message broker, that would be hugely interesting. It would probably also not be enough time, but that's another problem.

The biggest issue though is that, in an environment where established solutions already exist and are available, investing in building something that is at best equal in functionality is just not a particularly value-adding activity. It's fun, you might learn a lot, but you're spending a lot of time on something that can be achieved with a much smaller investment. Same result, orders of magnitude smaller invest.

Valuable Things. Those activities that get you closer to achieving the goal of what your team is working on at the moment. And that certainly doesn't mean just pushing for the fastest way from here to there – there are extremely valuable detours that you must take at times. But the crucial thing is to never really lose focus of what the actual goal is.

It's the one thing I struggle with the most at times in a leadership role - understanding that there is an incredibly interesting activity ahead – that we have to skip, because there's just very little value in performing it.

In a sense, software projects are the opposite of a road trip: The destination matters much more than the way there.

There's plenty of questions where I feel like I'm having good answers to. Software design, how to prioritize things, how to resolve conflicts, all the essentials of engineering management. The bigger the teams get that I'm responsible for, the more I'm wondering to what degree I should, and to what degree I actually must be hands-on - in order to be able to effectively lead a team.

Before I dive into my struggle of the day, I'd like to preface by saying that I generally do not believe in engineering leadership that is non-technical. I think you have to be familiar, sometimes intimately so, with the ins and outs of what your team is working on - and dealing with. While that certainly doesn't mean that you can only be effective if you're spending half the time working on and in the same environment as your team, some exposure is not nice to have - it's necessary. The opposite would be the only rely on the signals you're getting from the folks, and while you'll always want to be closely in touch with those, your own judgement needs to be built on your own first hand experience.

Back to the original question I keep thinking about. There's certainly no black or white answer, so the point I'd like to begin with is by looking at good reasons for being hands-on, and what the idea behind those good reasons is.

My first reason is to actually get a feeling. Getting a feeling for how easy (or hard) it is to work on a ticket or an issue is a great indicator on how well the overall system works. It's of course just one of many indicators, but building an impression from deep diving is incredibly valuable - especially when you're able to compare that to a plethora of previous projects.

That's not only a technical perspective - there's so much you can learn by trying to get something to production. Is it easy, is it hard? What's the process, and how does it feel like from an engineers perspective? Extremely valuable insights, right there - insights that can help make better decisions.

Another good reason is to understand what's going on. Jira boards, 1:1s and feature preparation meetings and docs are one part of the story - the commits and PRs flowing through a codebase are the other part. By having a closer look into what's actually flowing into the codebase, there's often a different perspective that's otherwise harder to get. Are there a lot of small PRs without planned work? Late night work that might otherwise not be too obvious? While I generally do not use metrics like number of contributions to judge anyone's performance, there's important data points that can be derived from the frequency, nature and quality of contributions. That's a very valuable source of information - one that shouldn't be discounted.

Now, being hands-on requires at least a solid level of understanding of the stack at hand, so there's some necessary investment there. Don't go crazy, but don't stay too shallow - just enough to make it work. I always find it to be super educational to spend time with my team on learning how they work - not only from a technical perspective, but for also building more deep connections.

So, those are my good reasons for staying hands-on, and there's a theme here - hands-on enough to understand, relate and be able to address the challenges the team is facing, as well as understanding how the team is actually working.

Keep in mind that this is one of many data points. And it's probably not the most important one.

How do you reason about how complicated software is?

We all strive to build maintainable and, hopefully, simple, yet powerful systems. And in our own ways, probably all the things that we help to create fulfilling our very own definition of that. I don't think there's a lot of folks who would build complicated stuff just for the sake of it - yet the amount of overly complicated solution out there is just staggering. And there's a bunch of really good reasons for why that happens - the most important probably being the 4th dimension, i.e. the effect of incremental change over time on any given design.

Even if you start with something simple, without proper stewardship, direction and someone saying no to wild ideas, even the best systems tend to invite technical debt, and that regularly manifests itself in a thing being more complicated than it should be.

And I'm going to pause here for a second. We need to clarify two terms before rolling on. The first term is that of complexity. It very closely means that a system with more components and interactions between those components is likely more complex than one with fewer components - it speaks to the intrinsic properties of a system or a problem. Complicated means extraneous elements that make dealing with something more difficult, without necessarily adding value.

Playing three games of chess at the same time might be complex, doing it while blindfolded is making it complicated.

One of the very first things I do when trying to understand really any system I'm dealing with is: How complicated is it? For that, I usually need to first understand the business domain, the building blocks and their connections. Once you understand the rough problem, it's all about determining what the delta between the necessary and the existing solution is. The bigger that difference, the more complicated your solution. There is no real world system where that difference is zero - at least I've never seen it.

But how do you actually quantify how complicated a system is, aside from judging it from an experience point of view and applying some gut measure metrics? I find that to be a really hard problem, as you're looking at plenty of different technologies at different flight levels at the same time - you're looking at code just as much as you're looking into a deployment model. You need to understand that having 3 services written in the same language, deployed using the same technologies is vastly different from having 3 services that are deployed in 3 entirely different ways. In my world, that is.

Coming back to my calculation model, the unfair bias here is that any delta between an ideal solution and the solution at hand is based on a ideal solution that is solely based on my preferences, experiences and my lack of information on what informed some of the decisions of the real system. So it's never an objective measure for how complicated is, it's only a proxy, and effectively only measures how complicated I think a system is.

The one way that I've been thinking about for a while is a point model, where each element in a system gets a number of points. Each service gets a point (or two or three, depending on how heavy it is), every connection between two services and so on. While this would certainly not be a perfect metric, it would help to gain an understanding, something quantifiable, on how complex a system is, to begin with. And that might help to understand how complicated something is - even though that's always subjective, isn't it.

IT is probably fun to work in because you're working with a bunch of folk that are pretty much successful because they're good at finding patterns in stuff, and then finding solutions that address problems that look just like that.

And that works really well for writing software.

And only works so well for running teams. Truth be told, it works really well for the aspect of actually managing a team. But not for leading one. Let's go.

Managing and Leading are two different things. Two almost entirely different things. This here is my definition, but since it's my blog, I do make the rules.

Management is the easy one. You're given a set of objectives (deliver those two projects), some resources, like a team or money, and you go. To a large extent, managing a software team comes down to understanding what has to be done, finding good ways to doing the work from a technological point of view, and then executing on it. It's here where frameworks like Scrum come in handy, as they formalize a lot of helpful practices to running a healthy team - like plannings, retros and creating transparency about upcoming work. Valuable things.

Leadership is different. Where Management refers to the act of distributing work, managing resources and observing KPIs, Leadership is inventive in its nature.

I was lucky enough to have worked with some really outstanding leaders, and the one dominating trait was that they come up with ideas all the time. And they have the skill required to shape those ideas into things that can be delivered or built by their organization.

And those ideas are the direction. How well the organization executes is the speed, but you can have a thousand horse powers, that's not helping you if you're clueless in regards to the direction of travel.

Those skill sets are complementary. Having great ideas is useless without the right skill to execute, and being great at executing is useless without executing on the right things.

To tie the loop back to the beginning - there's no process, no algorithm and no step-by-step guide that can replace competent leadership at the right position. You can't refine, groom or discuss your way out of a void of ideas, of a surrounding that doesn't dare to think big, and then even bigger.

What stands out to me is that, looking back, the most challenging times were those when we built something that felt almost impossible. Crazy ideas turned to reality. Those were also the most rewarding ones. That required, of course, an engineering team building something, but even more than that: bold leadership.

ideas are momentum

I've been on a vacation for a few days and gave myself a somewhat needed break from writing posts at a high frequency. I realized that even with only 500 words, the posts get a little diluted over time. That's ok, and it's mostly an observation that doesn't surprise me. What makes me happy is that there's some major themes I discovered in my thinking that I wasn't that clear on before I embarked on that little writing journey. Getting myself to sit down on plenty of evenings to write something down on those topics really helped to shape and evolve quite some trains of thought.

Another thing I discovered is that plenty of topics that I spent a lot of time thinking about have not only one, but many interesting facets worth exploring. You'll discover plenty of overlap in the posts itself, and so did I when writing them.

When I'm reflecting on how I write most of my stuff, it's usually a process consisting of three steps. The first one is usually just a "punchline", some tacky one-liner that I use to file the idea initially in my brain - the next post I'm working on has the line "leadership is antimechanical", and it's not more than that, other than some loose ideas and thoughts. I like to think of an idea in simple terms.

The next step is arguably the hardest - it's squeezing that one line and exploring where it goes. I usually do that while writing, and this can be rather exhausting. What helps me is to only look forward. I write a sentence, and then I write the next one. That sometimes leads to sentences that are just not good, but it also leads to a text that is usually above average. You win some, you loose some, but you can't win if you don't do.

The third step happens for some, but by far not all of my posts: A state of flow. Sometimes I just stop thinking and start writing. Interestingly, I feel the same when running, but also only sometimes. A state of flow where the activity itself becomes both very rewarding and feels rather effortless. It's those posts where I'm just crashing through my 500-word barrier as if it doesn't exist. Certainly those moments are the ones that make this whole experiment, this hobby, most fun.

I always enjoyed writing. For myself, in professional contexts, as long as there was a notion of long form and the opportunity to use the medium of written text to explore a topic, I was in. It helps me think. But that state of flow is something that I never felt before when writing plain language - it certainly happened while coding. Every language is a programming language if you try hard enough probably.

And one closing note: WordPress really works very well. Hasn't troubled me one bit since I set it up, and it really lets me focus on writing what I want to write.

So, off to the next posts then.

A key part of my strategy of leading engineering teams is removing complexity and shaping focus, to ensure the important things are clear and can actually be built. I've been doing this a lot, and I'm stuck wondering why we're building so many things that absolutely no one needs.

What do I mean by "so many things"? Everything you're building needs to connect to a problem. Big picture wise, the only reason you're getting paid for writing code is that the code you're writing helps in addressing some form of business problem. Pragmatically, you want to spend as much time as possible to actually solve that valuable problem, and as little time as feasible on anything else. And this is the part where I'm a bit baffled by what we, as an industry, actually do.

Let's just look at AWS. Yes, AWS offers incredible services at a considerable price point, yet it has become commonplace to rely on a single cloud provider for, partially, ridiculous offerings. Take Lambda as an example - yes, there's use cases where invoking a single function that's deployed independently of everything else makes perfect sense, but what are those cases? Like, if you were not already invested in the idea of going cloud-based or serverless, what would the actual, tangible benefit of having an independently deployed and invokable function be?

There's a sweet spot where (almost) seamless scalability from 0 to a really large number of invocations come in handy, but most application I've worked one - and we had some that had to scale really, really substantially - never ran into that problem. What they had in common that a narrative developed around them - making it necessary to go either serverless, rely on a NoSQL Database or give into the latest trend at that time. But that's because some teams wanted to solve a certain problem, not because a problem really existed. I've written about the tempting energy of excitement before, and maybe it's one of the many facets of getting too excited.

The antithesis to this has always been something like Ruby on Rails. Many folks have a huge aha-moment once they start working with Rails for the first time - and kind of a anti-aha-moment once you switch to a lot of other ecosystems. Rails has made an art of not getting in the way of you solving your actual problems - at least to a certain scale. Whatever else you need - Persistence, Background Jobs, Migrations, Bundling, i18n - it's all there and taken care of. You would expect major mainstream adoption of a framework that so universally makes life easier, but for some reason, outside of a rather small group of dedicated fans, Rails is rarely considered as an option, at least in the orgs that I've worked in. (Google Trends tends to agree with me on the overall popularity trends, comparing it with some other Web Technologies).

Why?

I have a theory, and that is that Rails (and other battery-include frameworks) make it just too easy. If it doesn't look hard, if there's nothing tricky to solve every day, then what's the point? A framework that's so good at solving all the things folk usually worry about is its own worst enemy - how can you pretend that you're doing some super complicated work when you're just not? There's a truth in there that might be upsetting - most of software development shouldn't be hard in 2024. Rails makes that obvious. But it's much more fun when it's hard. And so we take our Lambdas, our crazy distributed applications, our scalable somethings that will never scale instead of just writing boring software. Everything but admitting that it was, actually, never that hard.

Today, enjoying a day off and browsing around Twitter to get upset about nothing, I found something to get slightly excited about. I don't have the patience to find the tweet now, but it was very much along the lines that being able to do most things as a software developer without touching your mouse is an important goal. So, the act of navigating your development environment, going about daily tasks, without using your mouse. That's something to get excited about, for sure.

First of all, no, go use your mouse, or trackpad, or whatever pointing device you're computer has attached. Or don't. Or use IntelliJ or VS Code or Emacs or VI or whatever really does it for you. Or use notepad. Or nano. Don't do what slows you down, do what makes you fast. You, not anyone else. Don't listen to random folks on the internet that try to establish a connection between a nuanced mastery of some skill and eventual success in your trade. Correlation does not equal causation.

But to bring the bigger point home - being able to write code fast is useful, it's a good skill to have. Not having to try to find the right keys on the keyboard, not making too many typing mistakes definitely speeds up the act of "code creation", but with all the advances in AI and LLMs, the question is also for how long that's actually going to be relevant. But if you're a software engineer, only a fraction of your time is actually spent coding. Reading code, investigating bugs, documenting things, discussions, understanding problems and defining solutions, all of those are highly relevant activities. And true, some of them can be completed in your development environment, for the most part: Good luck solving them not touching the mouse.

Your job as a software engineer - or software developer, if that's the first line on your CV, is in fact not to write code, it's to make problems go away. Writing code is one of the many ways in which this can be accomplished, but it's really only one. I've seen people do crazy things when developing, from typing speed to mastery of a single environment, and yet they've never really grown beyond being keyboard cowboys - there's no extra points for being incredibly good at one thing and ignorant of all the rest. And a person being average at all the things that define a well-rounded software engineer will blow someone being incredibly good at one thing out of the water any day.

So, before learning to ignore a key input device for internet points, learn something that's actually relevant in your career - do a course on rhetoric, learn how to write well or join a discussion club. All of that will make a much more significant difference in how you'll progress as an engineer than your mastery of an IDE.

I'm a manager, and I'm spending a substantial part of my time writing things down, and hopefully creating clarity in doing so.

The reason why I'm doing this is because I want to be the antithesis to a go to person. While I generally appreciate when people come to me for advise, for chatting or for working on something, there's a specific thing that shouldn't require personal interaction: Understanding stuff. And there's a huge precedent for the point I'm trying to make here.

Take most well-maintained open source projects, something like React. How many calls with the maintainers does it take for you to get familiar with the core concepts? Exactly, none. Not only because you probably don't even have to chance to get one of the core folks on the phone, but also because there's just a huge body of available, free and up to date documentation on the concepts, how to use it and what changed in recent versions. That provides a ton of clarity, without needing to hop on a call with anyone. So far, that's a logistical concern, but there's something deeper.

By not transparently documenting things, sharing knowledge, you're actively preventing a deep and meaningful discourse. In order to be able to disagree with something, folks need to be able to understand what they could possibly agree or disagree with first. Not having that thing, in a fixed, written form in front of you, is just making that discussion entirely impossible. Because there is information asymmetry, and even the economists know that that's a bad thing.

Monopolising knowledge is preventing information symmetry. And that prevents discourse. And that leads - or can lead - to poorer decisions. So it's not about me not wanting to go on a call to explain something complicated, it's about all the folks that didn't want to speak to me, couldn't be bothered or didn't even know that something existed - discoverability is something to consider.

There's so many in ways in which you can be special - very helpful, very smart, very pragmatic, very witty, very fun to work with.

Just don't be special by hoarding information and only sharing it on demand.

I've got a bit of Monday advise. Mostly for me, to prepare myself for an interesting week to come (and a short one at that, with Wednesday being a bank holiday here). Since I've gotten myself a journal - a red one, just in case - I figured that there has to be something meaningful to write in there. Next week, I'll try to write down only when one of the five things below is happening. I'm hoping for plenty of entries.

Things we ship. I'll note every little thing my team is moving to production. Doesn't matter how small, every little thing.

Things we break. Observing every little thing we make worse with our actions. Might be trust, might be an app.

Decisions we make. Again registering when important decisions are being made, and also how they're being made.

Disagreements. There's plenty, but I'm speaking about the ones that require something like active resolution - a discussion, a meeting. Most disagreements dissolve before any active step is required, so I'm trying to focus on the more meaningful ones.

Misunderstandings. Mind you, I thought about calling this one "confusion" first, but for some reason, confusion always sounds like someone hit their head a little and is now wondering in what city they find themselves in. So Misunderstandings: When communication didn't go well and now the discussion is not even about the content of the message, but rather the message and its interpretation.

A lot of focus on the potentials to improve, but also on the things to make us proud.

Gotta sharpen my pencils now, see you in the new week.

In negotiations, there's a term called anchoring. It refers to the concept that whatever first number is dropped in a conversation, is the number that will be the mental anchor for the rest of the negotiation. It's why you should avoid saying the first number in any negotiation - if you're too low, you've anchored yourself needlessly on the wrong thing.

Engineering Teams do the same thing, at least I've seen it happen a few times. But that's not anchoring on price levels (probably also, but that's for another post), I'm talking about performance anchoring. When you're joining a team, and someone tells you that retrieving one entity from the Database takes 5 seconds, what's your reaction? It's a bit excessive for 2024, isn't it? Of course, you're diving a bit deeper into the context and the explanations and the history on how it came to be and, well, it's still excessive. But after a while, that edge wears off and it becomes a little more normal, day by day, hour by hour. And instead of driving a realization that some things might not be great, it's rather a journey to normalizing what should not be normalized.

And that's performance anchoring. Normalizing things that behave wildly different from reasonable assumptions. And it's something that drives me mad, because bad performance compounds. It's like a lasagne made up of only bad decisions and spineless management.

It's fair to characterize performance anchoring as just another face of the broken window theory, and it's really not far from the original statement, but there's value in being specific on the topic of performance. Performance is an interesting challenge in most teams - whether it's backend, frontend, mobile or even foundational teams that are not running their own stuff. And performance is highly subjective - is 2 seconds a good load time for a page? It depends. But it's also a truism that Google got to where they are today not only by having great results, but also by delivering a ridiculously fast site, back in the days. Performance in itself is a feature, and if it's overlooked, you won't be able to fix that with some half-assed measures. That's why you have to avoid any form of poor performance creeping into your system - and if it's already creeped in, you need to aggressively remove that.

Happy to write something that I can use to quote myself - and here's that: It's 2024. Most things we're doing can, and should be, fast. If your website is slow, it's not because you're doing some computations that no one has ever done before, no. It's because you dropped the ball on modelling the system correctly, figuring out the right architecture or deploying it to where it should be deployed. All of those things can be fixed, poor performance isn't just a fact of life - it's a result of specific actions. So yeah, go care about performance and make things fast.