Potemkin Life

Zen and Development

Good Software Engineers Are Rare

Good software engineers are rare. There are many reasons as to why. The most obvious reason is that to become a good software developer one has to master many skills (language proficiency, documentation literacy, debugging, text editing, architecture, and design, etc, etc). Because these are the obvious barriers to becoming good, there is an abundance of advice out there for young engineers on how to hone their skills. Most of this advice is either uninteresting, inefficient, or both. Why? Because the process of mastering a skill is typically independent of the skill being mastered. The process is almost always as simple as: learn (read technical material), practice (daily), have patience.

Unfortunately, mastering the skills is not enough to become a good engineer. If it were, the good software engineer would be no rarer than the hardworking, motivated engineer. Having the essential hard/technical skills mastered is only the start. To become great, the software engineer must adopt a mindset that often operates in opposition to the philosophical structure of the western world.

Software Products Are Unique

My development as an engineer began outside of the software world. I studied electrical engineering where I spent countless hours of my life in the lab: designing circuits, prototyping on breadboards, acid bathing PCBs. All of these efforts led to a finished, permanent product. Whether it be a power supply I could plug my computer into, a multimeter, an antenna, or some mostly useless gadget, I always had something I could hold in my hands that gave me a sense of pride and satisfaction. Productive societal activity tends to operate similarly. There is the expectation of tangible end results, and the success of an activity or task is contingent on the end result. For example, there are few companies or teams that do not set KRs (key results) each quarter, half, year. Developing software products, peskily enough, is decidedly unlike this. A software product is ephemeral. It can't be held or touched. There is no definitive "end result" or "finished product". What's more, the work done is impermanent. These traits aren't just unique, they clash with many fundamental expectations about the way we believe work/productive activity should behave. Whether consciously or subconsciously all good software engineers have had to internalize some sort of framework to allow themselves to reckon with this cognitive dissonance.

Look To The East

So what must a software engineer do if they can't rely on societal norms of the west to help them build the skills, temperance, and mindset to become competent, productive, and successful? They must look to the east. The western concepts that the software engineer must cast aside are:

 attachment
 permanence
 duality
 reason

The philosophy of Zen Buddhism rejects these cornerstones of western life. So it is in the delightful riddles/stories that the Zen Buddhists call Koans that we might look for answers. The following are three of my favourite Koans that have been helpful on my path to becoming that rare and golden, madly productive, madly exact engineer.

Non-Attachment

"Kitano Gempo, Abbot of Eihei temple, was ninety-two years old when he passed away in the year 1933. He endeavored his whole life not to be attached to anything. As a wandering mendicant when he was twenty he happened to meet a traveler who smoked tobacco. As they walked together down a mountain road, they stopped under a tree to rest. The traveler offered Kitano a smoke, which he accepted, as he was very hungry at the time.

"How pleasant this smoking is," he commented. The other gave him an extra pipe and tobacco and they parted.

Kitano felt: "Such pleasant things may disturb meditation. Before this goes too far, I will stop now." So he threw the smoking outfit away.

When he was twenty-three years old he studied I-King, the most profound doctrine of the universe. It was winter at the time and he needed some heavy clothes. He wrote his teacher, who lived a hundred miles away, telling him of his need, and gave the letter to a traveler to deliver. Almost the whole winter passed and neither answer nor clothes arrived. So Kitano resorted to the prescience of I-King, which also teaches the art of divination, to determine whether or not his letter had miscarried. He found that this had been the case. A letter afterward from his teacher made no mention of clothes.

"If I perform such accurate determinative work with I-King, I may neglect my meditation," felt Kitano. So he gave up this marvelous teaching and never resorted to its powers again.

When he was twenty-eight he studied Chinese calligraphy and poetry. He grew so skillful in these arts that his teacher praised him. Kitano mused: "If I don't stop now, I'll be a poet, not a Zen teacher." So he never wrote another poem. "

In this koan, Kitano displays an ability that every software engineer should find enviable: a relentless non-attachment to all things outside of the pursuit of Zen. A software project is in constant flux. So is the state of tools, technology, and frameworks in the field. There is a serious lack of permanence in software development, and the best way to cope with that impermanence is to become a solid practitioner of non-attachment.

The inability for ICs to remain unattached to particular pieces of work completed, or particular technologies is probably one of the most common and detrimental problems software development teams encounter. Features that have had blood, sweat, and tears poured into them but do not serve a purpose fail to get removed, technologies invested in that end up being duds do not get replaced. All of this results in projects and companies suffering major setbacks. Ben Horowitz frames the problem well in his book The Hard Things about Hard Things: "Early in my career as an engineer, I'd learned that all decisions were objective until the first line of code is written. After that, all decisions were emotional." The less attached a team of engineers is to their own work, the less emotional the decisions about a project will be. Good engineers are happy to delete their own code. A leaner code base is an easier codebase to work in. Good engineers are excited when a teammate finds a more robust, easier-to-work-with solution to the problem they have been working on. They don't create inertia and waste time by resisting switching to that solution, no matter how much work they have sunk into their own implementation. Good engineers are like Kitano.

It Will Pass

A student went to his meditation teacher and said, "My meditation is horrible! I feel so distracted, or my legs ache, or I'm constantly falling asleep. It's just horrible!"

"It will pass," the teacher said matter-of-factly.

A week later, the student came back to his teacher. "My meditation is wonderful! I feel so aware, so peaceful, so alive! It's just wonderful!'

"It will pass," the teacher replied matter-of-factly.

Mmm, such a beautiful dialogue. Reading it quells the destructive flames of pride and ego that flare-up in my stomach whenever I am on a productive kick. Many engineers thrive off of the energy of this fire, but it can be dangerous if untempered. Engineers, it has been said, are natural optimists. Engineers are also, I will posit, naturally prideful. When they are assigned a project or ticket and are asked to estimate how long it will take, they almost certainly will imagine themselves at their most productive when making the estimate. This results in the unenlightened engineer wreaking no small havoc on the logistics of managing a project. No engineer is at their most productive at all times; the productive fervor of a great session is fickle. The engineers who assess their own skills and compare themselves to their colleagues as if they are, hurt themselves and their team. They stunt their growth (by believing they are already outperforming all their colleagues), are hard to manage, and are hard to work with.

Recently, while working on extending our application's backend to be accessible via a GraphQL API, I came across a few blogposts by an engineer named Marc-Andre Giroux. Clearly, a quite talented engineer, I noticed a neat little feature on his site's homepage. As a play on the 10x engineer motif, he describes himself as an "Nx platform interface engineer", the N flitting randomly between decimal numbers from 0 to 10. Marc-Andre is clearly a talented engineer, yet is zen enough to appreciate that he is not always in the state of ultra-productive. Admittedly, sometimes he is even less than 1x engineer. He understands "It will pass" and is most likely a factor in his success.

Avoiding overestimation of yourself is not the only lesson to be learned. This koan helps me to avoid taking productive times for granted. Understanding that quieter seas and lulls in productivity and focus may be just around the corner is good motivation to keep pushing for more while you are in the zone. Zen has taught me to never let a good session go to waste.

Finally, this koan helps me during the unproductive and unfocused times as well. I often experience a quite intense physical aching and pain during those hours where I can't seem to get any code written, or worse when I can only seem to produce poor, convoluted, messy solutions. Knowing that such a state is somewhat inevitable and that It Will Pass, is a medicative and constructive mindset to have.

First There is a Mountain

At the first level on the path, he saw mountains as mountains and rivers as rivers.

On the second level of the path, he saw that mountains are not mountains and rivers are not rivers.

And at a third level, he saw once again mountains were mountains and rivers were rivers.

This koan is a description of the Zen student overcoming dualism. In Zen, the idea is called "Not Two". The student first sees the river as discrete from the mountains. The student then sees that they are not discrete, that their separation was really an illusion created by our ego/consciousness. In reality, neither can be called a river or a mountain, there is only the whole. Then finally, after the student has contemplated and understood reality as a whole, can she now see the different parts of that whole for what they are. Rivers become rivers and mountains become mountains.

For the software developer, this koan translates as something like the following: At the first level, the engineer saw velocity as getting through tickets quickly. At the second level, she saw that velocity has very little to do with getting tickets through tickets quickly. And at a third level, the engineer once more saw velocity as getting through tickets quickly."

The First Level

At the first level, the engineer focuses on tickets left to be done and the rate of completion of those tickets; velocity. They want to be fast and they want their team to be fast. They see their job as a race to finish a project board of separate and discrete tickets. If the engineer is attentive and introspective enough she may start to notice something odd: the harder she pushes, the slower the project progresses. Deadlines start slipping, so she pushes even harder. Shipping a final polished product begins to feel unattainable because new work seems to keep cropping up out of nowhere. What's going on?

Stories from The First Level

Let's try to answer that question with a couple of stories. Ten thousand miles from home I was drinking a warm Club on a rooftop in the Kisimenti neighborhood of Kampala. I was there to do product research and get valuable in-person time with the rest of the product team. It was the day after I had landed, having flown in with my brother and CTO of the company. We were having a drink at sundown with the CEO. This was quite early on in the life of the company and near the beginning of my career as a software engineer. We were struggling. Our product wasn't good enough and it didn't seem to be getting better fast enough with our current runway.

Our CEO brought this stark reality to my attention quite frankly. "We aren't iterating fast enough. Our product might be good, but if it is not great soon we are going to die. You are supposedly talented, smart engineers: why aren't we shipping features faster?"

That question hung in the air between us. It was essentially left unanswered. The excuses I was trying to muster all got caught in my throat, which I washed back down into me with gulps of the now warm Club. It felt like quite a blow for me. At that point in my short career, I had mostly experienced success. Furthermore, my previous managers had been kind enough to shield me from my mistakes or failures. Understanding that my contribution to the company my brother was trying to build from nothing as inadequate was devastating.

To right this wrong and do good by the rest of the team, I started to focus on velocity. A grave mistake. My work became sloppy. I attacked each ticket with feverish desperation. I grasped at straws to hurl myself over the finish line of each ticket. It seemed to be working, I was undoubtedly finishing tickets at a faster clip. But unbeknownst to me, I was also creating technical debt, shipping bugs, delivering work that demanded focused, and discerning code reviews from my coworkers. As a result of my efforts, if our team's velocity didn't end up slumping it was only due to the heroic efforts of the two talented engineers I was lucky to be working with.

Just the other day, a software developer friend of mine was telling me about the predicament his company was in. The CEO was putting a ton of pressure on the dev team to ship more. The pandemic being in full swing, and them being a struggling health tech company, he saw their salvation in a million different products and features that they could ship that would catapult the company to wild success. But a funny thing was happening. As the dev team all started working overtime, they began to ship less. This angered the CEO, thinking there was some sort of civil disobedience happening. It also angered my friend. His hard work and stress not bringing anything to fruition.

These stories illustrate why software development is hard. Depending on the level of maturity of the engineers working on a project, pushing them to work harder will not necessarily get results. It likely will slow things down. When the engineer sees tickets on the board as individual pieces of work, (i.e. sees mountains as mountains and rivers as rivers) velocity fails to materialize as velocity. Upper management (outside of the product org) is rarely able to appreciate this. Leveling up or limiting exposure to well-meaning but misplaced external pressures is the only way forward for the engineer on this level.

So, what does level two look like and how can the engineer get there?

The Second Level

The second level begins with understanding Not Two and overcoming duality. Separate and discrete tickets do not exist. The completion rate of such tickets is not velocity. The only reality that is of concern is the whole, i.e. the state of the entire project and codebase. Is it in a healthy state? Is it scalable? Is it riddled with bugs? Are solutions exact and well-tested? Is it hard to work in? Is it's health declining? These are the front of mind questions for the Zen engineer who has overcome duality. She does not prioritize velocity over the state of the whole. She understands that the only way to keep the project in a healthy state is through correct, thorough, clean, methodical solutions. Working harder does not mean working faster. It means working with more precision and more exactness. It means spending an entire day refactoring an old and messy piece of code, even if it currently works. It means standing up a brand new testing framework to increase coverage across the project. It means spending an entire sprint on improving tooling. It means finding tickets to add to the project board. In a state of zen meditation, this engineer has decoupled themselves from the motivations and objectives of the company; getting wrapped up in the frivolity of OKRs and deadlines is only a distraction from the state of enlightenment they seek: an orderly, clean, scalable, beautiful codebase that is a delight to work in. If she maintains her codebase in such a state, velocity will no doubt come.

The Unintuitive Nature of the Second Level

The behavior and decisions of the non-dualistic engineer are baffling to the engineer who is still on level one. There are many paradoxes in the way that she works. When I worked as a hardware engineer at a software startup, I was completely confused at the decision the dev team made to learn a new framework and re-write an entire working client codebase. While we were in the middle of fierce competition with another well-funded startup, the entire product team responsible for our core customer product didn't ship a new feature for at least a month because of this. Such a decision seems incomprehensible when rivers are rivers and mountains are mountains.

Engineers rely on logic and reason to succeed in their jobs, so how does one embrace such an unintuitive mindset? I would suggest measuring the ratio of new work to re-work completed. Focusing on increasing that ratio can adjust the priorities of an engineer trying to get through a difficult ticket as fast as possible.

The engineer who never leaves level two though risks becoming Fred.

The Third Level

Only once the engineer has completely internalized the idea that the state of the codebase is the number one priority, can they then start to focus on velocity again. Pushing themselves means working faster, and working faster no longer means sacrificing the health of the codebase. Obviously this is the ideal state. Rivers are rivers again, but not without understanding that they can't be separated from the whole. This engineer is powerful. She is more effective than her smarter and more technically proficient colleagues. She is able to expose herself to the needs of the organization without being detrimental to her team and product. The number one complaint I have heard from engineering managers about their smartest engineers is that they are incapable of identifying the correct problems to work on. They got to level two and stayed there.

Students of Zen

Unwittingly or not, good engineers are students of Zen. It is necessary to deal with all of the unconventional aspects of the discipline. Understanding the profession from this perspective can allow a young engineer to become an effective engineer quicker.

- 3 likes