Explaining through Skillful Means

In Buddhist philosophy, there is a concept whose Sanskrit name is upaya. This is often translated as "skillful means", though, as with many other ancient philosophical terms, it is hard to translate exactly and has many interpretations.

To illustrate one useful interpretation, I'll start by explaining a little bit about Buddhism. If you boil Buddhism down to a one-sentence summary, it is about reducing your suffering (and that of the world) by living a moderate life and practicing meditation.

As simple as this sounds, there is a lot of theory behind it, covering — for example — what it means to live a moderate life, why that helps reduce suffering, what good meditation is and how it works, and a host of other questions.

The theory and practice of Buddhism is in fact so complex that it is usually taken for granted that it takes years of study and practice to really understand it. To make matters worse, there are many different opinions about what kind of study and practice, and how much.

But most of the concepts can be understood at first in a simplified way, and since all students of any complex skill have to start somewhere, people often start with these simplified forms that aren’t quite right to a more advanced student.

For example, there is an uncomplicated form of Buddhism called Pure Land, that more or less believes that if you faithfully chant a particular mantra, you will be reborn in a special paradise. You can still see in this belief some of the elements in my one-sentence summary above: chanting a mantra is a form of meditation, being reborn in a paradise is a reduction of suffering, and faithfully practicing a good habit is a rudimentary form of living a disciplined life.

So even though a “more advanced” student of Buddhism might roll their eyes at the simplicity of this approach, they would have to admit that practicing it was still an advancement in the direction of the aims of Buddhism, and therefore better than no understanding of Buddhist principles.

This, in a nutshell, is what upaya means: it is OK to let someone have a “wrong” idea about something you are explaining to them, so long as it takes them one step closer to understanding.

What does this have to do with software development?

Software development, boiled down, is about solving practical problems by producing reliable applications with maintainable code bases. (All of these things, incidentally, also reduce suffering. ;-) )

There is a complex skill set and complex methodologies that must be learned to accomplish this task. It is taken for granted (at least by some people) that mastering them takes years of study and practice, but there are many different opinions about what kind of study and practice, and how much.

But many concepts have a simplified version, and since all learners have to start somewhere, you often see versions of good practices that a more advanced practitioner would see as not quite right.

For example, a developer with less experience — or perhaps just different experience — might chant a mantra such as “always document”, “never document”, “always design before you program”, “never design before your program”, or “paradigm/language/library/framework X is always better than Y”. These mantras might strike you as over-simplified or incorrect based on your experience (especially when you favour an opposite version of the chant. ;-) ).

You can still see that they share the core concern of solving the practical problem reliably and maintainably. Maybe it is OK to let them be “wrong” for now, so long as they are engaged with the ultimate concerns of software development and are using the mantra to get one step closer to understanding.

When mentoring developers as a team lead or when introducing new ideas, techniques and methodologies to your peers or to an on-line community, it can be hard to determine when to hold a hard line, to nitpick or to qualify and when not to. Often it is best to introduce a simplified form of the idea and let them run with it for a while, trusting that if they share the same goals they will come to a deeper understanding on their own and that they will ask for more input when they need it.

Sometimes the best policy is to let others be "wrong" for the time being.

Be Willing to See Failure

There is an old saying that I quite like: "If you’re never failing, you aren't trying."

The simplest way to read this adage is as an exhortation to take more risks, to stop playing it safe and undertake bigger and scarier projects.

An equally valuable second reading is to take this as a salve to failure -- it is OK to have failed or had a set back because it means you were trying something challenging.

But there is a subtler reading that I want to explore that comes into play when you are trying, and you doseem to be succeeding, that addresses a more dangerous form of failure: the failure to perceive failure.

Anyone who has spent any time doing development projects (for software or otherwise), has probably heard of, if not actively participated in, a project that went along just fine for months, only to burst into flame as some final deadline approached. And by going along just fine, I mean that people were trying: meetings were held, decisions were made, mountains of code were written, documents filled the shelves, metrics were monitored.

But mysteriously, as the looming deadline approached, all kinds of critical problems seemingly popped out of nowhere, delaying delivery, running up costs, creating a sudden, unforeseen crisis.

(Sounds a lot like our current world-wide financial crisis, doesn't it?)

So what went wrong?

Rarely do problems just pop out of nowhere. More often, we simply become aware of them suddenly. And sometimes that’s because we didn’t want to be aware of them, because we wanted to focus on the success instead of the failure.

Unfortunately, I think this is a major challenge in our North American cultural temperament. (People from other locales can assess this for their own situation.)

We tend to value optimism, the power of positive thinking, the positive spin to the point where any talk of real risks or real failings tends to sound like pessimism or cattiness. We tend to assume that those who question a person, project or process that seems to be succeeding are guilty of sour grapes or defeatism.

So a sub-prime mortgage crisis can "sneak up on us" and bring down an Alice-in-Wonderland financial system, and a project that was "going along fine" can suddenly go off the rails.

The solution to this problem is to accept that failures and problems are, as the adage suggests, a necessary side effect of trying to accomplish something, and that success doesn’t come from ignoring problems but from finding creative and effective solutions to solve or mitigate them. And once we’ve accepted this, we can change our team culture and practices to help flush problems and failures out of the bushes sooner.

In software development, there are a number of techniques that are gaining wide acceptance, such as iterative delivery, open communication between clients and teams, continuous integration, test-driven development, and others, that solve this very problem.

These techniques all work by creating more frequent opportunities to put a project through its paces, so that weaknesses, misunderstandings and oversights are spotted sooner, and so that successes have really proven themselves, and can be safely built upon.

A good rule of thumb is that if you are aren’t routinely finding unexpected problems or failures when using such techniques, you probably aren’t using them rigorously enough.

Which leads us back to the opening adage, but maybe in this context it should be changed to: "If you’re never failing, you aren't looking."

Minimize Locally, Succeed Globally

My first full-time programmer position was in the research institute of a nationally-known hospital, building database applications for research data and administration.

I have often joked about that time that I was a one-man development cycle, since I had was responsible for the process from beginning to end.

I was the "sales guy", drumming up business with internal clients. I was the business analyst, modeling the often complicated problem domain. I was also the programmer, the system administrator, and the tech support guy.

To complicate things further, there were a fair number of projects, some small, some dormant, but also some that were quite large, complex and active, and which grew more so over time. Eventually, I also added the roles of project manager and dev team lead when a family of the projects coalesced into a behemoth.

As you can imagine, having so many roles on so many projects was a bit of a tightrope walk. To survive, I had to constantly ask myself: "What is the simplest and most reliable way I can do this?"

As the "sales guy" (we did internal billing), there was no benefit to making grandiose promises to drum up more sales than I could safely deliver. I could only handle so many projects anyway. And failing to deliver on a couple projects would quickly have poisoned the well.

As the business analyst, I had no reason to nurture technicolor dreams for a particular project. I was going to have to implement and support it as well, so I had to just solve the priority problems elegantly and efficiently, with no frills.

As the programmer, I had no motivation to demonstrate my genius with complex algorithms and impenetrable code, or by layering on the latest tech toys. I needed my code to be easy to understand and fix, because I wasn't going to have a lot of time to build it or maintain it. Simplicity and clarity also helped with stability and reliability, which was good, since if the application wasn't stable and reliable, I was going to generate more tech support pain than I could deal with.

And since I didn't have time to be a tech writer and software trainer as well, I had to sacrifice a slick look or gee-whiz features to make simple, clearly-labeled interfaces that the most technophobic administrative assistant would be able to figure out reasonably well in 10 minutes or less. Otherwise, I was going to add the role of professional hand-holder to my roster.

Of course, in spite of my best efforts, none of this worked out 100% of the time, and I dreamed of the day when I would be using the heavy-duty tech on the big projects with the big boys, where most of my roles would be fulfilled by other people. And I got my wish soon enough.

But I discovered a hidden danger with larger projects where most of the participants don't have sight of the whole process: that each role maximizes itself to the detriment of the outcome.

The sales rep makes unfulfillable promises to get his sales. The business analyst gold-plates and over-specifies the requirements. The programmers over-engineer with too many libraries, frameworks, cool features and clever code. The tech support guys, who, to be fair, do the most suffering on the front lines when things go awry, try to toss some of their woes back over the wall to the developers.

And if this tendency isn't actively countered, instead of building bigger, better software, they start to build software that costs more, is less reliable, and less usable.

So now that I've seen both sides, I think maybe necessity had put me on to something way back then: being minimalist with the requirements, efforts and costs at each step in the process in order to optimize the workings of the whole.

And I think an organization that manages to foster that ethic through good communication and good leadership will produce the best software for the lowest cost in time and money.

Software is "Philosophy Made Manifest"

What does the name "Philosophy Made Manifest" have to do with software development? Here is the answer:

Though I have worked most of my adult life in software development, my original academic background is in Linguistics and East Asian Studies. Since this seems like a bit of a leap, I have often been asked what, if anything, I gleaned from those studies that helped me in my career.

My answer is: a surprisingly large amount. In fact, sometimes I think these things give me a big advantage over computer science or software engineering graduates.

The influence of Linguistics on my thinking about software development probably deserves its own post, but to explain "Philosophy Made Manifest", the most relevant area of my past studies was the philosophy portion of East Asian Studies: Buddhism, Taoism, Confucianism and others.

When you first dip into the surface of these systems of thought, some of the stuff sounds pretty far out there, especially from a modern western point of view, but once you get past the surface, it turns out that these philosophies are the end result of a long line of astute observers and thinkers who were concerned with very real and practical problems: how to live harmoniously with other people, how to set up effective government, how to live a happy and productive life, and many others.

In order to succeed as a student of these philosophies, you have to get good at figuring out the how these people in far away times and places understand the world. And to make their ideas relevant, you have to get good at seeing beyond the foreign trappings of the ancient vocabulary and forgotten social customs, to get at the heart of what they have to say about the unchanging realities of human life.

Once you've mastered this, you can then translate these ideas into more immediately applicable forms, ones that speak to the vocabulary and social customs of the here and now.

Which, perhaps surprisingly, leads us to software development.

Effective software development is not only about the clever use of technology, it is a profoundly human activity that requires an understanding of how people work together and how users, business leaders, developers and others think about the world.

At its core, software development is synthesizing a philosophy of some problem domain, based on these different world-views, and implementing it as a concrete, practical tool that supports and enables the activities of that problem domain.

This is how software development is "philosophy made manifest".