About a year ago, I took up playing a middle-eastern instrument called the oud.
One of the characteristic forms of music for the oud is called a taqsim , which is a short improvisation. I’ve always loved improvisational musical styles, and this was part of the draw of the oud for me.
Improvisational music can be a very pure form of spontaneous creativity, and when it is going well, it gives a powerful feeling of freedom and self-expression.
Many people who don’t play an instrument or who play an instrument only by reading sheet music often ask me how one can possibly make up music on the fly, and I always tell them it is easier than it sounds.
A common misconception is that to be creative means that you throw all rules and restraints out the window, pulling great stuff out of thin air, as if by magic. Many people think that creativity is somehow inherently chaotic and random. But a simple example can show that this is false.
Have you ever listened to a small child with no musical training hammer away at a piano? Based on the “creativity is chaos” misconception, this should be the height of creative expression, since the child is following no structures or rules aside from the physical limits of the piano. But most people would agree that the result is not music, but rather irritating noise.
The first thing that many piano methods do with our budding creative genius is to teach the scales. This is because these are the basic structures that underpin western music. To be either a competent listener or a competent player of western-style music begins by internalizing the sound landscape that is defined by the diatonic (or pentatonic) scales and the melodic and harmonic palette they imply.
After learning even one of the scales, our enriched child could hammer away the same as before, but restrict the hammering to the notes of that scale. Because there is some musical structure, most people would judge the result as more musical, and in that sense morecreative.
This is the trick with improvisation: you aren’t really making up something from scratch. All common forms of musical improvisation I’ve ever studied (Indian classical raga , the blues , Middle Eastern maqamat and Jazz , for example) actually have a lot of structure to begin with: there are specific scales, rhythms, sequences and patterns that have to be learned and internalized or else the result won’t sound like music, or at least not the kind of music you are trying to play.
Once the player (or the listener) has internalized all the rules and structure, then freedom from rules comes back into play. If you have too many rules followed too rigidly, the results become boring and predictable and the music loses its vitality. Great improvisers know where their freedom lies and know where and how to break the rules to produce something fresh and unexpected, and it is this quality that can make improvised music so vital and exciting for players and listeners. So-so improvisers sometimes fall flat on their face by going too far away from the basic structure and have to scurry back to it. But even this freedom to fail is part of the process.
What is true for music seems to be true for other domains with complex structure that require creative solutions: software development, management, politics and economics, for example.
In any of these domains, too few rules results in inharmonious chaos, and too many stifles inventive and effective approaches.
Any approach to these problem areas that looks only at adding regulation or only at removing regulation is probably missing the boat.
The real question always is: “How do you plan to balance freedom and control to maximize harmonious creativity?” Any philosophy of these domains that can’t answer this question is probably not worth considering.
September 30, 2009
July 7, 2009
Why proving that P ≠ NP is hard
A while ago I spent the time to “audit” online a Cambridge University course given by Tim Gowers on computational complexity.
In this course, Gowers takes the students through a proof by Razborov that there is no “natural proof”, for a particular form of the problem, that P ≠ NP, which I found very interesting.
This got me thinking about why this is a hard problem to solve, and I thought it might be useful to explain the intuitive reasons behind it.
So let’s think about this by looking at the Traveling Salesman Problem, since it is a fairly intuitive example of one of the hardest problems in the NP complexity class, known as NP-complete.
In this problem, you have to find the shortest path through a number of cities, visiting each city once and only once. This is a typical NP problem, where the naïve approach to solving the problem would be to just work through all the combinations possible, check the value of that combination (in this case the length), and pick the shortest one.
The problem with that is that the number of combinations gets very large very fast, so it really isn’t practical to do this.
Let me show you what I mean. Let’s take the case where there are 5 cities: A, B, C, D, E. Any path through these cities can be described by a sequence of these letters. For example, CDEAB would be the sequence starting at city C, then going on to city D, etc. To figure out how many combinations there are, I have to use each of the letters once and only once, so there are 5 possible choices for the first slot, four for the second, and so on. This gives me 5x4x3x2x1=5!=120.
In general, this brute force approach to solving the problem, which represents the worst case scenario among solutions, will take n! steps, where n is the number of cities we are considering. This gets very big, very fast. For example 10! is equal to 3628800, so you can see that as you add cities it becomes vastly harder to solve the problem this way.
Now it is very important to realize that this is the worst-case scenario only, or as it is described in mathematics, an “upper bound” on how hard the problem is. For example, if all the cities are in a straight line, you can solve the problem in many fewer steps: just start on one end and move to the next in line and so on
This works because we know that on a straight line, the shortest path will be on that line, so that if the cities are aligned ABCDE, we know right away that we aren’t going to have to consider any of the combinations that don’t follow the sequence on the line. In other words, there is a pattern to the data that allows us to ignore a number of combinations (in this case, all but the two, which are of equal length: ABCDE, and EDCBA).
But to say that some problem is in the class NP (and not in P) is essentially to say that there will always be some collection of cities you could choose that would require you to go through the worst case scenario to find the shortest one. (Technically, the actual worst case scenario for this problem is only exponential in complexity, but that is still pretty bad and doesn’t affect my argument here.)
That is, unless someone can prove that P=NP. This would mean that there was always a shortcut that allowed you to find the solution without having to work through the majority of the possible combinations. And this shortcut would almost certainly involve being able to consistently find some pattern in the data, just as we did with the straight line example, that simplifies the data down to a smaller number of cases.
So in a sense, it should be much easier to prove P=NP than that P ≠ NP. (That no one has proven the former yet is a pretty good reason to suspect that the latter is true, even though no one has proved it either.)
The reason is that to prove P=NP, you just need to show that there is always a pattern in the data that will simplify it, whereas to prove P ≠ NP, you would have to show that there can’t be such a pattern for some scenarios in the problem set.
Why is it easier to show that there is a pattern than that there can’t be a pattern? Well, if you have a pattern, you have a relatively small number of steps needed to describe that pattern, and it is much easer to work through those steps and convince yourself that you have covered all the bases. You can also try your pattern on all the hairiest known instances of the problem and show that they work.
In a sense, having a pattern simplifies the search space required for the proof in exactly the same way that having the pattern simplifies the search space to find the answer to the problem.
But how do you show that there is no possible pattern for some set of data?
Let’s say that I give you a list of digits and tell you that one of two scenarios generated these digits: either I am generating them randomly for each digit as you ask for them, or I am using some relatively simple-to-genarate pattern to generate them. Here is a sample list:
15926535897932384626433832795…
Can you see a pattern? You could do a statistical test on the digits to see if they show the properties of randomness, and in this case, I think you would find that they do show those properties.
So, can you then conclude that they are random?
No, unfortunately not, since there is a pattern here: these are the digits of pi after the well-known first three 3.14…
So the challenge is that you can never really be sure that there is no pattern to some data set; it could just be a really, really unobvious pattern that you haven’t thought of yet.
So you can see how very hard it would be to prove that there is always some configuration of cities that can’t be simplified by some pattern, which is only part of what you would have to do to prove that P ≠ NP.
This is why it is unlikely that anyone will find such a proof unless they can develop some hard-to-imagine technique that allows us to detect when there aren’t any patterns in a set of data.
In this course, Gowers takes the students through a proof by Razborov that there is no “natural proof”, for a particular form of the problem, that P ≠ NP, which I found very interesting.
This got me thinking about why this is a hard problem to solve, and I thought it might be useful to explain the intuitive reasons behind it.
So let’s think about this by looking at the Traveling Salesman Problem, since it is a fairly intuitive example of one of the hardest problems in the NP complexity class, known as NP-complete.
In this problem, you have to find the shortest path through a number of cities, visiting each city once and only once. This is a typical NP problem, where the naïve approach to solving the problem would be to just work through all the combinations possible, check the value of that combination (in this case the length), and pick the shortest one.
The problem with that is that the number of combinations gets very large very fast, so it really isn’t practical to do this.
Let me show you what I mean. Let’s take the case where there are 5 cities: A, B, C, D, E. Any path through these cities can be described by a sequence of these letters. For example, CDEAB would be the sequence starting at city C, then going on to city D, etc. To figure out how many combinations there are, I have to use each of the letters once and only once, so there are 5 possible choices for the first slot, four for the second, and so on. This gives me 5x4x3x2x1=5!=120.
In general, this brute force approach to solving the problem, which represents the worst case scenario among solutions, will take n! steps, where n is the number of cities we are considering. This gets very big, very fast. For example 10! is equal to 3628800, so you can see that as you add cities it becomes vastly harder to solve the problem this way.
Now it is very important to realize that this is the worst-case scenario only, or as it is described in mathematics, an “upper bound” on how hard the problem is. For example, if all the cities are in a straight line, you can solve the problem in many fewer steps: just start on one end and move to the next in line and so on
This works because we know that on a straight line, the shortest path will be on that line, so that if the cities are aligned ABCDE, we know right away that we aren’t going to have to consider any of the combinations that don’t follow the sequence on the line. In other words, there is a pattern to the data that allows us to ignore a number of combinations (in this case, all but the two, which are of equal length: ABCDE, and EDCBA).
But to say that some problem is in the class NP (and not in P) is essentially to say that there will always be some collection of cities you could choose that would require you to go through the worst case scenario to find the shortest one. (Technically, the actual worst case scenario for this problem is only exponential in complexity, but that is still pretty bad and doesn’t affect my argument here.)
That is, unless someone can prove that P=NP. This would mean that there was always a shortcut that allowed you to find the solution without having to work through the majority of the possible combinations. And this shortcut would almost certainly involve being able to consistently find some pattern in the data, just as we did with the straight line example, that simplifies the data down to a smaller number of cases.
So in a sense, it should be much easier to prove P=NP than that P ≠ NP. (That no one has proven the former yet is a pretty good reason to suspect that the latter is true, even though no one has proved it either.)
The reason is that to prove P=NP, you just need to show that there is always a pattern in the data that will simplify it, whereas to prove P ≠ NP, you would have to show that there can’t be such a pattern for some scenarios in the problem set.
Why is it easier to show that there is a pattern than that there can’t be a pattern? Well, if you have a pattern, you have a relatively small number of steps needed to describe that pattern, and it is much easer to work through those steps and convince yourself that you have covered all the bases. You can also try your pattern on all the hairiest known instances of the problem and show that they work.
In a sense, having a pattern simplifies the search space required for the proof in exactly the same way that having the pattern simplifies the search space to find the answer to the problem.
But how do you show that there is no possible pattern for some set of data?
Let’s say that I give you a list of digits and tell you that one of two scenarios generated these digits: either I am generating them randomly for each digit as you ask for them, or I am using some relatively simple-to-genarate pattern to generate them. Here is a sample list:
15926535897932384626433832795…
Can you see a pattern? You could do a statistical test on the digits to see if they show the properties of randomness, and in this case, I think you would find that they do show those properties.
So, can you then conclude that they are random?
No, unfortunately not, since there is a pattern here: these are the digits of pi after the well-known first three 3.14…
So the challenge is that you can never really be sure that there is no pattern to some data set; it could just be a really, really unobvious pattern that you haven’t thought of yet.
So you can see how very hard it would be to prove that there is always some configuration of cities that can’t be simplified by some pattern, which is only part of what you would have to do to prove that P ≠ NP.
This is why it is unlikely that anyone will find such a proof unless they can develop some hard-to-imagine technique that allows us to detect when there aren’t any patterns in a set of data.
June 14, 2009
What problem are you trying to solve?
Anyone who has worked in IT for any period of time is going to be familiar with the dilemma of choosing one particular solution over another. Should I learn language X or language Y? Should I use framework A or framework B for my next project? Should I use the Foo Architecture or the Bar Architecture? Is the Widget application better than the Grommet application?
Since sometimes it seems that the choice you make can have big costs in terms of your project’s success (and your future employability), these dilemmas can result in a lot of stress. It doesn’t help that for each solution you have some proponent telling you that if you don’t use their approach you are stupid and ugly (see Linus Torvalds talking about his source control tool ).
So what’s the smart way to decide?
I think the first step is to realize that you have the wrong question. Technology people by definition tend to be interested in technology (tools and solutions) and we tend, as a result, to give short shrift to the problems themselves, especially when they fall outside our domain. As a result, solutions pile up faster than you can evaluate them, and problems often get shoehorned in a pet solution. So what is the right question?
Do you remember those word problems we used to have to do in school? Trains leaving the station at certain times and speeds, Mary and Bill trying to divide up a pie, and all those hypothetical problems?
The secret of those problems was often just figuring out what the problem was in the simplest terms. Once you rephrased the question, you often knew exactly which technique from that week’s lesson to apply to solve it. You could just whack the problem with all the techniques you had learned, hoping to hit the right one, but you would probably take too much time or might even get an “answer” that was totally off base.
To make this brute force approach even less likely to succeed, the teachers and textbook sometimes threw in irrelevant details that sounded important but had nothing to do with the real solution, in the same way that mystery writers throw in red herrings to make it harder to guess whodunit.
So the only reliable way to solve these problems correctly was to learn to filter out the extraneous blather and find the pithiest answer to the question “What problem am I really trying to solve?”
So how can we use this to solve our solutions dilemma? Let’s start by recognizing that the real problem is not what solution to choose but rather resolving the problem the solution is for. Trying to choose a tool without explicitly defining the real problem is exactly like trying to solve those word problems by throwing all the techniques you know at them. You may get lucky, but more likely you will make a mess.
If you realize, having thought about it, that the problem you are trying to solve is to avoid having others think you are ugly and stupid, then there is an easy solution to the problem: give up on technology altogether. In technology there is always someone telling you are ugly and stupid because you didn’t choose the same solution as them, so you might as well get used to it. If you want to go one step further, reject all forms of ideological thinking on principle, and craft your own rational set of criteria by which to judge things.
If you discover instead that there is some substantive problem you want to solve, spend some time trying to understand the problem. Ask yourself what the essential characteristics of the problem are. Identify some things that might normally be of interest but don’t really apply in this case or have less importance than normal. The truth is that no solution is perfect, all solutions require trade offs, and the secret of success is focusing on the must-haves while sacrificing the merely nice-to-haves or, more often, even the not-quite-so-must-haves.
Sometimes I find it very useful to start sketching out my own solution from scratch. At some point you realize either that you have a nice simple solution and don’t need to go any further, or that, now that you have a better feel for the problem and its challenges, that you have clear criteria upon which to base the choice of an external solution.
Anyone who wants to get good at software architecture has to be willing to throw out conventional wisdom and make choices that fit the problem at hand, not the problem that advocates of particular solutions tell them they should have. Sometimes just spending some quality time with your specific problem clarifies what you really need and melts the solutions dilemma away.
Since sometimes it seems that the choice you make can have big costs in terms of your project’s success (and your future employability), these dilemmas can result in a lot of stress. It doesn’t help that for each solution you have some proponent telling you that if you don’t use their approach you are stupid and ugly (see Linus Torvalds talking about his source control tool ).
So what’s the smart way to decide?
I think the first step is to realize that you have the wrong question. Technology people by definition tend to be interested in technology (tools and solutions) and we tend, as a result, to give short shrift to the problems themselves, especially when they fall outside our domain. As a result, solutions pile up faster than you can evaluate them, and problems often get shoehorned in a pet solution. So what is the right question?
Do you remember those word problems we used to have to do in school? Trains leaving the station at certain times and speeds, Mary and Bill trying to divide up a pie, and all those hypothetical problems?
The secret of those problems was often just figuring out what the problem was in the simplest terms. Once you rephrased the question, you often knew exactly which technique from that week’s lesson to apply to solve it. You could just whack the problem with all the techniques you had learned, hoping to hit the right one, but you would probably take too much time or might even get an “answer” that was totally off base.
To make this brute force approach even less likely to succeed, the teachers and textbook sometimes threw in irrelevant details that sounded important but had nothing to do with the real solution, in the same way that mystery writers throw in red herrings to make it harder to guess whodunit.
So the only reliable way to solve these problems correctly was to learn to filter out the extraneous blather and find the pithiest answer to the question “What problem am I really trying to solve?”
So how can we use this to solve our solutions dilemma? Let’s start by recognizing that the real problem is not what solution to choose but rather resolving the problem the solution is for. Trying to choose a tool without explicitly defining the real problem is exactly like trying to solve those word problems by throwing all the techniques you know at them. You may get lucky, but more likely you will make a mess.
If you realize, having thought about it, that the problem you are trying to solve is to avoid having others think you are ugly and stupid, then there is an easy solution to the problem: give up on technology altogether. In technology there is always someone telling you are ugly and stupid because you didn’t choose the same solution as them, so you might as well get used to it. If you want to go one step further, reject all forms of ideological thinking on principle, and craft your own rational set of criteria by which to judge things.
If you discover instead that there is some substantive problem you want to solve, spend some time trying to understand the problem. Ask yourself what the essential characteristics of the problem are. Identify some things that might normally be of interest but don’t really apply in this case or have less importance than normal. The truth is that no solution is perfect, all solutions require trade offs, and the secret of success is focusing on the must-haves while sacrificing the merely nice-to-haves or, more often, even the not-quite-so-must-haves.
Sometimes I find it very useful to start sketching out my own solution from scratch. At some point you realize either that you have a nice simple solution and don’t need to go any further, or that, now that you have a better feel for the problem and its challenges, that you have clear criteria upon which to base the choice of an external solution.
Anyone who wants to get good at software architecture has to be willing to throw out conventional wisdom and make choices that fit the problem at hand, not the problem that advocates of particular solutions tell them they should have. Sometimes just spending some quality time with your specific problem clarifies what you really need and melts the solutions dilemma away.
May 28, 2009
Functional Programming is Not All or Nothing
I’ve noticed a flurry of discussion around the blogosphere lately about what is and is not a functional programming language and what is or is not stateful. After being interested in functional programming (FP) for many years, I’m glad that it is finally getting some mainstream attention. I think FP has some valuable lessons for all programmers, and that it provides a better default mindset for programming than traditional imperative programming.
However, a lot of the discussion seems to make the mistake of assuming that functional programming and statefulness are all-or-nothing properties, that either you have a total purity of statelessness or you have opened Pandora’s Box and let all the evils of state into your program or programming language. But the truth is that functional programming is best viewed as a style of programming with different default assumptions about state. And statefulness is best viewed as a continuum that varies along at least two dimensions — locality and monotonicity — both of which I will explain shortly.
Before I do that, I had best describe in simple terms what state is. The easiest way to understand state is to imagine that I have given you a distinctive-looking box. If each time you open the box you find exactly the same contents, then the box can be said to be stateless. (This is also known as referential transparency). If each time you open the box the contents vary — the box could be empty sometimes, but full later, it could have different items in it or the same items with some new ones added — then it is stateful.
As you can see from this description, most boxes that we use in the real world are stateful, since it is very unusual for their contents to remain the same throughout their lifetime. However, if you are doing something complicated, say keeping financial records from past years in banker’s boxes, you might impose a discipline on your boxes, such as designating a particular box to contain only those financial records from April 2002, so that you can reliably find the records for a given month when you are looking for them.
Programming is like this as well. “Normal” imperative programming uses variables that can have changing values throughout the life of the program, which matches our understanding of boxes in the real world. FP takes the banker’s view of boxes. Since programs are very complex, and you want to be assured of finding what you expect, then the default assumption is that the contents of boxes will be the same throughout the life of the program.
So the key difference between a functional and an imperative style of programming is not whether there is any state at all in the program but where you start from. With imperative programming you start by assuming total changeability of all values you work with and have to add disciplines to get that state under control, whereas with FP you start from a place of statelessness and strategically loosen the reins in small amounts when you need to. The great strength of the FP approach is that it is much easier to maintain control by letting the genie out of the bottle slowly than by starting with the genie out of the bottle and trying to jam it back in.
The first dimension of state I want to discuss, locality, should be the easiest for programmers to understand but, for some reason, often causes confusion. Locality for state is just like scope for names in that it ranges from totally private to global, depending on how widely in your program it can be seen. A well-known property that makes state more local is encapsulation, when it is used to limit the visibility of a particular bit of state to a small part of the program. For encapsulation to accomplish this, though, it must actually hide the effects of the state from the parts of the program outside its scope.
To explain this, it is helpful to consider a common objection to functional programming languages by people new to the subject: how can a stateless program be built on a computer, which is an inherently stateful machine? The answer is that so long as the state is completely localized “under the hood”, it isn’t state at the next level up.
If we go back to the box analogy, let’s say that you have a very high-tech box that has the same contents every time you open it, but that, instead of the contents just being there when the box is closed, it actually disassembles the item when you close it and re-assembles it when you open the box again. The box is stateless because its contents never change from your point of view, even though its contents change when you can’t see inside.
So you can reduce the state in your program by ensuring that its various components look to each other as if they are stateless, even if internally they are implemented with large amounts of state, say for efficiency.
Monotonicity of state is a little bit more subtle and probably less consciously familiar to programmers. Monotonic is a fancy way to say that once you add something, you can’t take it away. Using our boxes, the simplest example of monotonic state would be if, when you first open a box, you might find either nothing or something, but, as soon as you open the box and find something there, you must find the same something in the box every time you look in it thereafter. Upon finding the box empty, you could stop what you are doing and wait for something to show up in the box, after which you could be sure that it was not going to change, or you could take the opportunity to fill the box with something you want to fill it with, confident that someone else won’t be able to change it later. This is how dataflow variables work.
A more complex example of monotonic state would be a stream.
Once you have asked for the first value from the stream, it is as if you have filled the first box in the sequence, even if the box for the next item in the stream might still be empty.
With non-monotonic state, items can be added and taken away from the box at will, so that the box might have one thing in it the first time you open it, nothing in it the second time, and something totally new the third time. This is the kind of state that we normally think about when we talk about default imperative statefulness.
You can see that, though monotonic state is still state, it is much more predictable and manageable then full non-monotonic state, since you can always tell how much has changed from the last time you checked and you can be assured that what you found before will still be there. Furthermore, there is an explicit order to the changes in state that allows you to understand the history of operations, and thereby to make better automated decisions about what to do.
This is why the kind of message-passing concurrency, such as is found in Erlang is less stateful and relatively safer and more scalable than full shared-state concurrency, since an asynchronous message queue can be thought of as a monotonically-stateful stream.
While it would be much easier to talk about functional programming if statefulness were a nice black and white property that could be assigned to a particular language or program, I hope it is clear now that things are not quite so simple, and that degrees of state and “functionalness” could be present in any language or program. Obviously some languages and programs are going to make reduced state the default, and thus easier to implement, but the functional perspective can be applied to any language or program.
And making use of this perspective will help tame the complexity of software, especially in the face of concurrency.
However, a lot of the discussion seems to make the mistake of assuming that functional programming and statefulness are all-or-nothing properties, that either you have a total purity of statelessness or you have opened Pandora’s Box and let all the evils of state into your program or programming language. But the truth is that functional programming is best viewed as a style of programming with different default assumptions about state. And statefulness is best viewed as a continuum that varies along at least two dimensions — locality and monotonicity — both of which I will explain shortly.
What is state?
Before I do that, I had best describe in simple terms what state is. The easiest way to understand state is to imagine that I have given you a distinctive-looking box. If each time you open the box you find exactly the same contents, then the box can be said to be stateless. (This is also known as referential transparency). If each time you open the box the contents vary — the box could be empty sometimes, but full later, it could have different items in it or the same items with some new ones added — then it is stateful.
As you can see from this description, most boxes that we use in the real world are stateful, since it is very unusual for their contents to remain the same throughout their lifetime. However, if you are doing something complicated, say keeping financial records from past years in banker’s boxes, you might impose a discipline on your boxes, such as designating a particular box to contain only those financial records from April 2002, so that you can reliably find the records for a given month when you are looking for them.
Two styles of programming
Programming is like this as well. “Normal” imperative programming uses variables that can have changing values throughout the life of the program, which matches our understanding of boxes in the real world. FP takes the banker’s view of boxes. Since programs are very complex, and you want to be assured of finding what you expect, then the default assumption is that the contents of boxes will be the same throughout the life of the program.
So the key difference between a functional and an imperative style of programming is not whether there is any state at all in the program but where you start from. With imperative programming you start by assuming total changeability of all values you work with and have to add disciplines to get that state under control, whereas with FP you start from a place of statelessness and strategically loosen the reins in small amounts when you need to. The great strength of the FP approach is that it is much easier to maintain control by letting the genie out of the bottle slowly than by starting with the genie out of the bottle and trying to jam it back in.
Locality of State
The first dimension of state I want to discuss, locality, should be the easiest for programmers to understand but, for some reason, often causes confusion. Locality for state is just like scope for names in that it ranges from totally private to global, depending on how widely in your program it can be seen. A well-known property that makes state more local is encapsulation, when it is used to limit the visibility of a particular bit of state to a small part of the program. For encapsulation to accomplish this, though, it must actually hide the effects of the state from the parts of the program outside its scope.
To explain this, it is helpful to consider a common objection to functional programming languages by people new to the subject: how can a stateless program be built on a computer, which is an inherently stateful machine? The answer is that so long as the state is completely localized “under the hood”, it isn’t state at the next level up.
If we go back to the box analogy, let’s say that you have a very high-tech box that has the same contents every time you open it, but that, instead of the contents just being there when the box is closed, it actually disassembles the item when you close it and re-assembles it when you open the box again. The box is stateless because its contents never change from your point of view, even though its contents change when you can’t see inside.
So you can reduce the state in your program by ensuring that its various components look to each other as if they are stateless, even if internally they are implemented with large amounts of state, say for efficiency.
Monotonicity of State
Monotonicity of state is a little bit more subtle and probably less consciously familiar to programmers. Monotonic is a fancy way to say that once you add something, you can’t take it away. Using our boxes, the simplest example of monotonic state would be if, when you first open a box, you might find either nothing or something, but, as soon as you open the box and find something there, you must find the same something in the box every time you look in it thereafter. Upon finding the box empty, you could stop what you are doing and wait for something to show up in the box, after which you could be sure that it was not going to change, or you could take the opportunity to fill the box with something you want to fill it with, confident that someone else won’t be able to change it later. This is how dataflow variables work.
A more complex example of monotonic state would be a stream.
Once you have asked for the first value from the stream, it is as if you have filled the first box in the sequence, even if the box for the next item in the stream might still be empty.
With non-monotonic state, items can be added and taken away from the box at will, so that the box might have one thing in it the first time you open it, nothing in it the second time, and something totally new the third time. This is the kind of state that we normally think about when we talk about default imperative statefulness.
You can see that, though monotonic state is still state, it is much more predictable and manageable then full non-monotonic state, since you can always tell how much has changed from the last time you checked and you can be assured that what you found before will still be there. Furthermore, there is an explicit order to the changes in state that allows you to understand the history of operations, and thereby to make better automated decisions about what to do.
This is why the kind of message-passing concurrency, such as is found in Erlang is less stateful and relatively safer and more scalable than full shared-state concurrency, since an asynchronous message queue can be thought of as a monotonically-stateful stream.
Conclusion
While it would be much easier to talk about functional programming if statefulness were a nice black and white property that could be assigned to a particular language or program, I hope it is clear now that things are not quite so simple, and that degrees of state and “functionalness” could be present in any language or program. Obviously some languages and programs are going to make reduced state the default, and thus easier to implement, but the functional perspective can be applied to any language or program.
And making use of this perspective will help tame the complexity of software, especially in the face of concurrency.
May 13, 2009
Metaprogramming still needs programming discipline
This post will be a little bit more technical than previous posts, but I’ll try to keep it comprehensible to those who don’t already know what I’m talking about.
I want to talk about meta-programming. Meta-programming, in the most general sense, means making programs that produce programs or that change what existing programs do by altering the environment in which they operate. Writing an interpreter or a compiler for a programming language can be considered an example of meta-programming. Macros, monkey-patching and code generation are also examples. I would argue that the XML “configuration” files that haunt many Java frameworks, such as Spring, Hibernate, Struts, etc., are a form of meta-programming.
Now meta-programming is a very important idea, and I would go so far as to say that everyone who is at all serious about programming should learn about and understand meta-programming, even if only at a basic level. Meta-programming is often considered an advanced topic, and there certainly are advanced forms of it and advanced ideas that fall under its domain, but I think that anyone who is smart enough to program at all is smart enough to understand and perform basic meta-programming.
Now, since meta-programming is very important, has deep things to say about computation, and is intellectually stimulating, many people find it very exciting and even beautiful.
I will confess: I am one of those people. I have spent a non-trivial chunk of my leisure time throughout my adult life studying the semantics of programming languages, and all the supporting theories and math. And though knowing this stuff has been directly and indirectly useful in my software development career, I really did it because I enjoy it, and because I think it is beautiful and exciting.
Having said all these great things about meta-programming, whenever I see someone using meta-programming in a project, I get queasy. And that is because people often forget that just because meta-programming seems to let you go beyond the rules of “mere” programming, it still requires all the same discipline you would apply to any other kind of programming. For example, you still need to remember source code discipline and the enforcement of locality principles, such as modularity, the single-responsibility principle, encapsulation, don’t-repeat-yourself (DRY), and many others.
Take DRY for example. I have seen many programmers who would never tolerate cut-and-paste boilerplate in the source code blithely create megabytes worth of XML “configuration” files, or use a source code generator to do the same thing, even if there might be more acceptable alternatives with a bit of creativity.
Moreover, I think there are two kinds of locality that meta-programming should have that wouldn’t apply to single-level programming. First, meta-programming level code should be modularized away from the programming level code, and second, any domain-specific-languages (DSLs) or language variants created by the meta-programming should be clearly demarcated from the “normal” programming language (in their own source files if possible).
The reason for this is simple. Imagine if I started writing this post by alternating between English and some other language, say French. Some sentences or phrases in one language, some in the other. First of all, any member of the audience who is not fluent in both will probably be lost immediately. For those who are bilingual, there are many words that are spelled the same or very similarly in both languages, but may not have the same meaning, either subtly or not so subtly. So even if you are fluent in both, aside from the difficulty I’m adding by making you code switch, I may also cause you to misinterpret what I’m saying with similar or ambiguous words.
Just as the user of a programming library only wants to have to think about the interface to that library and not have to understand the gory details of how it actually works, the consumers of meta-programming “enhancements” need to be insulated from having to understand the details of how it works under the hood. This is where non-local meta-programming, such as reckless global monkey-patching, can really mess things up.
So anyone who wants to keep meta-programming beautiful should use the same judgment, taste and discipline that an experienced programmer would apply to keep any other type of programming beautiful. Otherwise, it can morph into something very, very ugly.
I want to talk about meta-programming. Meta-programming, in the most general sense, means making programs that produce programs or that change what existing programs do by altering the environment in which they operate. Writing an interpreter or a compiler for a programming language can be considered an example of meta-programming. Macros, monkey-patching and code generation are also examples. I would argue that the XML “configuration” files that haunt many Java frameworks, such as Spring, Hibernate, Struts, etc., are a form of meta-programming.
Now meta-programming is a very important idea, and I would go so far as to say that everyone who is at all serious about programming should learn about and understand meta-programming, even if only at a basic level. Meta-programming is often considered an advanced topic, and there certainly are advanced forms of it and advanced ideas that fall under its domain, but I think that anyone who is smart enough to program at all is smart enough to understand and perform basic meta-programming.
Now, since meta-programming is very important, has deep things to say about computation, and is intellectually stimulating, many people find it very exciting and even beautiful.
I will confess: I am one of those people. I have spent a non-trivial chunk of my leisure time throughout my adult life studying the semantics of programming languages, and all the supporting theories and math. And though knowing this stuff has been directly and indirectly useful in my software development career, I really did it because I enjoy it, and because I think it is beautiful and exciting.
Having said all these great things about meta-programming, whenever I see someone using meta-programming in a project, I get queasy. And that is because people often forget that just because meta-programming seems to let you go beyond the rules of “mere” programming, it still requires all the same discipline you would apply to any other kind of programming. For example, you still need to remember source code discipline and the enforcement of locality principles, such as modularity, the single-responsibility principle, encapsulation, don’t-repeat-yourself (DRY), and many others.
Take DRY for example. I have seen many programmers who would never tolerate cut-and-paste boilerplate in the source code blithely create megabytes worth of XML “configuration” files, or use a source code generator to do the same thing, even if there might be more acceptable alternatives with a bit of creativity.
Moreover, I think there are two kinds of locality that meta-programming should have that wouldn’t apply to single-level programming. First, meta-programming level code should be modularized away from the programming level code, and second, any domain-specific-languages (DSLs) or language variants created by the meta-programming should be clearly demarcated from the “normal” programming language (in their own source files if possible).
The reason for this is simple. Imagine if I started writing this post by alternating between English and some other language, say French. Some sentences or phrases in one language, some in the other. First of all, any member of the audience who is not fluent in both will probably be lost immediately. For those who are bilingual, there are many words that are spelled the same or very similarly in both languages, but may not have the same meaning, either subtly or not so subtly. So even if you are fluent in both, aside from the difficulty I’m adding by making you code switch, I may also cause you to misinterpret what I’m saying with similar or ambiguous words.
Just as the user of a programming library only wants to have to think about the interface to that library and not have to understand the gory details of how it actually works, the consumers of meta-programming “enhancements” need to be insulated from having to understand the details of how it works under the hood. This is where non-local meta-programming, such as reckless global monkey-patching, can really mess things up.
So anyone who wants to keep meta-programming beautiful should use the same judgment, taste and discipline that an experienced programmer would apply to keep any other type of programming beautiful. Otherwise, it can morph into something very, very ugly.
May 3, 2009
Good to Great
I finally got around to reading the book Good to Great. I first heard about it in early 2002 in the introductory speech of the incoming president at an organization I was in the process of leaving. I was quite impressed with what he had to say about leadership and organizational greatness and made a note to read the book as soon as it came out in paperback. (Hardcovers use up too much shelf space, which is at a premium in my home, so I tend to avoid buying them.) Since it is still in hardcover all these years later, I gave up and borrowed a copy from my brother.
There is a lot of criticism of the book online (here, here and here), but I think most of it misses the point. It doesn’t matter if you are happy with the experimental model the book was based on, or if the companies profiled didn’t stay good stock picks forever after the book came out, or if the advice Collins gives boils down to “obvious” suggestions.
The best business books tell you in an organized and compelling way what you already know to be true from your own experience. No business book can predict the future or give you a sure fire recipe for success, and Good to Great doesn’t pretend to anyway, so why do the critics expect it to?
What I did find in this book was a well-written and thoughtful explanation of the only reliable strategy I know of to accomplish any undertaking: for a disciplined group of people to pursue a focused goal in a determined manner, while being willing to see failure.
Now one can say “Hey, that’s obvious!”. But unfortunately, the obvious is more often observed in the breech than in the practice. The path of least resistance in many organizations is to let ineffective or obstructive people stick around way too long, to let the day-to-day crises and tempests-in-a-teapot derail their long-term goals, and to rationalize or ignore failures.
If Good to Great manages to inspire people by reminding them to struggle against these tendencies and to set a higher standard toward greatness, I think it deserves its place on the best-seller list.
There is a lot of criticism of the book online (here, here and here), but I think most of it misses the point. It doesn’t matter if you are happy with the experimental model the book was based on, or if the companies profiled didn’t stay good stock picks forever after the book came out, or if the advice Collins gives boils down to “obvious” suggestions.
The best business books tell you in an organized and compelling way what you already know to be true from your own experience. No business book can predict the future or give you a sure fire recipe for success, and Good to Great doesn’t pretend to anyway, so why do the critics expect it to?
What I did find in this book was a well-written and thoughtful explanation of the only reliable strategy I know of to accomplish any undertaking: for a disciplined group of people to pursue a focused goal in a determined manner, while being willing to see failure.
Now one can say “Hey, that’s obvious!”. But unfortunately, the obvious is more often observed in the breech than in the practice. The path of least resistance in many organizations is to let ineffective or obstructive people stick around way too long, to let the day-to-day crises and tempests-in-a-teapot derail their long-term goals, and to rationalize or ignore failures.
If Good to Great manages to inspire people by reminding them to struggle against these tendencies and to set a higher standard toward greatness, I think it deserves its place on the best-seller list.
May 2, 2009
Dancing Monkeys
Many years ago, before I started my software development career, I had a job call-center representative for a national automobile association. This included a twelve-hour shift by myself on Sundays.
The Sunday shift was a bit of a grind. It could be a long, silent, boring wait for calls that never came, or if the weather was inclement somewhere on the other side of the country, it could be very busy handling a spate of calls all alone.
During the normal workweek, when there were a bunch of us working, we would pass the slow times doing paperwork. By Sunday, when I was often the only person in the building, the paperwork was usually all done, so to pass the time I took to listening to the radio. As soon as the phone rang I would shut the radio off and pick up. There was no doubt in my mind that I was doing my job fully and faithfully, in spite of the discomfort and difficulty of the job.
One Sunday, the Vice President of the company decided to come to the office to catch up on some work. At some point, he wandered back to the call-center where it was a slow day, and I was sitting there by myself listening to the radio. He may have greeted me, or asked me how busy I was, but he didn’t stay long and he didn’t say anything significant.
However, on Monday I heard from my boss that he had complained that I was listening to the radio instead of working.
This was my first significant experience of a phenomenon that I call “Dancing Monkeys”: the tendency of arms-length leadership to prefer situations where everyone “looks busy” or is “hopping to”, even when the expected effort would have no additional effect or might even be counterproductive.
There are many reasons for this phenomenon some pernicious and some benign. The pernicious ones that we will all have seen at some time or another is pure ego-tripping: “I’m the top monkey here, so you lesser monkeys start dancing!”
A common, more benign reason is lack of understanding of the work domain under observation. Software development is particularly prone to this one.
For example, more than once I’ve heard some executive complain that he stopped by the development team area and no one was typing, with the implication being that no work was getting done, since as every non-technical person knows programming is all about lines per minute of code typed.
Another common scenario is that inevitable day when, under some unforeseen deadline pressure, some executive asks the development manager when he is going to institute overtime to help meet the deadline. Because of course, as any non-programmer knows, there is no degradation to the quality of software development when the team is exhausted, since typing lines of code is a purely mechanical process.
To come back to my call-center job, effective execution of my duties required that when I got a run of long, stressful calls, I had the energy and focus needed to solve the clients’ problems efficiently and effectively. Anyone who has ever made a service call to a droning zombie service rep knows what happens when someone doing that job lets their energy reserve run down.
Listening to the radio helped me recharge between calls, kept my morale up on quiet days and improved my performance. Doing mindless, unnecessary busy-work would have sapped that energy and morale. Asking my boss to find work for me to do that was meaningful but would not interrupt my real work would have just increased her workload without increasing the efficiency or effectiveness of our department.
So that Vice President had to ask himself: was making himself feel better by getting a dancing monkey really in the best interest of his company?
The Sunday shift was a bit of a grind. It could be a long, silent, boring wait for calls that never came, or if the weather was inclement somewhere on the other side of the country, it could be very busy handling a spate of calls all alone.
During the normal workweek, when there were a bunch of us working, we would pass the slow times doing paperwork. By Sunday, when I was often the only person in the building, the paperwork was usually all done, so to pass the time I took to listening to the radio. As soon as the phone rang I would shut the radio off and pick up. There was no doubt in my mind that I was doing my job fully and faithfully, in spite of the discomfort and difficulty of the job.
One Sunday, the Vice President of the company decided to come to the office to catch up on some work. At some point, he wandered back to the call-center where it was a slow day, and I was sitting there by myself listening to the radio. He may have greeted me, or asked me how busy I was, but he didn’t stay long and he didn’t say anything significant.
However, on Monday I heard from my boss that he had complained that I was listening to the radio instead of working.
This was my first significant experience of a phenomenon that I call “Dancing Monkeys”: the tendency of arms-length leadership to prefer situations where everyone “looks busy” or is “hopping to”, even when the expected effort would have no additional effect or might even be counterproductive.
There are many reasons for this phenomenon some pernicious and some benign. The pernicious ones that we will all have seen at some time or another is pure ego-tripping: “I’m the top monkey here, so you lesser monkeys start dancing!”
A common, more benign reason is lack of understanding of the work domain under observation. Software development is particularly prone to this one.
For example, more than once I’ve heard some executive complain that he stopped by the development team area and no one was typing, with the implication being that no work was getting done, since as every non-technical person knows programming is all about lines per minute of code typed.
Another common scenario is that inevitable day when, under some unforeseen deadline pressure, some executive asks the development manager when he is going to institute overtime to help meet the deadline. Because of course, as any non-programmer knows, there is no degradation to the quality of software development when the team is exhausted, since typing lines of code is a purely mechanical process.
To come back to my call-center job, effective execution of my duties required that when I got a run of long, stressful calls, I had the energy and focus needed to solve the clients’ problems efficiently and effectively. Anyone who has ever made a service call to a droning zombie service rep knows what happens when someone doing that job lets their energy reserve run down.
Listening to the radio helped me recharge between calls, kept my morale up on quiet days and improved my performance. Doing mindless, unnecessary busy-work would have sapped that energy and morale. Asking my boss to find work for me to do that was meaningful but would not interrupt my real work would have just increased her workload without increasing the efficiency or effectiveness of our department.
So that Vice President had to ask himself: was making himself feel better by getting a dancing monkey really in the best interest of his company?
April 2, 2009
Ideology vs Principles
Ideologies are strangely seductive.
Ideologies promise a sure-fire recipe to solve all problems. They often give you a single value to maximize, and equally often tell you who to blame.
Got a problem? Easy. Just trust the free market. Or increase government spending. Or blame the wealthy. Or blame the poor.
Whichever solution an ideology offers, that is supposed to be the first solution to apply whenever a problem arises, and if that doesn't work you're supposed to do apply it again until it does work.
Given the complexity of some of the problems that face us in our daily lives as individuals and even more so in our collective life as a society, it is easy to see the allure of straightforward solutions you don't have to think about too much.
But history has shown us over and over again that ideologies are fairly poor at solving problems in the short run, and in the long run usually create worse problems of their own.
If you want effective problem solving, you're better off with principles rather than ideology.
A set of principles is like an ideology in that it expresses values upon which to base decisions, but principles can't be applied unconditionally. Principles compete with each other, forcing you to make trade-off decisions. The exact trade-offs have to be worked out differently for each instance of a problem.
Ideology is like the imaginary fifties family where everyone listens to Dad and the kids never compete for attention. Principles are like a real family, where you love everybody, but you can't always make everybody happy.
The amount of work required to implement a decision is about the same for both ideological and principled approaches is. The real difference is in the amount of certainty you have, both before you choose a solution, and after you've made a choice.
In a principled approach, you always have to wonder if you could have gotten the balance of competing values better, and you may have to constantly rejig the balance as new situations develop. With an ideology, you always have it right, even if you aren't getting the results you want.
As nice a feeling as it is to be certain, if you really care about outcomes, uncertainty has the benefit that it keeps you more closely in touch with the realities of your problem. It makes you listen more attentively to the feedback from your solution. You are much more likely to succeed in the end, even if paradoxically you are less sure at any one point in time that you have found the right solution.
Abandoning certainty leads to greater confidence of real success.
This idea has so many applications to software development, project management, government, and other governance and systems thinking domains, I'll leave elaboration to future posts.
Ideologies promise a sure-fire recipe to solve all problems. They often give you a single value to maximize, and equally often tell you who to blame.
Got a problem? Easy. Just trust the free market. Or increase government spending. Or blame the wealthy. Or blame the poor.
Whichever solution an ideology offers, that is supposed to be the first solution to apply whenever a problem arises, and if that doesn't work you're supposed to do apply it again until it does work.
Given the complexity of some of the problems that face us in our daily lives as individuals and even more so in our collective life as a society, it is easy to see the allure of straightforward solutions you don't have to think about too much.
But history has shown us over and over again that ideologies are fairly poor at solving problems in the short run, and in the long run usually create worse problems of their own.
If you want effective problem solving, you're better off with principles rather than ideology.
A set of principles is like an ideology in that it expresses values upon which to base decisions, but principles can't be applied unconditionally. Principles compete with each other, forcing you to make trade-off decisions. The exact trade-offs have to be worked out differently for each instance of a problem.
Ideology is like the imaginary fifties family where everyone listens to Dad and the kids never compete for attention. Principles are like a real family, where you love everybody, but you can't always make everybody happy.
The amount of work required to implement a decision is about the same for both ideological and principled approaches is. The real difference is in the amount of certainty you have, both before you choose a solution, and after you've made a choice.
In a principled approach, you always have to wonder if you could have gotten the balance of competing values better, and you may have to constantly rejig the balance as new situations develop. With an ideology, you always have it right, even if you aren't getting the results you want.
As nice a feeling as it is to be certain, if you really care about outcomes, uncertainty has the benefit that it keeps you more closely in touch with the realities of your problem. It makes you listen more attentively to the feedback from your solution. You are much more likely to succeed in the end, even if paradoxically you are less sure at any one point in time that you have found the right solution.
Abandoning certainty leads to greater confidence of real success.
This idea has so many applications to software development, project management, government, and other governance and systems thinking domains, I'll leave elaboration to future posts.
March 30, 2009
Explaining through Skillful Means
In Buddhist philosophy, there is a concept whose Sanskrit name is upaya. This is often translated as "skillful means", though, as with many other ancient philosophical terms, it is hard to translate exactly and has many interpretations.
To illustrate one useful interpretation, I'll start by explaining a little bit about Buddhism. If you boil Buddhism down to a one-sentence summary, it is about reducing your suffering (and that of the world) by living a moderate life and practicing meditation.
As simple as this sounds, there is a lot of theory behind it, covering — for example — what it means to live a moderate life, why that helps reduce suffering, what good meditation is and how it works, and a host of other questions.
The theory and practice of Buddhism is in fact so complex that it is usually taken for granted that it takes years of study and practice to really understand it. To make matters worse, there are many different opinions about what kind of study and practice, and how much.
But most of the concepts can be understood at first in a simplified way, and since all students of any complex skill have to start somewhere, people often start with these simplified forms that aren’t quite right to a more advanced student.
For example, there is an uncomplicated form of Buddhism called Pure Land, that more or less believes that if you faithfully chant a particular mantra, you will be reborn in a special paradise. You can still see in this belief some of the elements in my one-sentence summary above: chanting a mantra is a form of meditation, being reborn in a paradise is a reduction of suffering, and faithfully practicing a good habit is a rudimentary form of living a disciplined life.
So even though a “more advanced” student of Buddhism might roll their eyes at the simplicity of this approach, they would have to admit that practicing it was still an advancement in the direction of the aims of Buddhism, and therefore better than no understanding of Buddhist principles.
This, in a nutshell, is what upaya means: it is OK to let someone have a “wrong” idea about something you are explaining to them, so long as it takes them one step closer to understanding.
What does this have to do with software development?
Software development, boiled down, is about solving practical problems by producing reliable applications with maintainable code bases. (All of these things, incidentally, also reduce suffering. ;-) )
There is a complex skill set and complex methodologies that must be learned to accomplish this task. It is taken for granted (at least by some people) that mastering them takes years of study and practice, but there are many different opinions about what kind of study and practice, and how much.
But many concepts have a simplified version, and since all learners have to start somewhere, you often see versions of good practices that a more advanced practitioner would see as not quite right.
For example, a developer with less experience — or perhaps just different experience — might chant a mantra such as “always document”, “never document”, “always design before you program”, “never design before your program”, or “paradigm/language/library/framework X is always better than Y”. These mantras might strike you as over-simplified or incorrect based on your experience (especially when you favour an opposite version of the chant. ;-) ).
You can still see that they share the core concern of solving the practical problem reliably and maintainably. Maybe it is OK to let them be “wrong” for now, so long as they are engaged with the ultimate concerns of software development and are using the mantra to get one step closer to understanding.
When mentoring developers as a team lead or when introducing new ideas, techniques and methodologies to your peers or to an on-line community, it can be hard to determine when to hold a hard line, to nitpick or to qualify and when not to. Often it is best to introduce a simplified form of the idea and let them run with it for a while, trusting that if they share the same goals they will come to a deeper understanding on their own and that they will ask for more input when they need it.
Sometimes the best policy is to let others be "wrong" for the time being.
To illustrate one useful interpretation, I'll start by explaining a little bit about Buddhism. If you boil Buddhism down to a one-sentence summary, it is about reducing your suffering (and that of the world) by living a moderate life and practicing meditation.
As simple as this sounds, there is a lot of theory behind it, covering — for example — what it means to live a moderate life, why that helps reduce suffering, what good meditation is and how it works, and a host of other questions.
The theory and practice of Buddhism is in fact so complex that it is usually taken for granted that it takes years of study and practice to really understand it. To make matters worse, there are many different opinions about what kind of study and practice, and how much.
But most of the concepts can be understood at first in a simplified way, and since all students of any complex skill have to start somewhere, people often start with these simplified forms that aren’t quite right to a more advanced student.
For example, there is an uncomplicated form of Buddhism called Pure Land, that more or less believes that if you faithfully chant a particular mantra, you will be reborn in a special paradise. You can still see in this belief some of the elements in my one-sentence summary above: chanting a mantra is a form of meditation, being reborn in a paradise is a reduction of suffering, and faithfully practicing a good habit is a rudimentary form of living a disciplined life.
So even though a “more advanced” student of Buddhism might roll their eyes at the simplicity of this approach, they would have to admit that practicing it was still an advancement in the direction of the aims of Buddhism, and therefore better than no understanding of Buddhist principles.
This, in a nutshell, is what upaya means: it is OK to let someone have a “wrong” idea about something you are explaining to them, so long as it takes them one step closer to understanding.
What does this have to do with software development?
Software development, boiled down, is about solving practical problems by producing reliable applications with maintainable code bases. (All of these things, incidentally, also reduce suffering. ;-) )
There is a complex skill set and complex methodologies that must be learned to accomplish this task. It is taken for granted (at least by some people) that mastering them takes years of study and practice, but there are many different opinions about what kind of study and practice, and how much.
But many concepts have a simplified version, and since all learners have to start somewhere, you often see versions of good practices that a more advanced practitioner would see as not quite right.
For example, a developer with less experience — or perhaps just different experience — might chant a mantra such as “always document”, “never document”, “always design before you program”, “never design before your program”, or “paradigm/language/library/framework X is always better than Y”. These mantras might strike you as over-simplified or incorrect based on your experience (especially when you favour an opposite version of the chant. ;-) ).
You can still see that they share the core concern of solving the practical problem reliably and maintainably. Maybe it is OK to let them be “wrong” for now, so long as they are engaged with the ultimate concerns of software development and are using the mantra to get one step closer to understanding.
When mentoring developers as a team lead or when introducing new ideas, techniques and methodologies to your peers or to an on-line community, it can be hard to determine when to hold a hard line, to nitpick or to qualify and when not to. Often it is best to introduce a simplified form of the idea and let them run with it for a while, trusting that if they share the same goals they will come to a deeper understanding on their own and that they will ask for more input when they need it.
Sometimes the best policy is to let others be "wrong" for the time being.
March 29, 2009
Be Willing to See Failure
There is an old saying that I quite like: "If you’re never failing, you aren't trying."
The simplest way to read this adage is as an exhortation to take more risks, to stop playing it safe and undertake bigger and scarier projects.
An equally valuable second reading is to take this as a salve to failure -- it is OK to have failed or had a set back because it means you were trying something challenging.
But there is a subtler reading that I want to explore that comes into play when you are trying, and you doseem to be succeeding, that addresses a more dangerous form of failure: the failure to perceive failure.
Anyone who has spent any time doing development projects (for software or otherwise), has probably heard of, if not actively participated in, a project that went along just fine for months, only to burst into flame as some final deadline approached. And by going along just fine, I mean that people were trying: meetings were held, decisions were made, mountains of code were written, documents filled the shelves, metrics were monitored.
But mysteriously, as the looming deadline approached, all kinds of critical problems seemingly popped out of nowhere, delaying delivery, running up costs, creating a sudden, unforeseen crisis.
(Sounds a lot like our current world-wide financial crisis, doesn't it?)
So what went wrong?
Rarely do problems just pop out of nowhere. More often, we simply become aware of them suddenly. And sometimes that’s because we didn’t want to be aware of them, because we wanted to focus on the success instead of the failure.
Unfortunately, I think this is a major challenge in our North American cultural temperament. (People from other locales can assess this for their own situation.)
We tend to value optimism, the power of positive thinking, the positive spin to the point where any talk of real risks or real failings tends to sound like pessimism or cattiness. We tend to assume that those who question a person, project or process that seems to be succeeding are guilty of sour grapes or defeatism.
So a sub-prime mortgage crisis can "sneak up on us" and bring down an Alice-in-Wonderland financial system, and a project that was "going along fine" can suddenly go off the rails.
The solution to this problem is to accept that failures and problems are, as the adage suggests, a necessary side effect of trying to accomplish something, and that success doesn’t come from ignoring problems but from finding creative and effective solutions to solve or mitigate them. And once we’ve accepted this, we can change our team culture and practices to help flush problems and failures out of the bushes sooner.
In software development, there are a number of techniques that are gaining wide acceptance, such as iterative delivery, open communication between clients and teams, continuous integration, test-driven development, and others, that solve this very problem.
These techniques all work by creating more frequent opportunities to put a project through its paces, so that weaknesses, misunderstandings and oversights are spotted sooner, and so that successes have really proven themselves, and can be safely built upon.
A good rule of thumb is that if you are aren’t routinely finding unexpected problems or failures when using such techniques, you probably aren’t using them rigorously enough.
Which leads us back to the opening adage, but maybe in this context it should be changed to: "If you’re never failing, you aren't looking."
The simplest way to read this adage is as an exhortation to take more risks, to stop playing it safe and undertake bigger and scarier projects.
An equally valuable second reading is to take this as a salve to failure -- it is OK to have failed or had a set back because it means you were trying something challenging.
But there is a subtler reading that I want to explore that comes into play when you are trying, and you doseem to be succeeding, that addresses a more dangerous form of failure: the failure to perceive failure.
Anyone who has spent any time doing development projects (for software or otherwise), has probably heard of, if not actively participated in, a project that went along just fine for months, only to burst into flame as some final deadline approached. And by going along just fine, I mean that people were trying: meetings were held, decisions were made, mountains of code were written, documents filled the shelves, metrics were monitored.
But mysteriously, as the looming deadline approached, all kinds of critical problems seemingly popped out of nowhere, delaying delivery, running up costs, creating a sudden, unforeseen crisis.
(Sounds a lot like our current world-wide financial crisis, doesn't it?)
So what went wrong?
Rarely do problems just pop out of nowhere. More often, we simply become aware of them suddenly. And sometimes that’s because we didn’t want to be aware of them, because we wanted to focus on the success instead of the failure.
Unfortunately, I think this is a major challenge in our North American cultural temperament. (People from other locales can assess this for their own situation.)
We tend to value optimism, the power of positive thinking, the positive spin to the point where any talk of real risks or real failings tends to sound like pessimism or cattiness. We tend to assume that those who question a person, project or process that seems to be succeeding are guilty of sour grapes or defeatism.
So a sub-prime mortgage crisis can "sneak up on us" and bring down an Alice-in-Wonderland financial system, and a project that was "going along fine" can suddenly go off the rails.
The solution to this problem is to accept that failures and problems are, as the adage suggests, a necessary side effect of trying to accomplish something, and that success doesn’t come from ignoring problems but from finding creative and effective solutions to solve or mitigate them. And once we’ve accepted this, we can change our team culture and practices to help flush problems and failures out of the bushes sooner.
In software development, there are a number of techniques that are gaining wide acceptance, such as iterative delivery, open communication between clients and teams, continuous integration, test-driven development, and others, that solve this very problem.
These techniques all work by creating more frequent opportunities to put a project through its paces, so that weaknesses, misunderstandings and oversights are spotted sooner, and so that successes have really proven themselves, and can be safely built upon.
A good rule of thumb is that if you are aren’t routinely finding unexpected problems or failures when using such techniques, you probably aren’t using them rigorously enough.
Which leads us back to the opening adage, but maybe in this context it should be changed to: "If you’re never failing, you aren't looking."
March 28, 2009
Minimize Locally, Succeed Globally
My first full-time programmer position was in the research institute of a nationally-known hospital, building database applications for research data and administration.
I have often joked about that time that I was a one-man development cycle, since I had was responsible for the process from beginning to end.
I was the "sales guy", drumming up business with internal clients. I was the business analyst, modeling the often complicated problem domain. I was also the programmer, the system administrator, and the tech support guy.
To complicate things further, there were a fair number of projects, some small, some dormant, but also some that were quite large, complex and active, and which grew more so over time. Eventually, I also added the roles of project manager and dev team lead when a family of the projects coalesced into a behemoth.
As you can imagine, having so many roles on so many projects was a bit of a tightrope walk. To survive, I had to constantly ask myself: "What is the simplest and most reliable way I can do this?"
As the "sales guy" (we did internal billing), there was no benefit to making grandiose promises to drum up more sales than I could safely deliver. I could only handle so many projects anyway. And failing to deliver on a couple projects would quickly have poisoned the well.
As the business analyst, I had no reason to nurture technicolor dreams for a particular project. I was going to have to implement and support it as well, so I had to just solve the priority problems elegantly and efficiently, with no frills.
As the programmer, I had no motivation to demonstrate my genius with complex algorithms and impenetrable code, or by layering on the latest tech toys. I needed my code to be easy to understand and fix, because I wasn't going to have a lot of time to build it or maintain it. Simplicity and clarity also helped with stability and reliability, which was good, since if the application wasn't stable and reliable, I was going to generate more tech support pain than I could deal with.
And since I didn't have time to be a tech writer and software trainer as well, I had to sacrifice a slick look or gee-whiz features to make simple, clearly-labeled interfaces that the most technophobic administrative assistant would be able to figure out reasonably well in 10 minutes or less. Otherwise, I was going to add the role of professional hand-holder to my roster.
Of course, in spite of my best efforts, none of this worked out 100% of the time, and I dreamed of the day when I would be using the heavy-duty tech on the big projects with the big boys, where most of my roles would be fulfilled by other people. And I got my wish soon enough.
But I discovered a hidden danger with larger projects where most of the participants don't have sight of the whole process: that each role maximizes itself to the detriment of the outcome.
The sales rep makes unfulfillable promises to get his sales. The business analyst gold-plates and over-specifies the requirements. The programmers over-engineer with too many libraries, frameworks, cool features and clever code. The tech support guys, who, to be fair, do the most suffering on the front lines when things go awry, try to toss some of their woes back over the wall to the developers.
And if this tendency isn't actively countered, instead of building bigger, better software, they start to build software that costs more, is less reliable, and less usable.
So now that I've seen both sides, I think maybe necessity had put me on to something way back then: being minimalist with the requirements, efforts and costs at each step in the process in order to optimize the workings of the whole.
And I think an organization that manages to foster that ethic through good communication and good leadership will produce the best software for the lowest cost in time and money.
I have often joked about that time that I was a one-man development cycle, since I had was responsible for the process from beginning to end.
I was the "sales guy", drumming up business with internal clients. I was the business analyst, modeling the often complicated problem domain. I was also the programmer, the system administrator, and the tech support guy.
To complicate things further, there were a fair number of projects, some small, some dormant, but also some that were quite large, complex and active, and which grew more so over time. Eventually, I also added the roles of project manager and dev team lead when a family of the projects coalesced into a behemoth.
As you can imagine, having so many roles on so many projects was a bit of a tightrope walk. To survive, I had to constantly ask myself: "What is the simplest and most reliable way I can do this?"
As the "sales guy" (we did internal billing), there was no benefit to making grandiose promises to drum up more sales than I could safely deliver. I could only handle so many projects anyway. And failing to deliver on a couple projects would quickly have poisoned the well.
As the business analyst, I had no reason to nurture technicolor dreams for a particular project. I was going to have to implement and support it as well, so I had to just solve the priority problems elegantly and efficiently, with no frills.
As the programmer, I had no motivation to demonstrate my genius with complex algorithms and impenetrable code, or by layering on the latest tech toys. I needed my code to be easy to understand and fix, because I wasn't going to have a lot of time to build it or maintain it. Simplicity and clarity also helped with stability and reliability, which was good, since if the application wasn't stable and reliable, I was going to generate more tech support pain than I could deal with.
And since I didn't have time to be a tech writer and software trainer as well, I had to sacrifice a slick look or gee-whiz features to make simple, clearly-labeled interfaces that the most technophobic administrative assistant would be able to figure out reasonably well in 10 minutes or less. Otherwise, I was going to add the role of professional hand-holder to my roster.
Of course, in spite of my best efforts, none of this worked out 100% of the time, and I dreamed of the day when I would be using the heavy-duty tech on the big projects with the big boys, where most of my roles would be fulfilled by other people. And I got my wish soon enough.
But I discovered a hidden danger with larger projects where most of the participants don't have sight of the whole process: that each role maximizes itself to the detriment of the outcome.
The sales rep makes unfulfillable promises to get his sales. The business analyst gold-plates and over-specifies the requirements. The programmers over-engineer with too many libraries, frameworks, cool features and clever code. The tech support guys, who, to be fair, do the most suffering on the front lines when things go awry, try to toss some of their woes back over the wall to the developers.
And if this tendency isn't actively countered, instead of building bigger, better software, they start to build software that costs more, is less reliable, and less usable.
So now that I've seen both sides, I think maybe necessity had put me on to something way back then: being minimalist with the requirements, efforts and costs at each step in the process in order to optimize the workings of the whole.
And I think an organization that manages to foster that ethic through good communication and good leadership will produce the best software for the lowest cost in time and money.
March 27, 2009
Software is "Philosophy Made Manifest"
What does the name "Philosophy Made Manifest" have to do with software development? Here is the answer:
Though I have worked most of my adult life in software development, my original academic background is in Linguistics and East Asian Studies. Since this seems like a bit of a leap, I have often been asked what, if anything, I gleaned from those studies that helped me in my career.
My answer is: a surprisingly large amount. In fact, sometimes I think these things give me a big advantage over computer science or software engineering graduates.
The influence of Linguistics on my thinking about software development probably deserves its own post, but to explain "Philosophy Made Manifest", the most relevant area of my past studies was the philosophy portion of East Asian Studies: Buddhism, Taoism, Confucianism and others.
When you first dip into the surface of these systems of thought, some of the stuff sounds pretty far out there, especially from a modern western point of view, but once you get past the surface, it turns out that these philosophies are the end result of a long line of astute observers and thinkers who were concerned with very real and practical problems: how to live harmoniously with other people, how to set up effective government, how to live a happy and productive life, and many others.
In order to succeed as a student of these philosophies, you have to get good at figuring out the how these people in far away times and places understand the world. And to make their ideas relevant, you have to get good at seeing beyond the foreign trappings of the ancient vocabulary and forgotten social customs, to get at the heart of what they have to say about the unchanging realities of human life.
Once you've mastered this, you can then translate these ideas into more immediately applicable forms, ones that speak to the vocabulary and social customs of the here and now.
Which, perhaps surprisingly, leads us to software development.
Effective software development is not only about the clever use of technology, it is a profoundly human activity that requires an understanding of how people work together and how users, business leaders, developers and others think about the world.
At its core, software development is synthesizing a philosophy of some problem domain, based on these different world-views, and implementing it as a concrete, practical tool that supports and enables the activities of that problem domain.
This is how software development is "philosophy made manifest".
Though I have worked most of my adult life in software development, my original academic background is in Linguistics and East Asian Studies. Since this seems like a bit of a leap, I have often been asked what, if anything, I gleaned from those studies that helped me in my career.
My answer is: a surprisingly large amount. In fact, sometimes I think these things give me a big advantage over computer science or software engineering graduates.
The influence of Linguistics on my thinking about software development probably deserves its own post, but to explain "Philosophy Made Manifest", the most relevant area of my past studies was the philosophy portion of East Asian Studies: Buddhism, Taoism, Confucianism and others.
When you first dip into the surface of these systems of thought, some of the stuff sounds pretty far out there, especially from a modern western point of view, but once you get past the surface, it turns out that these philosophies are the end result of a long line of astute observers and thinkers who were concerned with very real and practical problems: how to live harmoniously with other people, how to set up effective government, how to live a happy and productive life, and many others.
In order to succeed as a student of these philosophies, you have to get good at figuring out the how these people in far away times and places understand the world. And to make their ideas relevant, you have to get good at seeing beyond the foreign trappings of the ancient vocabulary and forgotten social customs, to get at the heart of what they have to say about the unchanging realities of human life.
Once you've mastered this, you can then translate these ideas into more immediately applicable forms, ones that speak to the vocabulary and social customs of the here and now.
Which, perhaps surprisingly, leads us to software development.
Effective software development is not only about the clever use of technology, it is a profoundly human activity that requires an understanding of how people work together and how users, business leaders, developers and others think about the world.
At its core, software development is synthesizing a philosophy of some problem domain, based on these different world-views, and implementing it as a concrete, practical tool that supports and enables the activities of that problem domain.
This is how software development is "philosophy made manifest".
Subscribe to:
Posts (Atom)