In my previous post, I talked about lessons from Confucius that I think are still useful in the modern world, but there is a field of endeavour that I think can particularly benefit from understanding the Confucian worldview: software development.
At first glance, it seems highly unlikely that Confucianism might have some application to software development. The Confucian values of respect for the past, harmonious social relations, decorum and moral leadership don’t have an obvious affinity for an industry that is known for its focus on what is new and shiny, and which is stereotypically populated by raging individualists with a disregard for social standards.
But it is worth considering that the Confucians represent one of the earliest groups of knowledge workers in human history. They were trained in specialized knowledge for specialized tasks in a complex society, and they had a sense that their specialized knowledge made them an elite group in society.
An important concept in Confucian texts that bears on this is junzi. Etymologically, it means “ruler’s son, prince”, but already in Confucius’ time it is used more metaphorically, often translated into English as “superior man”, “gentleman”. The Yiddish term “mensch” has a similar ring. It represents what every Confucian was striving to be.
In spite of its etymology, junzi is one of the earliest-known notions of elite status that is not derived from the happenstance of one’s birth, such as being the literal son of a ruler, or a free man of Athens. The kind of “superior man” we are talking about here is defined entirely by his knowledge and skills, and his savoir faire in using them. Confucianism is a truly meritocratic system of thought, and a modern IT specialist can easily relate to such an ethic.
A second consideration is the fundamentally social nature of software development. If it ever truly existed, the age of the lone genius changing the world with his software is over. Any non-trivial software development these days involves a whole team of people with different specialties and knowledge, and building reliable and maintainable applications requires that these people work together as an effective and harmonious community.
The Confucian junzi has a sense of noblesse oblige, a sense that the status conferred on him by his knowledge and skills requires that he use them for the betterment of his society and to achieve collective goals. He is willing to lead and mentor new members of the fraternity of knowledge workers, and does this not by pontificating, but by providing an example through good practices.
The very fact that these kind of values are not what most of us think of when we consider software development suggests that there is still much benefit and advantage to be gained by nurturing them in software teams, and a team lead could do worse than to study the Analects of Confucius to prepare for the challenges they face.
After all, Confucius and his followers have several centuries of experience organizing and training knowledge workers to draw on.
February 16, 2010
January 15, 2010
Lessons from Confucius
When I was in university studying East Asian studies, it was the height of the political correctness era. Of the three major streams of Chinese “religious” thought — Confucianism, Taoism and Buddhism — Confucianism was considered to be the “bad guy”, representing everything that is authoritarian, sexist and hierarchical in Chinese culture.
By contrast, everyone loved the individualistic enthusiasm of Taoism, which extolled the virtues of the feminine, or the egalitarian austerity of Buddhism. Confucians, though, were the “dead white men” of Chinese studies.
So, while I got an early grounding in Taoist and Buddhist thought, it wasn’t until I had been in the work-world for some years that I came back to Confucius, starting by reading the Analects in translation, and later, in the original.
The Confucius I found was quite different from my pre-conceived understanding of him, and I was surprised to find that his outlook and advice were unexpectedly relevant to modern life. I also found an attitude quite different from the stern authoritarian stereotype I had previously accepted.
In fact, my surprise started with the first line of the Analects (translations are my own):
I had, of course, learned as a student about the traditional Chinese respect for learning, but I had been left with the impression that the kind of learning that was meant was rote memorization in strict conformance to orthodox interpretation. But this primary source of Confucius’ personal thought starts off with a child-like enthusiasm for the simple pleasure of learning, a feeling I knew well. This attitude struck me as even more relevant for our modern world, where there is so much to learn and where knowledge and skills are the currency of our society.
Another traditional value of the Confucians I had learned about in my student days was often translated as “ritual”, but it seems to encompass politeness, observance of correct social forms and social hierarchy, as well as what we would think of as actual rituals. We used to roll our eyes at this, since we tended to think that these things are intended to suppress our sincere, individual feelings – that they are empty formalities intended to ensure conformity. But in another line from the Analects, I got another surprise:
Confucius is genuinely concerned with the observance of social forms (and the social forms of his time and place sometimes seem very foreign to us), but he doesn’t recommend them as a replacement for individual feeling, but rather as a vehicle to allow the free expression of individual feeling in harmony with the functioning of the community.
This sense of communal harmony is underlined by the cardinal virtue of Confucian thought, often translated as “benevolence”. It is related to the Chinese word for “person”, and I can’t help but feel that the best modern equivalent for it is the Yiddish word “mensch”. Etymologically, “mensch” means “person”, but its full meaning is someone who has a strong sense of community, someone who can be relied upon to help others and who has the courage to stand up for what is right. This is a pretty close match with the Confucian principle.
Confucius is also very concerned with good leadership, and a recurring preoccupation of Confucians (and ancient Chinese thinkers in general) is how to be an effective ruler. Here is another representative passage from the Analects:
Contrary to my early stereotype, Confucius has no time for the “because I said so” school of leadership. This is all the more remarkable when you consider that rulers in ancient China did have absolute power. His prescription for leadership is firmly in the “lead by example” camp. Those millennia ago, Confucius had already recognized that people can tell the difference between a cynical, self-serving leader and one who genuinely has the collective good at heart. He knew that the difference between genuine commitment and mere grudging compliance rests on this distinction.
So, my re-examination of Confucian thought not only changed my mind about its essence but actually showed me that there were values and lessons to be learned that I could apply to my life, personally and professionally.
A genuine love of learning, accepting social forms as ways of collectively expressing individual feeling, cultivating sincere feelings of communality with others, and leading by moral example: all of these ideas are still relevant in the 21st century and, if practiced, can make a real difference in one’s life as a leader and as a human being.
By contrast, everyone loved the individualistic enthusiasm of Taoism, which extolled the virtues of the feminine, or the egalitarian austerity of Buddhism. Confucians, though, were the “dead white men” of Chinese studies.
So, while I got an early grounding in Taoist and Buddhist thought, it wasn’t until I had been in the work-world for some years that I came back to Confucius, starting by reading the Analects in translation, and later, in the original.
The Confucius I found was quite different from my pre-conceived understanding of him, and I was surprised to find that his outlook and advice were unexpectedly relevant to modern life. I also found an attitude quite different from the stern authoritarian stereotype I had previously accepted.
In fact, my surprise started with the first line of the Analects (translations are my own):
To learn something, and to review it now and again, isn’t it pleasurable?
I had, of course, learned as a student about the traditional Chinese respect for learning, but I had been left with the impression that the kind of learning that was meant was rote memorization in strict conformance to orthodox interpretation. But this primary source of Confucius’ personal thought starts off with a child-like enthusiasm for the simple pleasure of learning, a feeling I knew well. This attitude struck me as even more relevant for our modern world, where there is so much to learn and where knowledge and skills are the currency of our society.
Another traditional value of the Confucians I had learned about in my student days was often translated as “ritual”, but it seems to encompass politeness, observance of correct social forms and social hierarchy, as well as what we would think of as actual rituals. We used to roll our eyes at this, since we tended to think that these things are intended to suppress our sincere, individual feelings – that they are empty formalities intended to ensure conformity. But in another line from the Analects, I got another surprise:
Lin Fang asked about the basics of ritual. Confucius said: A big question! In ritual, prefer modesty to extravagance; in mourning, prefer sincere sadness to formality.
Confucius is genuinely concerned with the observance of social forms (and the social forms of his time and place sometimes seem very foreign to us), but he doesn’t recommend them as a replacement for individual feeling, but rather as a vehicle to allow the free expression of individual feeling in harmony with the functioning of the community.
This sense of communal harmony is underlined by the cardinal virtue of Confucian thought, often translated as “benevolence”. It is related to the Chinese word for “person”, and I can’t help but feel that the best modern equivalent for it is the Yiddish word “mensch”. Etymologically, “mensch” means “person”, but its full meaning is someone who has a strong sense of community, someone who can be relied upon to help others and who has the courage to stand up for what is right. This is a pretty close match with the Confucian principle.
Confucius is also very concerned with good leadership, and a recurring preoccupation of Confucians (and ancient Chinese thinkers in general) is how to be an effective ruler. Here is another representative passage from the Analects:
Ji Kang asked: How can the people be made to respect the ruler, to be loyal and take his advice? Confucius said: Ruling them with solemnity will result in respect; showing respect for elders and being kind will result in loyalty; promoting good and instructing the unskilled will result in persuasion.
Contrary to my early stereotype, Confucius has no time for the “because I said so” school of leadership. This is all the more remarkable when you consider that rulers in ancient China did have absolute power. His prescription for leadership is firmly in the “lead by example” camp. Those millennia ago, Confucius had already recognized that people can tell the difference between a cynical, self-serving leader and one who genuinely has the collective good at heart. He knew that the difference between genuine commitment and mere grudging compliance rests on this distinction.
So, my re-examination of Confucian thought not only changed my mind about its essence but actually showed me that there were values and lessons to be learned that I could apply to my life, personally and professionally.
A genuine love of learning, accepting social forms as ways of collectively expressing individual feeling, cultivating sincere feelings of communality with others, and leading by moral example: all of these ideas are still relevant in the 21st century and, if practiced, can make a real difference in one’s life as a leader and as a human being.
September 30, 2009
Creativity is a Balance of Freedom and Control
About a year ago, I took up playing a middle-eastern instrument called the oud.
One of the characteristic forms of music for the oud is called a taqsim , which is a short improvisation. I’ve always loved improvisational musical styles, and this was part of the draw of the oud for me.
Improvisational music can be a very pure form of spontaneous creativity, and when it is going well, it gives a powerful feeling of freedom and self-expression.
Many people who don’t play an instrument or who play an instrument only by reading sheet music often ask me how one can possibly make up music on the fly, and I always tell them it is easier than it sounds.
A common misconception is that to be creative means that you throw all rules and restraints out the window, pulling great stuff out of thin air, as if by magic. Many people think that creativity is somehow inherently chaotic and random. But a simple example can show that this is false.
Have you ever listened to a small child with no musical training hammer away at a piano? Based on the “creativity is chaos” misconception, this should be the height of creative expression, since the child is following no structures or rules aside from the physical limits of the piano. But most people would agree that the result is not music, but rather irritating noise.
The first thing that many piano methods do with our budding creative genius is to teach the scales. This is because these are the basic structures that underpin western music. To be either a competent listener or a competent player of western-style music begins by internalizing the sound landscape that is defined by the diatonic (or pentatonic) scales and the melodic and harmonic palette they imply.
After learning even one of the scales, our enriched child could hammer away the same as before, but restrict the hammering to the notes of that scale. Because there is some musical structure, most people would judge the result as more musical, and in that sense morecreative.
This is the trick with improvisation: you aren’t really making up something from scratch. All common forms of musical improvisation I’ve ever studied (Indian classical raga , the blues , Middle Eastern maqamat and Jazz , for example) actually have a lot of structure to begin with: there are specific scales, rhythms, sequences and patterns that have to be learned and internalized or else the result won’t sound like music, or at least not the kind of music you are trying to play.
Once the player (or the listener) has internalized all the rules and structure, then freedom from rules comes back into play. If you have too many rules followed too rigidly, the results become boring and predictable and the music loses its vitality. Great improvisers know where their freedom lies and know where and how to break the rules to produce something fresh and unexpected, and it is this quality that can make improvised music so vital and exciting for players and listeners. So-so improvisers sometimes fall flat on their face by going too far away from the basic structure and have to scurry back to it. But even this freedom to fail is part of the process.
What is true for music seems to be true for other domains with complex structure that require creative solutions: software development, management, politics and economics, for example.
In any of these domains, too few rules results in inharmonious chaos, and too many stifles inventive and effective approaches.
Any approach to these problem areas that looks only at adding regulation or only at removing regulation is probably missing the boat.
The real question always is: “How do you plan to balance freedom and control to maximize harmonious creativity?” Any philosophy of these domains that can’t answer this question is probably not worth considering.
One of the characteristic forms of music for the oud is called a taqsim , which is a short improvisation. I’ve always loved improvisational musical styles, and this was part of the draw of the oud for me.
Improvisational music can be a very pure form of spontaneous creativity, and when it is going well, it gives a powerful feeling of freedom and self-expression.
Many people who don’t play an instrument or who play an instrument only by reading sheet music often ask me how one can possibly make up music on the fly, and I always tell them it is easier than it sounds.
A common misconception is that to be creative means that you throw all rules and restraints out the window, pulling great stuff out of thin air, as if by magic. Many people think that creativity is somehow inherently chaotic and random. But a simple example can show that this is false.
Have you ever listened to a small child with no musical training hammer away at a piano? Based on the “creativity is chaos” misconception, this should be the height of creative expression, since the child is following no structures or rules aside from the physical limits of the piano. But most people would agree that the result is not music, but rather irritating noise.
The first thing that many piano methods do with our budding creative genius is to teach the scales. This is because these are the basic structures that underpin western music. To be either a competent listener or a competent player of western-style music begins by internalizing the sound landscape that is defined by the diatonic (or pentatonic) scales and the melodic and harmonic palette they imply.
After learning even one of the scales, our enriched child could hammer away the same as before, but restrict the hammering to the notes of that scale. Because there is some musical structure, most people would judge the result as more musical, and in that sense morecreative.
This is the trick with improvisation: you aren’t really making up something from scratch. All common forms of musical improvisation I’ve ever studied (Indian classical raga , the blues , Middle Eastern maqamat and Jazz , for example) actually have a lot of structure to begin with: there are specific scales, rhythms, sequences and patterns that have to be learned and internalized or else the result won’t sound like music, or at least not the kind of music you are trying to play.
Once the player (or the listener) has internalized all the rules and structure, then freedom from rules comes back into play. If you have too many rules followed too rigidly, the results become boring and predictable and the music loses its vitality. Great improvisers know where their freedom lies and know where and how to break the rules to produce something fresh and unexpected, and it is this quality that can make improvised music so vital and exciting for players and listeners. So-so improvisers sometimes fall flat on their face by going too far away from the basic structure and have to scurry back to it. But even this freedom to fail is part of the process.
What is true for music seems to be true for other domains with complex structure that require creative solutions: software development, management, politics and economics, for example.
In any of these domains, too few rules results in inharmonious chaos, and too many stifles inventive and effective approaches.
Any approach to these problem areas that looks only at adding regulation or only at removing regulation is probably missing the boat.
The real question always is: “How do you plan to balance freedom and control to maximize harmonious creativity?” Any philosophy of these domains that can’t answer this question is probably not worth considering.
July 7, 2009
Why proving that P ≠ NP is hard
A while ago I spent the time to “audit” online a Cambridge University course given by Tim Gowers on computational complexity.
In this course, Gowers takes the students through a proof by Razborov that there is no “natural proof”, for a particular form of the problem, that P ≠ NP, which I found very interesting.
This got me thinking about why this is a hard problem to solve, and I thought it might be useful to explain the intuitive reasons behind it.
So let’s think about this by looking at the Traveling Salesman Problem, since it is a fairly intuitive example of one of the hardest problems in the NP complexity class, known as NP-complete.
In this problem, you have to find the shortest path through a number of cities, visiting each city once and only once. This is a typical NP problem, where the naïve approach to solving the problem would be to just work through all the combinations possible, check the value of that combination (in this case the length), and pick the shortest one.
The problem with that is that the number of combinations gets very large very fast, so it really isn’t practical to do this.
Let me show you what I mean. Let’s take the case where there are 5 cities: A, B, C, D, E. Any path through these cities can be described by a sequence of these letters. For example, CDEAB would be the sequence starting at city C, then going on to city D, etc. To figure out how many combinations there are, I have to use each of the letters once and only once, so there are 5 possible choices for the first slot, four for the second, and so on. This gives me 5x4x3x2x1=5!=120.
In general, this brute force approach to solving the problem, which represents the worst case scenario among solutions, will take n! steps, where n is the number of cities we are considering. This gets very big, very fast. For example 10! is equal to 3628800, so you can see that as you add cities it becomes vastly harder to solve the problem this way.
Now it is very important to realize that this is the worst-case scenario only, or as it is described in mathematics, an “upper bound” on how hard the problem is. For example, if all the cities are in a straight line, you can solve the problem in many fewer steps: just start on one end and move to the next in line and so on
This works because we know that on a straight line, the shortest path will be on that line, so that if the cities are aligned ABCDE, we know right away that we aren’t going to have to consider any of the combinations that don’t follow the sequence on the line. In other words, there is a pattern to the data that allows us to ignore a number of combinations (in this case, all but the two, which are of equal length: ABCDE, and EDCBA).
But to say that some problem is in the class NP (and not in P) is essentially to say that there will always be some collection of cities you could choose that would require you to go through the worst case scenario to find the shortest one. (Technically, the actual worst case scenario for this problem is only exponential in complexity, but that is still pretty bad and doesn’t affect my argument here.)
That is, unless someone can prove that P=NP. This would mean that there was always a shortcut that allowed you to find the solution without having to work through the majority of the possible combinations. And this shortcut would almost certainly involve being able to consistently find some pattern in the data, just as we did with the straight line example, that simplifies the data down to a smaller number of cases.
So in a sense, it should be much easier to prove P=NP than that P ≠ NP. (That no one has proven the former yet is a pretty good reason to suspect that the latter is true, even though no one has proved it either.)
The reason is that to prove P=NP, you just need to show that there is always a pattern in the data that will simplify it, whereas to prove P ≠ NP, you would have to show that there can’t be such a pattern for some scenarios in the problem set.
Why is it easier to show that there is a pattern than that there can’t be a pattern? Well, if you have a pattern, you have a relatively small number of steps needed to describe that pattern, and it is much easer to work through those steps and convince yourself that you have covered all the bases. You can also try your pattern on all the hairiest known instances of the problem and show that they work.
In a sense, having a pattern simplifies the search space required for the proof in exactly the same way that having the pattern simplifies the search space to find the answer to the problem.
But how do you show that there is no possible pattern for some set of data?
Let’s say that I give you a list of digits and tell you that one of two scenarios generated these digits: either I am generating them randomly for each digit as you ask for them, or I am using some relatively simple-to-genarate pattern to generate them. Here is a sample list:
15926535897932384626433832795…
Can you see a pattern? You could do a statistical test on the digits to see if they show the properties of randomness, and in this case, I think you would find that they do show those properties.
So, can you then conclude that they are random?
No, unfortunately not, since there is a pattern here: these are the digits of pi after the well-known first three 3.14…
So the challenge is that you can never really be sure that there is no pattern to some data set; it could just be a really, really unobvious pattern that you haven’t thought of yet.
So you can see how very hard it would be to prove that there is always some configuration of cities that can’t be simplified by some pattern, which is only part of what you would have to do to prove that P ≠ NP.
This is why it is unlikely that anyone will find such a proof unless they can develop some hard-to-imagine technique that allows us to detect when there aren’t any patterns in a set of data.
In this course, Gowers takes the students through a proof by Razborov that there is no “natural proof”, for a particular form of the problem, that P ≠ NP, which I found very interesting.
This got me thinking about why this is a hard problem to solve, and I thought it might be useful to explain the intuitive reasons behind it.
So let’s think about this by looking at the Traveling Salesman Problem, since it is a fairly intuitive example of one of the hardest problems in the NP complexity class, known as NP-complete.
In this problem, you have to find the shortest path through a number of cities, visiting each city once and only once. This is a typical NP problem, where the naïve approach to solving the problem would be to just work through all the combinations possible, check the value of that combination (in this case the length), and pick the shortest one.
The problem with that is that the number of combinations gets very large very fast, so it really isn’t practical to do this.
Let me show you what I mean. Let’s take the case where there are 5 cities: A, B, C, D, E. Any path through these cities can be described by a sequence of these letters. For example, CDEAB would be the sequence starting at city C, then going on to city D, etc. To figure out how many combinations there are, I have to use each of the letters once and only once, so there are 5 possible choices for the first slot, four for the second, and so on. This gives me 5x4x3x2x1=5!=120.
In general, this brute force approach to solving the problem, which represents the worst case scenario among solutions, will take n! steps, where n is the number of cities we are considering. This gets very big, very fast. For example 10! is equal to 3628800, so you can see that as you add cities it becomes vastly harder to solve the problem this way.
Now it is very important to realize that this is the worst-case scenario only, or as it is described in mathematics, an “upper bound” on how hard the problem is. For example, if all the cities are in a straight line, you can solve the problem in many fewer steps: just start on one end and move to the next in line and so on
This works because we know that on a straight line, the shortest path will be on that line, so that if the cities are aligned ABCDE, we know right away that we aren’t going to have to consider any of the combinations that don’t follow the sequence on the line. In other words, there is a pattern to the data that allows us to ignore a number of combinations (in this case, all but the two, which are of equal length: ABCDE, and EDCBA).
But to say that some problem is in the class NP (and not in P) is essentially to say that there will always be some collection of cities you could choose that would require you to go through the worst case scenario to find the shortest one. (Technically, the actual worst case scenario for this problem is only exponential in complexity, but that is still pretty bad and doesn’t affect my argument here.)
That is, unless someone can prove that P=NP. This would mean that there was always a shortcut that allowed you to find the solution without having to work through the majority of the possible combinations. And this shortcut would almost certainly involve being able to consistently find some pattern in the data, just as we did with the straight line example, that simplifies the data down to a smaller number of cases.
So in a sense, it should be much easier to prove P=NP than that P ≠ NP. (That no one has proven the former yet is a pretty good reason to suspect that the latter is true, even though no one has proved it either.)
The reason is that to prove P=NP, you just need to show that there is always a pattern in the data that will simplify it, whereas to prove P ≠ NP, you would have to show that there can’t be such a pattern for some scenarios in the problem set.
Why is it easier to show that there is a pattern than that there can’t be a pattern? Well, if you have a pattern, you have a relatively small number of steps needed to describe that pattern, and it is much easer to work through those steps and convince yourself that you have covered all the bases. You can also try your pattern on all the hairiest known instances of the problem and show that they work.
In a sense, having a pattern simplifies the search space required for the proof in exactly the same way that having the pattern simplifies the search space to find the answer to the problem.
But how do you show that there is no possible pattern for some set of data?
Let’s say that I give you a list of digits and tell you that one of two scenarios generated these digits: either I am generating them randomly for each digit as you ask for them, or I am using some relatively simple-to-genarate pattern to generate them. Here is a sample list:
15926535897932384626433832795…
Can you see a pattern? You could do a statistical test on the digits to see if they show the properties of randomness, and in this case, I think you would find that they do show those properties.
So, can you then conclude that they are random?
No, unfortunately not, since there is a pattern here: these are the digits of pi after the well-known first three 3.14…
So the challenge is that you can never really be sure that there is no pattern to some data set; it could just be a really, really unobvious pattern that you haven’t thought of yet.
So you can see how very hard it would be to prove that there is always some configuration of cities that can’t be simplified by some pattern, which is only part of what you would have to do to prove that P ≠ NP.
This is why it is unlikely that anyone will find such a proof unless they can develop some hard-to-imagine technique that allows us to detect when there aren’t any patterns in a set of data.
Subscribe to:
Posts (Atom)