On Understanding Software Agility – A Social Complexity Point Of View Joseph Pelrine, E:CO 2011
With the background understanding of the Cynefin framework under our belts, today we turn our attention to the paper recommended by Linda Rising in her GOTO Copenhagen conference keynote earlier this month: “On understanding software agility – a social complexity point of view.” The author, Joseph Pelrine, worked as an assistant to Kent Beck in developing eXtreme Programming, and was Europe’s first certified ScrumMaster Practitioner and Trainer.
The paper opens with something of a lament, the original spirit of Agile has often been replaced by cargo-culting of rules and checklists:
Too many teams have come to regard Agile as something like a cookery recipe – follow this set of instructions and procedures for a tasty result. But in software development, as in cooking, what you get out is not simply the sum of what you put in.
Software development is a complex process, and it is conducted by teams of people, which are themselves complex and unpredictable. Discuss!
George Rzevski defined a set of seven criteria for a complex system:
- Interaction – a complex system has a large number of diverse components (agents) involved in rich interaction
- Autonomy – agents are largely autonomous but subject to certain laws, rules or norms; there is no central control but agent behaviour is not random
- Emergence – global behaviour of a complex system “emerges” from the interaction of agents and is therefore unpredictable
- Far from equilibrium – complex systems are “far from equilibrium” since frequent disruptive events do not allow the system to return to equilibrium
- Nonlinearity – nonlinearity occasionally causes an insignificant input to be amplified into an extreme event
- Self-organisation – complex systems are capable of self-organisation in response to disruptive events
- Co-evolution – complex systems irreversibly coevolve with their environments
We have gone from the assumption that everything can be modelled given enough time, intelligence or processing power, to the realization that not everything we experience can be drilled into predictable patterns that we can recognise and understand. The human mind does not readily grasp complexity. It is counterintuitive; we prefer to recognise patterns in mechanistic systems.
There is a temptation to think that building software is merely complicated. But if we accept that it has elements of the complex, then we are in the domain of wicked problems (Rittel & Webber, 1973). Aspects of their definition of wicked problems certainly ring true for me when thinking about software development:
Wicked problems have incomplete, contradictory, and changing requirements; and solutions to them are often difficult to recognize as such, because of complex interdependencies. Rittel and Webber stated that while attempting to solve a wicked problem, the solution of one of its aspects may reveal or create other, even more complex problems.
If software development is indeed a wicked, complex problem, then how should we be going about it? In such a domain, even though it’s tempting to lay out careful plans up front, the chances of them surviving first contact with the problem are remote. The dangers of the temptation are compounded by the fact that we tend to cling to initial assumptions and plotted courses, even when evidence suggests we should do otherwise. As we saw yesterday, in a complex domain, the appropriate method is “probe, sense, respond.”
Over a number of years, the author ran the Cynefin ‘butterfly stamping’ exercise with over 300 people involved in Agile software development. Here’s an example of the results from those workshops, looking at how the participants made sense of various software development related activities:
The highest percentage of tasks and activities were in the complex domain:
Although this is not sufficient to argue that software development as a whole is complex, it does suggest that many parts of it are amenable to analysis and treatment using complexity-based tools and techniques.
In a complex domain, one of the hardest things for us to let go of is our desire for linear causality. But in the complex domain there is no meaningful causality, and any causality is only retrospectively coherent. That is, it makes sense looking backwards after the fact, but that doesn’t mean it is predictable or repeatable.
Success in software development is only retrospectively coherent.
A good example of retrospective coherence is task estimation. Once you’ve completed a task, you now exactly how long it took and why. But before you do any (complex) task, it’s almost impossible to estimate.
The same goes for projects. As a project goes on, the reasons for its success become established, not before. After it’s over, you can say that the project was a success, and that certain things took place during the project – but you cannot say that the project was a success because certain things took place!
(A whole genre of business books seem to fall into this trap too).
Human nature and our desire to find predictive patterns makes us want to repeat the things that took place as if they were guarantees of future success, but this doesn’t hold in the complex domain.
Contrary to Einstein’s definition, in a socially complex system, insanity is doing the same thing over and over again and expecting the same result!
So what should we do? The Cynefin framework suggests to us that in a complex domain, we should use a probe-sense-respond technique:
- Set boundaries for the system to emerge
- Employ numerous probes providing feedback on what works and what doesn’t
- Apply sense-making analysis to the feedback results
- Continue or intensify the things that work, correct or change those that don’t.
Tighten this into a small iterative loop, observing emergent patterns, amplifying good ones, and disrupting bad ones, until you end up successful at your endeavour. In fact, it’s often the case that by applying this process, you discover value at points along the way. Your final product can end up looking very different to the original plan, and being very much better. You could not have defined these benefits at the outset and aimed for them; these are an example of emergent properties of the complex system.
And this, concludes Pelrine, is what the true spirit of Agile is all about: Agile’s ‘apply-inspect-adapt’ model is a probe-sense-respond model. Social complexity methods such as Cynefin provide the theoretical underpinnings that we need to understand in order for such methods to become truly scalable and sustainable.