Yesterday’s irritated blurt was a matter of feels, not an attempt to offer insight or ideas. I got some queries about it, so let me wipe that slate and be a little less irritated. For now, anyway. 🙂
In geekery — in all tool use beyond a certain point, but my focus here is on the modern software development synthesis — we confront problems that have a size that is "too big to eat all at once". This is hardly news. Human beings have been solving problems that were too big solve in one go since well before recorded history.
Yuval Harari argues convincingly that this is because of our unique ability to form shared imaginary worlds. (Francisco Varela also argued thus.) One solves big problems by decomposition: we split the big problem into several smaller problems, possibly recursively for a few levels, until all the problems inside the big problem seem soluble. Then we solve them.
None of this is remarkable, and my story begins exactly there: the unremarkability of this idea, the unstudiedness of it, can create for us significant mistakes in our action.
An idea like "decompose problems until we have problems that are small enough to eat at one go", we’ll shorthand it to decomposition, comes to us by both nature and training.
And like nearly all ideas, it comes with lots of nearby concepts, relationships, attitudes, and so on. Ideas always come dirty like this. (that statement is in fact the beginning of my critique of positivist/neo-platonist/essentialist worldviews.) a metaphor: decomposition is purest gold. We mine the world of ideas looking for that kind of gold. But we never find it, really. What we find is the ore out of which we might be able to refine gold. When we let the unrefined ore substitute for real gold — taking it in and using it as if it were the pure metal — we can easily create big problems for ourselves.
What are these impurities? They are the less valuable "co-presenting" ideas that surround the good stuff. I could go on at length, and normally do, but let’s see if I can focus on the specific combination of ideas that co-present with decomposition that lead so many folks astray.
(Let me interject: the part that gets rage-tweets isn’t that we make mistakes. Mistakes are necessary and good. It’s that we make the same mistakes over and over because we don’t adequately challenge the ideas that lead to them.)
So, co-presenting with the idea of decomposition — dirtying its application in the world — are a host of ideas that are far less pure. The specific behavior they lead to that troubles me the most in the world of geekery is the "endpointing long-term production". Endpointing is choosing a far-off target and restricting action (& words & analysis) to what we judge that will be relevant when (if) we get there. That’s pretty abstract, let’s bounce one step closer.
"This feature will take a quarter. We will decompose it in to these stories, each of which will take about two weeks. Database changes, service api’s, data collection, event bus, business rules, and UI, and hardening, each of which will take two weeks." to some of you this will seem a straw man: nobody works that way, you’ll suggest. A) you are mistaken, plenty of shops still work that way. B) it is entirely transmutable into a 13-day story instead of a 13-week one, and still has all the same problems.
Co-presenting ideas, then.
- Decomposition is best done in abstract mind-space because humans can manipulate ideas there for free, and there are no scope limitations there.
- All decompositions that work in abstract mind-space work in real life, because mind-space is like real life to an abitrary epsilon at all scales.
- Problem-definitions are stable enough that they will still decompose the same before we solve them as they are after, regardless of how long it takes.
- The only thing that matters is the final recomposition at the endpoint, all other factors are irrelevant.
Notice that the value of those co-presenting ideas is highly sensitive to the duration from here to the endpoint. It’s entirely non-linear. At 13 weeks, not one of them is true. At 13 minutes, all of them are. What about 13 days?
At 13 days, the odds are still slanted very heavily against you. In many 13-day durations, none of them are true. In some, a couple of them are. Very rarely do all four co-presenting ideas hold water.
And yet — in spite of very serious analysis, and considerable data — we routinely structure our actions and our analysis around the idea that it is "efficient" to hold those co-presenting ideas to be true at 13 days.
So. The rage-tweet was here:
IT IS NOT MORE EFFICIENT TO BREAK THE BIG PROBLEM INTO SLIGHTLY SMALLER PROBLEMS WHOSE SOLUTIONS YOU CAN NOT SHIP UNTIL THEY ARE ALL SOLVED.
this is bullshit, and if you take yourself seriously as a geek, you'll stop buying it.
— Michael D. Hill (@GeePawHill) August 2, 2018
I hope it makes more sense now. The situation that triggered it was not a single horizontal decomposition — into sub-problems none of whose solutions were shippable until all of them were — but the policy of using that kind of decomposition and justifying it as efficient.