When large teams struggle with trunk-based development (TBD) or continuous integration/deployment (CI/CD), a good strategy is to re-orient how the teams face "backsteps", moments in our workflow where a discovery forces us to re-open a stage we thought was closed.
It’s been a hard stretch for those of us seeking peace and equity. I offer this geekery not to suggest it matters as much as that does, but only to give us a little respite.
Stay safe, stay strong, stay angry, stay kind. Black Lives Matter.
Large teams & projects often pre-date the use of modern synthesis techniques like TBD or CD, and during adoption, intermixing their Before with this new After can produce quite a bit of discomfort. If it gets bad enough, it can even kill off the improvement effort. What to do?
It’s common to think of a chunk of programming work as flowing through a directed graph — normally but not necessarily a straight line — of steps or states moving left to right. At the left end, we have "stuff we haven’t started". At the right, we have "stuff we shipped".
A backstep happens when a story goes backwards in this flow, from right to left.
Some will immediately jump to the words "bug" or "defect", but I am choosing "backstep" for a reason: such orgs usually already use "bug", but only when the backstep happens in the rightmost state.
Further, in process-heavy orgs, "bug" and "defect" are socially fraught terms, and accordingly, they’re usually swathed in great sticky wads of pilpul: hair-splitting logic, difference without distinction, the elaborate picking of nits, impressive performant rationalization.
Process-heavy teams often have many states in their workflow. Some of the transitions between them are more by way of progress markers, and others are handoffs, where the work product actually changes hands. Orgs spend a lot of time debating, drawing, editing these flows.
So the point of saying "backstep" instead of "bug" or "defect" is to sidestep prior practice: we don’t care which state we left or which state we returned to. If we go backwards from any state, that’s what we call a backstep.
A brief tangent: some of you may be all SMDH over this.
"You don’t need states," or "Ensembles eliminate handoffs," or any of a number of other possible objections based around the fact that these orgs aren’t "agiling right".
Remember my recent critique of "Agile method"? A missing — at best insufficiently described — ingredient from just about every branded method is the pathing, how we get from wherever we are now to the bright shining Brand X City on the Hill.
Well, gotcha.
Old joke:
A city-slicker pulls up to the country store, asks the two old-timers out on the porch, "Say, fellas, what’s the best way to get to New York city?"
They look at each other for a second, then one of ’em says:
"Well, I wouldn’t start from here."
Yes, we’re talking about orgs that aren’t doing it right, not yet. They have one foot on their old way and one brave toe pointing towards a new way. They can’t do it right because they’re still becoming. In fact, the most agile of teams are permanently becoming.
They aren’t agiling right, and if we’re going to help them, we’re going to have to find ways that they can do it wrong that are more likely to open up into ways of doing it right.
So what’s this mean, "re-orient how teams face backsteps"?
Often, teams don’t do anything at all with a backstep whose origin isn’t the "Shipped" state. When they get one of those, there are a variety of responses, but if we set aside the parts of those responses that are the blame game, they amount to "fix this as soon as possible."
The re-orientation, then, is twofold. 1) Every backstep deserves attention. 2) The focus of that attention is "fix this as early as possible and as wide as possible and as forever as possible".
Those three can be taken in order, in fact, as each is a deeper and more positive development of the last. Here’s what they mean.
As early as possible:
The backstep happened at stage X, towards the righthand side, because we either accidentally or on purpose detected it at that point. Could we have detected that backstep at an earlier stage, maybe at stage W, or at L, or even at A?
As wide as possible:
The backstep is a single failure, sometimes even a single line. We can fix that one line, and fix that one backstep. But backsteps often run in families. Could we change things so that the whole family can be caught & killed instead of just one member?
As forever as possible:
The backstep is caused by some bad code. That bad code happened because it was possible to happen. Could we change our code and design so that that particular bad code is impossible to even write in the first place?
An example might help right about now. Say our backstep was caused by a service aiming its URLs at the wrong instances of its upstream services. This isn’t at all uncommon, and sometimes even happens in production.
(Ouch.)
Upon investigation, we have one of those ubiquitous configuration files y’all seem so enamored of for no reason I can detect, and the offending service’s entry for its upstreams had the wrong text in it.
What’s the old way, "as soon as possible"? Change the text in the file and re-deploy the image. Easy-peasy, the old way produces an answer. But its a shallow answer, and when we stop there, we’re almost certain to see the same or similar problem elsewhere.
What’s one "as early as possible" fix? We could easily roll a test that checks that property against a set of environment URL’s and hollers at us if it isn’t one of them. We could do that before we even commit any change in that service’s code.
What’s one "as wide as possible" fix? If we can do that for one URL, we can do it for all of them, every time. We can regularize, normalize, and generalize this as an automated step in our build. Good thing we got us some programmers layin’ around.
What’s an "as forever as possible" fix? Write a tested shared library that resolves a single keyword to get the URLS for the corresponding environment, and use it in every client. Write your Todo client to use Upstream objects, not strings at all.
Note: There are other fine answers for those last two questions, too. I just picked some easy-to-describe ones. The point of my discussions is rarely that there is only one right answer.
There are many good answers to a problem like this, but here’s the thing: you won’t find any of them if you don’t look for them.
And that is the point of re-orienting how your team faces backsteps.
So how do you go about doing this re-orientation?
The truth is there are so many different org/workflow/tech-stack/cultural variants out there, there just isn’t one answer.
Some teams story-ize backstep work: you’re not allowed to take a new value story if there’s a backstep story in the queue.
Other teams just do it, regardless of Jira and stories and boards and managers.
Some orgs designate a whole team whose job is just this.
Each of those ideas, and there are other good ones, too, has pros & cons depending on how they fit into your culture as it stands today. Whichever you use, rest assured it will change that culture, and there’ll be other ideas right behind it.
Organizational politics is tricky , of course, but the key understanding you need to get across the gap: when we have a lot of backsteps, it looks like we’re going faster because of the sound and the fury, but we’re actually going slower.
When the backstep is across a handoff transition? We’re not going a little slower, we’re going a lot slower. If your org wants more value faster, it wants to kill off every backstep it encounters. Not just as soon as possible, but as early as, as wide as, and as forever as.
If your large team is struggling on their way to trunk-based development or continuous deployment, consider how we should re-orient ourselves towards backsteps in the flow.
Openly responding to backsteps is a key element of agility.
Get Involved!
If you love the GeePaw Podcast, consider a monthly donation to help keep the content flowing. You can also subscribe to get weekly posts sent straight to your inbox. And to get more involved in the conversation, jump into the Camerata and start talking to other like-minded Change-Harvesters today.