Understanding Incremental Switchover

---

The incremental switchover approach is my default response to any transformation problem I can’t resolve in an hour.

It’s the secret to successful brownfield development, but it’s not widely understood & used. Let’s take some time to understand it.

As is my wont, I will again say that geekery isn’t really the most important story right now. I, and I hope you, understand this.

Stay safe. Stay strong. Stay kind. Stay angry. Work on equity, when you can, how you can. Black lives matter.

Incremental switchover might be called "the art of the path". At its simplest, given two points A, where we are, and Z, where we wish we were, incremental switchover concerns itself with B, C, D, and so on, the route we take.

We talked about RAT a couple of days ago: Rework Avoidance Theory. RAT is also concerned with path, but it applies its logic to a series of simple premises that are not reliable at scale. Because its premises are so often invalid, so are its conclusions.

Read full article here – RAT: Rework Avoidance Theory

If you’re a geek, I’m sure you’ve experienced the joy of pure greenfield work. The sensation of a blank page, a new start, a chance to put what you’ve learned into play without having to undo the mistakes you’ve learned from.

The loveliness of greenfield are the source of the nearly ubiquitous rewrite projects: the old one, fielded, is so wrong that it’s impossible to change effectively. We tend to blame this on a variety of aspects about the old one. Architecture, tech stack, feature-slamming.

So we will stop trying to change it, and instead write a new one. The day the new one ships, it’ll do everything the old one does, only better, and be more changeable, too!

(My mom will help us sew the costumes!)

The reason this hardly ever works: the straightforward fact that we’re changing everything about the old one except the method we used to make it. That method we used is why Cthulhu found it so boringly easy to control every fiber of its being. And why it will happen again.

In particular, we will repeatedly use RAT to justify large steps — batch-size-gigundous — instead of Incremental Switchover — batch-size-one-hour. The Elder Dark has no face to smile, but predators always prefer fresh meat.

Incremental Switchover isn’t one pattern or technique, it’s more meta- than that, it’s truly a style, a way of seeing, an approach to development. It demands cleverness, practice, and patience. The hardest part: it will appear to make you slower while actually making you faster.

The central goal of Incremental Switchover (going forward let’s call it IS):

"Change live code (at least) every hour without ever making the system worse."

Two aspects of this need calling out, "live code" and "not worse". Both of these ideas have a pure hard core and a gradient boundary, and both of them require the use of judgment.

Live Code

1. "Live code" pure: Code that is actually in the call-graph, the dependency tree, the runtime execution, of the deployed application. Code that is actually being run when its feature is triggered.
2. "Live code" gradient close: Mirror code, where the new is running alongside the old from some point down to the metal and back out. Field code, where a single element is coming from the new stuff but all the others are coming from the old.
3. "Live code" gradient far: QA code, where new and old are both showing results simlutneously if some global QA attribute is set. Versioned code, (endpoint-specific), where the new is only being run if the request versions or endpoint are different.

Not Worse

1. "Not worse" pure: the new code doesn’t make anything in the shipping app better, but it doesn’t make it any worse, either, through every means we can use to tell.
2. "Not worse" gradient near: The new code makes the app a little slower, or hits two databases where it used to hit one. The new code makes the source briefly worse, because it creates two segments of the codebase that do "the same thing".
3. "Not worse" gradient far: The new code makes a less-commonly used corner of the user’s experience less desirable, but is still possible. The new code creates multiple paths to achieving "the same thing", the old path and the new path.

I stress this: though there are situations where we can maintain the pure form of these two ideas, there are real situations where we have to make judgment calls and overlook known issues and forge ahead. All geekery requires human judgment, a horrible hole in our pedagogy.

The point isn’t to never break IS purity, the point is to use the absract ideal to suggest & guide concrete action. Here’s the thing, if you active search for IS openings, you will find them. But you gotta do the search.

Let’s look at a couple of examples.

Down in nitty-gritty code, you have an API that returns a list when you realize you need a more complex data structure. If you change that API’s return, you’ll have to change a hundred call sites. Batch-size-gigundous.

Add a new API that returns your complex data structure. Give that complex data structure the ability to (more expensively) compute the same list. TEST THAT IT IS GUARANTEED TO. Gradually switch call sites to use the new API. When no one calls the old API, kill it.

Now, inch by inch, take those call-sites and undo the part where they ask for the old list at all. Make each one of them learn how to use the new structure, one at a time. When no one wants the slowly-computed list call anymore, kill it.

You will normally be able to do this in a series of "dead safe" steps, minutes at a time, pausing at any point, pushing after every incremental switch, going all the way through your CI/CD pipeline.

Up at the top of your enterprise web app: You have a complex form with dozens of fields that you’re wanting to turn in to a flow of stepped pages, one-to-three fields at a time.

Your RAT approach: write a brand new set of pages, put every field from the original on the new, make sure it works in exactly the same way. Wait six months. Wait two more months. Wait three weeks. Ship it. Work 20 hour shifts on hotfixes.

Your IS approach: group the fields on the original form the way you want them stepped. Replace one group at a time with a frame. Make your new page embed that frame, along with the next/prev and instructional texts. Rinse, lather, repeat.

Web pages don’t care what endpoints their data comes from, and they don’t care if there are multiple endpoints on one page. If your framework won’t let you mix old-style with framework style, choose another framework.

IS is fundamental to the planning and story-writing process, of course. The most standard thing in the world is to use RAT to create huge stories because if we don’t do it that way, we’ll have to change code we’ve already written. A classic: the login.

We specify a login, with roles, a password, an email registration confirmation, an email password recovery, and a beautifully drawn page that shows it all off. The result: a four-week story.

What if we gave them a login with password today? What if we implemented password-scoring tomorrow? What if we remember-me’d after that? What if we implemented automated password reset next? And on, and on. Stories are a natural place for the IS approach, if you will just see it.

So, two things: First, notice how each case takes more and smaller steps than their RAT variation. This will be seen as a negative, but it’s not: counting steps as a measure of path-value does not work because it abstracts away the considerable inherent benefit of steps.

Steps give us testability, steerability, confidence, interruptability, energy, and user-detectable progress. When we abstract away those benefits, "more steps" = "bad". That’s RAT working on you. Put those benefits back into your calculus, and you’ll see how value IS really is.

Second: none of these cases is rocket-science. It’s not harder to do Incremental Switchover, it’s usually actually easier. It’s not even harder to see it. The missing ingredient is looking for it, thinking about it, considering it, in a RAT-dominated world.

I don’t "always" manage Incremental Switchover in my development pathing. Sometimes I can’t figure a way to do it. But I do start every transformation that takes longer than an hour by asking myself if there’s an IS way to go. Very often, there is, and very often, it’s better.

---

Supporting The PawCast

If you love the GeePaw Podcast, consider a monthly donation to help keep the content flowing. Support GeePaw Here. You can also participate by sending in voice messages to be included in the podcasts. These can be questions, comments, prompts, etc. Submit A Voice Message Here.