TDD & The Lump Of Coding Fallacy

Hey, it’s GeePaw, and if you’re just starting to look at TDD, refactoring, the modern technical synthesis, we need to start with a couple of minutes about the Lump Of Coding fallacy.

You’re a working geek: you spend your days coding for money to add value to your company. And one day some random schmoe like me comes up to you and says, hey friend you really ought to try TDD, because that value that you’re adding, you could have added even faster if you used TDD. So, you check it out, because sometimes random schmoes actually have a clue, and you see pretty quickly that TDD means making lots of automated tests against your code. So what do you do?

Well, you back away slowly and you casually slide your hand into your pocket to get that little reassuring grip on your pepper spray, just in case.

Why is that? Well, it’s pretty simple really.

Look, you say to me, I already spent all day rolling code to add value. But if I do TDD, I do automated tests, and try to keep up with me here, doing automated tests is also rolling code.

So, if I adopt TDD, I suppose you think I’ll just spend all night coding the tests that cover the value I spend all day adding. Look random schmo, in the immortal words of Sweet Brown Wilkins,

“Ain’t nobody got time for that.”

OK. OK, my random schmo response to this is to suggest that your analysis is off. And specifically, that it suffers from what we call the lump of coding fallacy. You think of what you do all day as a single behavior, coding. A large undifferentiated lump of work. And you think that TDD means adding to that one lump of coding a whole different equally large lump in order to make the test.

Three Things We Do During A Coding Day

Here’s the thing, I don’t think what you do all day is a single behavior. I think it’s made up of several different behaviors. Let’s take a look.

  • One thing we do, we actually program the computer, and there’s two parts to that. There’s actually changing the source and then there is what we call designing, which is imagining forward how we’re about to change the source. That’s programming the computer. For many of us, it’s the best part of our day.
  • Next, we study. It’s not possible to change the source without knowing something about it. And the studying is how we know something about it. Again, there’s a couple of different parts. We scan source, which is flickering quickly from place to place. And then we read source, which is far more intense. It’s more line by line, a deep analysis of what’s really going on in the code. So that’s study.
  • The third thing we do during a coding day is what we call GAK activity. GAK is an acronym. It means geek at keyboard. And what it means is, running the program in order to accomplish some end or another. In GAK inspection, we’re running the program to see how it works right now. In GAK testing on the other hand, we’re running the program again, but this time to see whether some source change that we just made had the desired effect. And of course, there’s always GAK debugging, where we’re running the program, this time in debug mode, or print statements, or whatever, to see why that source change did not have the desired effect.

Two Points For Later

Now, before we go any further, I want to make sure you know two key aspects of the TDD world. First, when you’ve adopted TDD every source file really becomes a pair of files. One holds the source code we ship, and the other holds the source code for the microtests. And we use both of these files all the time, because they both offer us a great deal of information. That testing code forms a kind of scaffolding around the shipping code and we’re going to take advantage of that.

Second, TDD uses very small, very fast tests called microtests, and it runs them separately from running your entire application. The reason we can get away with testing only parts of the app, is because what matters most about our work is the branching logic that’s in it. And that’s the part we test most heavily using microtests. We run them in a separate app for speed, selectability, and ease of use.

So, take those two points and set them aside. They’re going to become important here in a minute. Hold onto them.

The Proportions Of The Three Activities

OK, let’s go back to our three activities. So, take these three things together, changing codes, studying code, and GAK activity, and you see there isn’t just one solid lump called coding. There’s all these activities. Of course, they’re totally intermingled throughout the day. And that’s why we think of it as a big lump. The truth is, they actually take up very different proportions of our programming day.

Programming the computer, the best part of the day, is often the very smallest part. The GAK activity, much of which is just waiting around for things to run, or clicking through screens and typing in data in order to get to the part where you wanted to see something, that is the largest part of the day by quite a bit. And studying, the scanning and the reading, well, it’s somewhere in the middle. So those are your basic proportions.

When TDD tells you that writing automated tests will make your life better, the lump of coding analysis of the idea is both right and wrong. Lets grow this picture a little bit. We’ll call what we’ve got now before TDD, and then I’m going to disappear, and we’ll put after TDD over on the right.

The After Picture

The lump of coding fallacy is absolutely right about one thing, automated tests are more code that has to be written. Somewhere between half again as much and twice as much as you write now. Let’s say that part of our day doubles. On the other hand, the lump of coding fallacy is totally wrong about the rest of the picture.

First, study time will go down after TDD. It’s not that we have to study any less code in the after picture than in the before. Rather, it’s that studying the same amount of code gets faster. Why? Because those twin files we talked about, one with shipping code and one with testing code, it’s almost like the test code forms a kind of Cliff’s Notes for the shipping code. A scaffolding that makes it easier for us to study, and this makes it far easier to tell what’s going on. This will cut our code study time in about half.

Finally, we come to the GAK time, and this is the big payoff. TDD reduces the amount of time you spend in GAK by 80% or 90%. Because TDD tests run in that special tool kit. They’re fast. They don’t fire up your application. They don’t depend on things like logins, or database permissions, or waiting around for the web to load. They are built to be fast, small, and grouped into convenient suites. Nothing completely eliminates the need for GAK work, but TDD slashes the amount of time you spend GAK-ing during the course of the workday.

So, when we look at the left and the right, from before TDD to after it, you can see it for yourself. We write more code, automated tests. And far from losing our productivity, we actually gain it. All this takes is for you to look past that single lump of coding fallacy.

GeePaw Advises

OK, it’s time for my advice. The first advice is the same advice I always give. Notice things. In your next full working day, notice how much time you spend in the three behaviors, actual programming, code study, and various GAK activities. Once you see it yourself, you might want to consider making some changes.

The second thing, well, TDD doesn’t come in a day. It takes some lessons and some practice. There’s a lot of course material out there, including mine. Watch some videos. Read a little, and really try the various exercises. Start with some toy code. Almost anything will do. Then find a small problem in your day job that has few or no dependencies on other classes. Do this two or three times. And again, notice what happens. If you like the result, well, at that point, you’re ready to get serious about TDD. And then, well, we can take it from there.

So, I’m GeePaw.

Drop that Lump Of Coding fallacy,

and I’m done!

How Long (Technique Remix)

how long (redux)?

in the technical side of the modern synthesis, we develop code by writing and passing tests then re-working the code to make it as change-enabled as we can.

the key “how long” aspect to this: how long does the code stay not working? that is, how long in between the times we could push the code straight to production? the desiderata in the modern synthesis is that we try to measure that number on a scale using only two digits of minutes.

it’s as if we’re playing a game, and the object is to keep the application running correctly even as we are changing it. over time, as you get stronger and stronger at it, this becomes an obsession for the geek. it’s a great fun game, btw, and an old hand like me derives much pleasure from the challenges it presents. like many others, i not only like playing it for real in my codebase, i also enjoy playing in my imagination for other people’s code. 🙂

but of course, it’s not *there* just for the kicks it gives us. there are several benefits it provides for a team that seeks to ship more value faster.

  • first and most importantly, it narrows mental scope. recall that managing this, the number of conceptual balls you’re jugging in your mind, is a key part of being able to move quickly in code.
  • second, still valuable, it reinforces a deep commitment to “always shippable”. the drive for ways to change the code in small chunks without breaking the app frees us from complex source-branching strategies, and lets the value-defining teammates “turn on a dime”.
  • third, it has its own indirect effects on the code base. code that’s built this way over repeated applications of the process is almost insensibly becomes fully change-enabled in to the future.

anyway, the effects of working this way are profound, and to some extent they define the technical side of the modern synthesis. this lunch isn’t free, of course, it involves coming up with solutions to at least three different kinds of problem.

the first problem is controlling how the change is exposed to the user. largescale changes in code don’t happen in ten minutes. that will mean there’ll be a time when a given new feature or changed behavior is building in the codebase, but shouldn’t be visible to non-dev users.

in a pure “add” situation, there’s a pretty basic solution: some kind of feature or rollout toggle that makes the added functionality invisible in one state and available in the other.

pure adds are relatively rare, though. most of the time, practicing evolutionary design, the code we’re changing is spread through multiple locations in the code. here things become more interesting.

two of the original design patterns come in handy pretty frequently here. 1) the Strategy pattern makes it easier for me to supply ‘pluggable’ behavior in small units. 2) the Factory pattern lets client code magically get the *right* strategy depending on toggle values.

a caution: the road to hell is lined on either side with “one ring to rule them all” strip malls that will make you want to only ever use one technique for controlling user exposure. we know what lies at the end of that road. 🙂

the second problem is the certainty problem. how can we be certain we have not broken the app as we make our changes?

the toggle-ish solutions from before can aid us here, *if* we have very high confidence that the toggling itself is idempotent. that is, if putting the toggle mechanism in doesn’t *itself* break the app.

a preferable solution, usually faster, once you’ve set yourself up for microtesting and refactoring, is to bring the app’s business logic under a high degree of scrutiny with microtests.

microtests are exceptionally cheap & powerful in asserting “this is how it works now”. if the awkward parts of what’s being tested are isolated, we can microtest around considerable chunks of the code. they give us *tremendous* confidence that we can’t go blindly backwards.

(note: that’s not certainty. there is no certainty. that’s confidence, though, and confidence goes a long way.)

the third problem we have to solve is sequencing. how can we chain together a bunch of steps so that a) we get there eventually, and b) the outcome of each step doesn’t break the app and c) we can still follow a broadly coherent path?

sequencing is the meat and potatoes for the refactorer, and when we change code, we very often also have to refactor what’s already there.

sequencing very often involves inventing steps that are “lateral” rather than “towards”. what i mean is, effective sequencing often involves steps that don’t immediately aim at the endpoint.

an example: it’s not at all uncommon to do things like add a class to the code that you intend to remove from the code four steps down the line.

instead of twenty-five steps, all towards the endpoint, we take one side-step, four steps towards, and one more side-step at the end. fewer steps is better if they’re all the same size. (forthcoming video of an example, in my copious free time.)

learning the modern synthesis — the technical side at any rate — is very much about just learning the spirit of this approach and mastering the various techniques you need to move it from on-paper to in-practice. it’s not an instant fix. learning this takes time, energy, and great tolerance for mis-stepping. but the payoff is huge, and in some of these cases it can also be nearly immediate.

it all begins with this: “how long?”

how long will the code be not working?

pro-tip: shorter is better.

How Long?

how long?

what amount of time passes between you saving the file you just changed and you seeing the results of that change?

if that answer is over 10 seconds you might want to wonder if you can make it shorter. if that answer is over 100 seconds, please consider making a radical change in your approach.

thinking is the bottleneck, not typing, we’ve been over this a bunch. but waiting you can bypass — *all* waiting you can bypass — is waste.

i know firmware folks who routine wait 20 minutes from save to run. why? because they are running on the target, and that means sending the image to the target, stashing it in the NVRAM and rebooting. every time. wow, that must be some intricate h/w-dependent rocket science stuff they’re working, eh? and sometimes the answer is yes. but i’ll let you in on a secret. not usually. usually it’s logic that’s got almost nothing to do with the peculiarities of their h/w target. i’m not joking. http and ftp servers, things like scheduling — *calendar* scheduling, not multi-task scheduling.

before we all pile on, tho, maybe we better check ourselves on this.

  • do you have to bounce a server or a service inside a container to see your results? do you have to switch to a browser? do you have to login and step through three pages to get to the right place?
  • do you have to manually or automatedly clear the cache or wipe the database? do you have to go look up stuff in a special place?
  • do you have to edit a config file, or just quick like a bunny bounce to the shell and type in those same 87 characters again (except for the three in the middle)?

hmmmmmm.

and checking the results. do you do that in the browser, too? or maybe you study the log output. or, again, bounce to the shell and tweak 3 characters of the 87. of course, maybe you don’t check the results at all, one does see that from time to time. “it ran, so it worked.”

if i don’t know whether my change did what i wanted in 10 seconds i get grouchy. sometimes i suffer it for an hour, cuz the change i’m making is the one that will free me to get the faster feedback. but generally, long wait-states between change & results turn me into Bad GeePaw.

the answer, nearly every time, is to use one of several techniques to move your code-you-want-to-test from a place where that takes time to a place where that’s instant. sometimes doing this is a heavy investment the first time. most non-tdd’ed code is structured in one of several ways that make this difficult. demeter wildness, intermixing awkward collaboration with basic business logic, god objects, dead guy platonic forms, over-generalization, primitive obsession, all of these work against readily pulling your code apart for testing it.

the trick is always one step at a time. remember the conversation the other day about supplier and supplied? start there.

you’re either adding a new function or changing an old one, yeah? if you’re greenfield in your tiny local function’s context, try this: put it in a new object that only does that tiny thing. it sits by itself. declare one in a test and call it. lightning speed.

if you’re brownfield, you can do the same thing, but it’s harder. the key is to first extract just exactly as much of the brown field code you need to its own method. then generally pass it arguments rather than using fields. now rework it for supplier/supplied changes. finally, once again, pull it to a new class. note: you don’t have to change the existing API to do this. that API can new the object and call it, or if it’s of broad use, new it in the calling site’s constructor.

the hardest problem you’ll meet: high fan-out methods or classes. these are classes that depend directly on every other package in your universe. for these, well, don’t start there. 🙂

legacy rework is not for sissy’s.

My TDD Isn’t Judging Your Responsibility

An old blog of mine, We’re In This For The Money, recently got some attention, delighting me, of course. A full h/t to Jim Speaker @jspeaker for that! He quoted me thus:


Too busy to #TDD?
Too busy to pair?
Too busy to #refactor?
Too busy to micro-test?

I call bullshit. My buddy Erik Meade calls it Stupid Busy, and I think that nails it pretty well. All you’re really saying is that you’re too busy to go faster.

~@GeePawHill


Most of the responses were RT’s and likes, both lending me fresh energy. A modest number of folks chatted with me, and that gives even more juice. (Readers please note: the sharers of these things you enjoy need to hear that you enjoy them, early and often. It is what gives us the will to share.)

I Believe What I Said…

Since that article, from 2014, I’ve reworked the ideas in it many times. I have developed theory to back it up, experiments by which someone can prove it to themselves, and a great deal of teaching material. I have written and spoken repeatedly on the topic of shipping more value faster. It’s part of almost everything I write about the modern software development synthesis. About the best recent statement, though I am still formulating new takes on it all the time, is in the video (w/transcript) Five Underplayed Premises Of TDD.

If you want to ship more value faster, you might want to consider TDD, as it will very likely help you do that, for nearly any definition of value you wish, provided only that the value involves writing logically branching code. I genuinely believe that the modern synthesis is really the fastest way to ship value that we have at this time.

…But I Don’t Agree With Everyone Who Believes What I Said

And yet. I can easily see how one might infer from that quote — even from that whole blog, or from other parts of my output — that I am suggesting that using the full modern synthesis is a responsibility for a professional developer.  Or to reverse the sense, that I am saying it is irresponsible to be a working programmer who works in another way.

Some who responded took me as saying that and opposed that idea, others took me as saying that and favored that idea. I’ve no idea what the RT’s and likes were inferring.

I am not saying that it is irresponsible of a working programmer to not do TDD et al, because I do not believe that.

Frankly, the idea gives me the willies. So I feel it’s incumbent on me to give you some detail about what I would actually say about this question of responsibility.

(Writers don’t get to decide what readers make of their ideas. You’ll take the money premise and all the rest of my output you decide to bother with, and you’ll make of it what you will. All the same, if can see how you might think I was saying that, and if I disagree strongly with that, I feel my own — normal, individual, human — sense of responsibility to do what I can to make my ideas more transparent.)

Near-Universal, Individual, And Fraught With Internal Conflict

Nearly everyone develops a sense of what they are obliged, by their own description of who and what they are, to do in this world. We normally call this the sense of responsibility. For most of us, the twin pillars of that sense are 1) our deep beliefs about what is morally right or wrong, 2) our deep connections in the many networks of humans in which we find community.

Those senses of responsibility are rich, vague, staggeringly complex, and full of outliers, odd twists, preferences, and, when put in to language,  startling contradictions.

Above all, they are individual. There are aggregate and statistical trends, of course, and if one could find an N-dimensional space big enough to hold the variables, one would see bell-curve like manifolds all through it. I have a sense of responsibility, and the odds are overwhelming that you do, too. Our respective responsibility-senses may appear to be similar, even highly similar. But they are not. It only appears that way because of the limits of language and the locality of the context we use to frame them. Our responsibility-senses are different because we are different.

If you read widely, you will have seen hundreds, maybe thousands, of situations in which one responsibility a person senses is pitted against another responsibility that same person senses. If you don’t read at all but you are old and thoughtful enough to seriously review your past, you will also have seen hundreds, maybe thousands, of situations like this in your own personal history, and that of others you know well.

Cutting To The Chase

Boy oh boy was I ever about to sprawl out and produce a nine-volume treatise on ethics and responsibility. From my simple dog-like goodness, I will spare us all. Instead, I will sum up my beliefs:

  • that the modern synthesis is the best current way we have of shipping more value faster, whenever that value depends on logically branching code with lifetime longer than half a day.
  • that I don’t honor my own sense of responsibility at all times, because I am weak or foolish, or because I carry conflicting senses simultaneously.
  • that it is not for me to say what is responsible or irresponsible — global words — about any local context involving people who are strangers to me.
  • that “doing your job well” is very often not dependent on shipping more value faster.
  • that “doing your job well” can mean many things that are not good for you or the company you work for.
  • that “doing your job well” is in any case not particularly more important to me than any number of other responsibilities I carry.

If you are a working programmer, you are a grown-up.

You are in the difficult position of balancing dozens of competing needs, using only your heart and body and mind, and I am well aware that this is a task far more demanding in real life than it seems, on paper, in a blog, or spoken by an idol. I won’t do that for you because I can’t do that for you. I only sometimes manage to do it for me.

In One (Long) Sentence

Please don’t take me, even in a crabby mood, as someone who assesses your sense of responsibility, commitment, or heaven forbid, morality, on the basis of whether or not you do TDD or any other part of the modern synthesis in your job.

 

How Much Design? When To Start & When To Stop

Alright, pre-coding design. By that, I mean, the serious thinking one does before one starts typing code into the computer.

The first puzzle of pre-coding design is to identify when to start & went to stop.

Start When You Have A Story

When to start: when a story has been selected for implementation.

That starting condition will be tricky for noobs, so let me blather for a minute. There’s no time to discuss stories in general, but in short, by story I mean “slice of functionality that goes from ‘pixels to registers and back’ doable in less than a couple of days.”

All the way in and out is very important. A couple of days is very important. There’s other very important, too, such as valuable, and some unimportant parts noobs often think are most important. No way we have room to discuss how you get to stories like that, we’re gonna stash that for another day.

Now, don’t go off telling everybody I said they should shut up until we have a story picked. We’re gonna talk, kids. We’re gonna talk, and it’s gonna include design ideas. We need that burble of ideas and insights before we even pick a story, after all. That’s fine. What I’m saying is, don’t take it very seriously. Sketch this, poke at that, look this up, schmooze it all over. That’s fine.

But design that specifies changes at the code level in any detail at all waits until the story has been chosen. You start when you have a story and you’ve decided to implement it.

Stop When You’re Excited BUT Nervous

When to stop: when you are at the place where you feel excited that you’re probably right but nervous that you might be wrong.

This surprises a lot of folks.

People who’ve dabbled in this movement will have encountered a great deal of railing against “Big Design Up Front”, and they think that means there’s no designing allowed. They’re surprised I encourage us to design at all. Relax. The method works for you, you don’t work for the method.

People who’ve trained and worked their entire careers old school, on the other hand, think that this is way too little design. One hears the word “cowboy” among the epithets. Relax. We’re not done designing, we’re done pre-code designing.

“Hey. I think we got it.”

That’s the sweet spot for stopping the pre-coding design. Notice that “think” isn’t “know”.

If You Pass The Big Black Bulldog With The Droopy Eye…

Let me throw out a few “too far” indicators, so you can get a feel for that sweet spot to start with. If you pass any of these points, you’ve gone too far in your pre-code design:

* you think all the thinking is done.
* there’s zero risk.
* you’re so detailed there’s a program that could just translate it directly into code.
* you can just send all the lesser lights out to type it in.

Right? If you see one of those droopy-eyed bulldogs you’ve taken that pre-code design too far.

Keep Going Past Where The Dairy Queen Used To Be…

Well, if we’ve got “too far” indicators, let’s look at some “not far enough” ones.

* you have no idea what files/modules/objects are likely to change.
* you are terrified it won’t fit in a couple of days.
* you don’t know how you’ll respond to a generic failure in the new functionality.
* you don’t know how you’ll enable/disable the new functionality.

When you hit the keyboard, you want just enough to make you feel you know where to start typing. Don’t worry that you haven’t done enough thinking. You haven’t, and that’s the correct situation to be in.

Hey, That’s A *Feeling* Not An Algorithm!

Yes. It’s a feeling, not an algorithm. (The force is strong in this one. <cue eye-roll>.)

There’s no computable algorithm for determining when you’ve done enough design. Very intelligent bozo’s have been trying and failing to find one for quite some time. Or, maybe there is one, maybe one exists, but it doesn’t matter because we don’t know it yet.

And before you bemoan this too much, try to remember: the current non-computability of the software design problem is why you aren’t on the night shift at 7-11. Or hiding from SkyNet in the rubble.

So you’re going to have to go with your gut.

Collaboration May Be Called For

The decision point will vary from one context to another, depending on factors including but not limited to: codebase knowledge, mad geek chops, domain knowledge, really offensive arrogance, number of eyeballs, most recent collossal failure, etc.

In a collaborative situation, people will disagree about their gut feelings. I’ll offer a pro-tip: don’t disrespect folks who are less confident than you. Find a 50/50 compromise, then bias it a little towards the nervous side, and do it explicitly and openly and cheerfully. The time difference isn’t that significant. Good nervous cheer from your pair is very significant.

(On the other hand, I usually resist compromising with corporate handbook nervousness. I especially target here company-wide design rules, which are nearly always borked beyond any ready solution.)

Is That All The Design We Do?

Of course not.

The stopping point above isn’t the stopping point of design, far from it. Remember, we’re talking here about pre-coding design.

When we say we’re ready to code we most definitely don’t mean we’re ready to not design. The real design is just heating up, in fact. All through the implementation process we will be making further design decisions.

The fear is that we haven’t done enough thinking to really solve the problem.

The reason we have that fear is because that’s the situation: we haven’t done enough thinking to really solve the problem.

But I’ll let you in on a little secret. Thinking is done with thoughts. Really solving the problem is done with code. You will never really solve the problem by thinking, you will only ever really solve the problem with the code.

That doesn’t mean we don’t think. It means we think enough to code, then enter a tight little cycle of code-think-code-think-code-think.

Design Enough To Code,

Then Code-Design-Code-Design-Code-Design

Pain And TDD


“We were doing TDD just like we’re supposed to, but at some point it started hurting, so we not only quit, we told everyone else on the interwebs that TDD doesn’t work.”

I met Kent Beck after a couple of years reading and writing around him, at the first ObjectMentor Extreme Programming (XP) Immersion, a wonderful week I’ll never forget, where I met so many awesome geeks and coaches. (Kent Beck, Mike Feathers, James Grenning, Bob Martin, Ron Jeffries, Martin Fowler, and more! You get the idea it was a special week?)

And Beck, answering a question, told the old Smith & Dale joke:

SMITH: Doctor, it hurts when I do this.
DALE: Don’t do that.

(The visuals include a funny wagging arm movement by Smith.)

Don’t Do That!

I chuckled. It was merely years before I grasped the deeper level of this old joke.

One might interpret it as saying “TDD hurts me, so I won’t do TDD,” but that’s actually the wrong way to interpret the joke.

You see, TDD works for some of us. And it works very well.

But it’s really important that you hear what I’m not saying.

I’m not saying that you didn’t — or aren’t — feeling pain. I’m not saying you weren’t — or aren’t — trying. I’m not saying TDD works for us because we’re smart and you’re stupid. 

What I’m saying is that you are not responding to the pain you’re feeling in the TDD way.

Pain Means Something Is Wrong

TDD is not painful.

I don’t mean that it’s not painful in the sense of “you’ll just feel a little pinch”. I mean it’s not painful at all, really.

So when we feel pain while doing TDD, something is going wrong.

The somethings that could go wrong are exactly three:

  • You are working in legacy.
  • You are learning.
  • You are working with code that needs refactoring.

That’s it. There are really no other situations where TDD should be causing pain.

Legacy Pain

TDD can leverage you out of legacy. I’ve done it. But working in legacy is just painful.

The key to successful legacy work is to become a master of extraction refactorings. Before you TDD a change, extract the change site so that it is optimal for microtesting.

When you can’t write a real microtest because of the location or accessibility or complexity, either move, extract, or simplify the code via refactorings. It’s not hard, but it’s tedious as hell, and it takes a certain boldness.

This will help. It will ease your immediate TDD pain, and it will — very gradually — lift you out of legacy work altogether.

But yeah. Legacy hurts for as long as it’s legacy, and there are not two ways about it.

Learning Pain

Most of the time, learning is a source of great joy to us. After all, we’re geeks. That’s what we do, is learn things and delight in them.

But there are some situations where learning hurts. Most of these are situations where you have to learn something while 7 people are shouting at you that if we don’t fix this we will simply all die. Let me tell you, that’s gonna hurt no matter what you do. Suggesting, of course, that we teach people not to do that.

There’s another time when it hurts. It hurts to learn when your current goal is too far over your head.

I have worked from time to time with folks with weak TDD, weak confidence, and weak knowledge of a language or a domain. That’s a painful combination.

If you are stuck at the incantation stage in a language, where you’re just barely avoiding syntax errors, and the fundamentals of the language are not yet within your grasp, it’s sore tempting to drop TDD altogether and just focus on typing the magic in correctly.

But it’s contraindicated. The reason TDD is a great design technique is because it organizes the design. That organization, plus simple extended exposure, is going to be the thing that ratchets your knowledge up most rapidly. Abandoning TDD will slow down your learning, not speed up.

Code Pain

Ahhhhhhh. Now we come to it.

I have a strong belief that code pain is the main reason people leave TDD behind.

And here’s what they’re doing wrong: gritting their teeth and trying to play through it.

That’s not the TDD way. In TDD, when we feel pain we change the code until we don’t feel it anymore. And we add what we’ve learned to our daily codesense.

Codesense isn’t one trick, it’s a thousand. And it isn’t an algorithm. Codesense comes into play at large scales and at smalls. But it all comes from feeling pain and resolving not to feel it any more.

Coming soon to a blog near you: live-action codesense examples, where pain leads to refactoring leads to new design.

We don’t play through the pain in TDD.

We use it to change our behavior & build our codesense.

Codesense: The Basis of Successful TDD

Okay, so, I’ve been a little grumpy lately.

But of course, it’s not enough to just shout at people who think TDD doesn’t work.

It must be a big weakness of our pedagogy that so many people feel that way. So I ask myself, what are we not teaching, or not teaching enough of, or not enough of us teaching?

Given the descriptions from the unfortunates who tried not-TDD under the guise of TDD, it seems clear we’ve under-emphasized the means by which the process detects flaws and forces code change.

(In one sentence, I’ll describe TDD as a process. You can go back all over this blog and http://anarchycreek.com to get details.  The TDD process is a matter of deciding what to change, forcing that change using one or more microtests, and concurrently optimizing the design by using your sense of the new friction, coupling, and general infelicity of the code that works.)

Wait. There it is.

Codesense Is Critical

TDD was created *by* geeks of a very high caliber. But it was created *for* all geeks, of any caliber. The result is that whole swath’s of technique have been somewhat assumed, when such assumptions were ill-warranted. To be clear, many folks have contributed much wisdom, but that wisdom is insufficiently studied and taught.

I will tell you now, as I’ve said before in the context of advocating microtests and microsteps: TDD will not work without good codesense.

Codesense is the developer’s basic sense that a given chunk of code is or isn’t optimal.

It’s how we know we’re coding well or less well.

I say the word ‘optimal’ like we’re supposed to already know that that means. The fact is there are bazillions of definitions of code optimality. Many of them capture all or part of what I mean, but others capture little or nothing.

Codesense is like common sense in three ways:

  • it’s incredibly difficult to teach;
  • once learned it seems obvious;
  • it’s noticeably uncommon.

Judgment Is The Keystone Of Codesense and of TDD

I’m gonna go off-track for a moment to make one really important point: there is no algorithm for measuring the goodness of code.

Converting codesense into mechanism is a mug’s game, but it’s been ridiculously common in the industry, and still is. There are dozens of applications you can download, buy, or liberate that purport to identify “code goodness” categorically. Some have great ideas, but they don’t really work.

If I could tell every developer — and consumer of developer services — in the world just one thing, it would be this: people write code, not computers, and that is because individual human judgment is required.

In software development, we are permanently, irrevocably, irremediably, and happily, dependent on individuals using their individual judgment.

Maybe the biggest consequence of this dependence: there are very few fixed rules in assessing code.

Codesense Changes Over Time

What rules we do have frequently change. Nothing anyone said ten years ago should be taken as for-granted truth. Nothing. The game changes too often for that.

To take a straightforward case: structured programming, and there are many many people in our field who were raised on structured programming, says that every method should have exactly one entrance and exactly one exit.

The modern synthesis is different. Nowadays we’d write this code like so:

Is the difference important? I’ll re-use a sentence I wrote nearly ten years ago:

“To a professional, *everything* counts.”

Getting Started

So, this is my first blog about codesense, but I guarantee it won’t be my last. And now that we’ve met the creature in the abstract, let’s get down to brass tacks.

Codesense is hard to teach, but it’s not that hard to learn. It starts with heightening your sensitivity to code.

I can readily recommend all of these books:

It’s important to understand: I’m not telling you to do everything these guys tell you to do. It’s not about getting an instruction book.

I’m telling you to *think* and *observe* and *notice* the code you write and what happens to it.

What’s worth noticing? What’s worth changing? How might you change it? These books and others will help you get your mind open to codesense.

Once that happens, it’s time to start looking at your daily existence as a software geek. There is no substitute for this work. The folks above are top-notch writers and thinkers about programming. They can help you learn how to judge. But the judging will still be yours.

One last thought.

Start Today, With Pain.

Notice Every Pain Code Brings You

 

 

 

 

 

Your Design Is Broken: It’s TDD, not TDYAR

Think about your last huge unrequited crush.

A real one, I’m talking. A crush so big you go to sleep thinking of your future together, and wake up, ummm, guilty. The kind of crush that great romantic stories are made from, and for. The wobbly-knee’d kind.

And a complete non-starter: when I say unrequited, I’m being clinical in an effort to minimize the likelihood of breaking out into laughter, as doctors and nurses in every ER around the world do.

Now, in that kind of crush, you have a vision, and that vision is glorious beyond compare. It captures entirely your ideal notions of successful love. I’m not going to ask you to describe it. In fact, please don’t describe it. I have sensitive comment readers here. But picture that imagined future, just for yourself.

Was it magnificent? Brilliant? A tale for the ages?

I’d bet it was all this and more.

In fact, there’s really only a single flaw in that entire vision.

That Future Romance Is In Your Head

That tremendous romantic vision is just that, a vision. And when it came time for that blueprint-for-love to hit the shredder we call real life, it just wasn’t up to the challenge.

Thus with your design.

You see, your design lives in your mind, like your romantic vision. Oh, it may have slightly more contact with reality, because, after all, I still can’t believe you thought that — never mind.

And, notice, I’m not suggesting you’re not the greatest software designer in known space. My point in no way depends on you being a sub-par designer. Surely, you’re not a sub-par designer of romantic visions for you, are you? Far from it.

Codeless Designs Are Relatively Worthless

A design in your head elides all messy details, tricky complications, and inevitable exceptions.

A problem your company wants to solve never does.

So how do you cope with this divergence?

You code up your design, and as the working code begins to implement the design in your head, you keep an eagle-like watch out for its inevitable infelicities, and you change the code — and your design — until it is still working and no longer has its problems.

That’s what TDD does. A sketch of a design leads to rolling some microtested working code which is re-designed in microsteps. Red. Green. Refactor. Commit.

Design-In-Head Vs. Design-In-Working-Code

When your starting design meets the emerging design that comes from repeatedly applying the TDD process, there will be clashes. Thankfully, most of them aren’t as bad as that great unrequited love. But, on the record, some of them are.

When it happens, which design is right?

The one in the working refactored microtested code is right.

Your Design Is Broken

That’s hard, but you have to let it go.

Use a mental design to work your way into the code using TDD, and your broken design will become a masterwork of productivity. Ignoring the code’s design in favor of the mind’s, and you will pay a great price, starting tomorrow and lasting until your company inevitably goes out of business.

People rail against TDD.

A great many of them are refusing to confront this basic clash of designs, and they just can’t let go of their mental construct in favor of their silicon one. (The most recent such, from the famous @DHH, reeks to me of this.)

And that explains for me a large amount of TDD criticism, and an even larger number of failed TDD adoption efforts.

If you think TDD is supposed to work w/o changing the shape of your designs, you have missed the point. It is not surprising to me that you find yourself hating TDD.

I see a lot of “TDD and my design clash, therefore TDD doesn’t work”. I’ve seen even more of it lately.

TDD Is Test Driven Development,

not

Test Driven “You Are Right”

One Page Intro To Microtests

A microtest has the following properties:

  • It is short, typically under a dozen lines of code.
  • It is *always* automated.
  • It does not test the object inside the running app, but instead in a purpose-built testing application.
  • It invokes only a tiny portion of the code, most usually a single branch of a single function.
  • It never *reaches* inside an object’s boundaries, though the tester may certainly *look* inside them.
  • It is coded to the same standard as shipping code, i.e. the team’s best current understanding of coding excellence.
  • It is vault-committed source, with a lifetime co-terminous with the functionality it tests.
  • In combination with all other microtests of an app, it serves as a ‘gateway-to-commit’.  A developer is encouraged to commit anytime all microtests run green.
  • It takes complete control of the object-under-test and is therefore self-contained, i.e. running with no dependencies on anything other than the testing code and its dependency graph.
  • It runs in an extremely short time, milliseconds per test.
  • It provides precise feedback on any errors that it encounters.
  • It usually (not always) runs entirely inside a single computer.
  • It usually (not always) runs entirely inside a single process, i.e. with few extra-process runtime dependencies.
  • It is part of a collection all or any subset of which can be invoked with a single programmer gesture.
  • It is written before the code-change it is meant to test.
  • It avoids most or all usage of ‘awkward’ collaborators via a variety of slip-and-fake techniques.
  • It rarely involves construction of more than a few classes of object, often one or two, usually under five.

 

We’re In This For The Money

(When I’m teaching and coaching, I have a number of themes that I discuss and that will show up somewhere in everything I do. Here’s the money theme, which I usually bring up at the earliest opportunity.)

Look, kids, we’re in this for the money.

Somehow you got the idea that TDD, pairing, micro-stepping, and refactoring are not about increasing your productivity.

It could be the misguided craftsmanship movement, who give you the excuse to say “we don’t have time for art.” Or maybe you heard it from nay-sayers who are against these notions from the get-go. Maybe you read stuff from the early days, when we still didn’t understand that internal quality was such a huge determinant of success. Or, worst of all, some of you heard it from people who are in favor of these ideas, but get lost in the thick of things and say stuff they really should know better than to say.

It doesn’t matter how you got the idea, though. The simple fact is it’s entirely untrue.

TDD is in every respect about increasing software development productivity.

It is not about craftsmanship or art or beauty.

It is not about being a good citizen.

It is not about taking some illusory long view instead of the short.

It is not about you being a better person.

It’s not about improving quality.

It’s about money.

Do you want to make more money? Does your company want to make more money?

There are various goals you could aim at to do that. Maybe you need to ship more quality. Maybe you need lots more features. Maybe you need a whole new approach to the problem.

Funny thing, though. No matter what your target is, or even what time-scale it lives in, you will get there faster if you are more productive.

Driving with microtests increases productivity.

The microtests do a number of things to make us faster. See my articles on how TDD works its magic, for the many cases:

  • They multiply developer intentions, which provides a powerful “online” memory.
  • They demonstrate with a flick of the wrist that a change you just made did not hurt anything.
  • Their blinding speed far exceeds any other means of feedback in utility.
  • They *force* us to work in small objects with small methods.

In a single phrase? TDD increases productivity because the hard part of programming is thinking, and they make thinking faster.

Pairing increases productivity.

This one’s been demonstrated so many times its not funny. Pairing — at a real pairing station — has the following impacts on productivity:

  • Pairs find dumb-assed syntax errors faster than solos.
  • Pairs find *shared* language for programmer intention, the most subtle and elusive part of the primary task of programming, which is understanding the code in front of you at a  glance.
  • Pairs share energy back and forth, which means they can focus on a single task for far longer than solos.
  • Pairs are more fun. (Okay, that’s not about productivity. Fine.)
  • Pairing is the most efficient mechanism for knowledge transfer.

In a single phrase? Pairing increases productivity because the hard part of programming is thinking, and they make the thinking go faster.

Refactoring increases productivity.

The reason this one’s so hard for the noobs to get is that they haven’t yet understood that the code-to-which-you-are-adding is the single biggest determinant of the speed with which you add. Continuous micro-refactoring provides these benefits to productivity:

  • They render the code-base smaller. Smaller = less to grasp. Less to grasp = faster to change.
  • They reveal intention, which is the hardest part of code-grasping. Not, “what does this code do”, but “why do we want this code to do this”.
  • They continuously improve the overall design, including performance, of the code.

So, you know, in a single phrase? Refactoring increases productivity because the hard part of programming is the thinking, and they make the thinking go faster.

Micro-stepping increases productivity.

We do all of our work, testing, designing, refactoring, really *everything*, guided by one simple question: “what’s the smallest interesting step I can take?” And we do this because micro-steps improve our productivity in these ways:

  • They mean smaller scope, meaning fewer independent concepts, meaning easier reasoning.
  • They make most bugs emerge in those same small scopes. Time debugging is dramatically reduced.
  • They provide readily testable and trackable change. Continuous change is faster than chunky change.

Oh, in a single phrase? Micro-stepping increases productivity because the hard part of programming is the thinking, and micro-stepping makes the thinking go faster.

In The Beginning…

The only time these activities slow you down is in the beginning, when — guess what — you’re still learning their basic mechanics.

Here’s the deep dark secret: working this way is not how you learned to work. How you learned to work was either on-the-job-under-the-gun or buttressed around with perfectly reasonable theories that have no bearing on actual successful practice.

Learning to work this way is hardest for skilled geeks: they have a lot more practice to unlearn. But they’ll also have the biggest productivity gains once they get it.

Step Zero? Set up a decent comfortable pairing rig and invite experts to come and play. They’ll come. Pairing is the beginning, not the end, of this technique.

Learn Your Damned Trade

Too busy to TDD?

Too busy to pair?

Too busy to refactor?

Too busy to micro-test?

I call bullshit. My buddy Erik Meade calls it Stupid Busy, and I think that nails it pretty well. All you’re really saying is that you’re too busy to go faster.

We’re In This For The Money

So Stop Screwing Around And Start Going Faster