The Hardest Problem In The Geek Trades

software development is a *temporal* activity: it takes place over time.

any temporal activity can be shown as a series of snapshots. this is a way of “speeding them up”, usually for the purposes of analysis or instruction. it’s a kind of “as if” thing. for some kinds of thinking, we can just elide, ignore, or lightly annotate the face that the activity takes time to happen.

there’s nothing inherently sinful about these “as if” visions, but sometimes, when their conclusions are ramified into prescription, especially rule-centric process, they can produce unexpectedly unpleasant results. the whole snapshotting conversation is really about the framerate we choose in which to talk about our way of making software.

how often do i take a snapshot of “my way” to make it valuable to onlookers? and the answer depends almost entirely on who those onlookers are and what they want to do with the resulting sequence of images.

the most common answers in development pedagogy seem to be 1) just show the final result and explain it. 2) one second = one second, and, the most common 3) snapshot once an interesting change has occurred.

at different times, i’ve used all these methods as consumer and producer both. i generally prefer the third when i’m learning something brand new, and the first when i’m out there looking for “how i implemented X?” type answers. i want to hone in on the third, because i think it’s most useful for the largest part of the trade. if we double the population of working geeks every five years, that means that any given time half of the trade is composed of — no offense — noobs.

what noobs — often including me, depending on domain — seem to benefit the most from is sequential frames where each frame has an “interesting” change from the frame before it.

didja ever do something dumb that got your mom mad at you? and she said, “oh, child, what were you *thinking*!?!?” as a noob forging my way into a new domain, that is the question i most often want to get the answers to. “what were you thinking?”

geekery is the sociotechnical act of integrating high levels of creativity with high levels of rigorous technique.

a rewrite: geekery is the sociotechnical act of integrating high levels of creativity with high levels of rigorous technique over susbtantial periods of time to make artifacts that please.(i added that rewrite cuz it mentions or implies all the muddinesses present in this very muse.)

if we are to learn, do, analyze, teach, or give advice about geekery, we have to incorporate *all* of this into our vision.we have to address the artifacts, the pleasing, the creativity, the technique, the sociotechnicality, the humanity, and temporal flow of the enterprise. honestly, nothing else will do.

perhaps my biggest whine — this is not a rant today — about these branded prescriptive processes i encounter every day everywhere all the time: it is the extent to which they seem so often to leave out one of these key aspects.

what i am after is a description of software development that is formulated to include all of it.i don’t know that i have such a thing inside me, or that any of the geeks i love and respect have it in them, either.

at the end of the nineteenth century, david hilbert produced a famous list of “problems we must solve” in mathematics. i guess i believe the most pressing problem we must solve, as an industry, a trade, a discipline, a way of being and living and working, is just this: how can those of us who are successful at this express what we do to those of us who are less successful at it?

What TDD Tests Prove

if u think the semaphore mechanism in your library works, then test to the level of setting the semaphore. if u think it *doesn’t* work, then u need another library or o/s. testing across threads is for people who are testing semaphores.

if u think that your JSON transport works — from your UI’s perspective or your backend’s perspective, then test to the level of feeding it JSON. if u think it *doesn’t* work, then u need another library. testing across JSON transfer is for people who are testing JSON libraries.

if u think SQL works, then test to the level of emitting SQL or logic’ing recordset values. if u think it *doesnt’* work, then idunno even what u should do. testing across SQL lines is for people who write SQL libraries.

i could belabor this, but i feel i’ve done enough listing of the pattern.

sometimes my simple string manipulation creates bad output. the odds are 10,000 to 1 at least that that is because the *billions* of users of that library haven’t found the lurking substring bug. nope. far more likely, i just screwed up on my end.

play the odds.

the money premise says we’re in this to “ship more value faster”. we are not in this, unless we’re the authors of said libraries, to prove that semaphores, SQL execution, JSON transmission, or strings, work correctly. pro-tip: they generally do. now, let me head the yabbit chorus off at the pass: are there library bugs? oh, hell yeah. to cite a well-known case, the c++ STL shipped for *years* with operations that were thought to be thread-safe but weren’t. (look up the “rope” class.)

but when my string manipulation goes south, i don’t *start* with the silicon, the cpu, the language, or the library. i start with what i typed. this is a hard call for juniors. 1) they’re still thinking that the purpose of tests is to prove that all things are meet & just. 2) they don’t really know how their string library works at all.

that’s cool.

1) nothing proves that all things are meet & just, nothing. not one thing. nothing. nope nope nope.

2) there are a variety of techniques for learning how library X works, and you should try whichever ones will help you. i know my string library, but i quite often write tests just to see if i’ve invoked something (a little less common than string) correctly.

i use tests for this cuz it’s a handy way to probe something: u get to invoke a toy situation to understand how the library will behave when u use it for real. but that’s just convenient in context. i usually toss those once i get it. those tests are useful for me to run. they *might* be useful for me to keep, if the library is a little unstable or my learning is still wobbly. but they’re not the point of TDD.

i use TDD not to establish that all the things in my program are meet & just, and not to establish that, say, semaphores work. i use it to establish that what i *said* is what the computer *heard* is what i *meant*. and if u think that is a very low bar, then i suggest that you’re either not a working geek, or you’re young, dumb, and full of memory, and convinced that your capacity is infinite. i’ve encountered many folks of both stripes.

making sure that what u *said* is what the computer *heard* is what u *meant* is, in fact, the hardest part of professional programming for the first X (3? 7?) years. it eventually transforms in to design, where the hardest part is breaking non-toy problems into toy problems.

i am prepared, in the fullness of time, to discover that 90% of my programming failures come from other source than just the trivial breakdown of one of those two links, said -> heard -> meant, from me to the ‘puter. i would like *so* *much* for something else to be at fault. i would love it if every complain logged against my code was because a) i didn’t understand what was wanted by the customer, or b) the customer-interpreters i work with are vague and somewhat mentally less capable.

sadly, that’s not my *main* problem. my main problem is that the code doesn’t do what *i* thought it did, regardless of whether it does what i was told it should do or whether what i was told it should do is what the buyer actually wants or needs. i know such problems exist. of course they do. but they are of a higher order. it doesn’t matter that it isn’t what i was told or that it isn’t any good for the market if it does not do what i want it to do.

TDD proves to *my* satisfaction that *my* code does what *i* think it does. proving that is just prima facie evidence of awesomeness, but disproving that makes all other links in the chain irrelevant.

Take A Minute To Just Marvel: GeePaw on Coaching

when you think back to first generation Hamurabi, and then play Oxygen Not Included, or Civilization or any of the other current world-builder games, you see the power of steady accretion of improvement. do you know that all of this has happened well within my lifetime? i played hamurabi in 1979, killing time in the one and only computer class i’ve ever taken (and did not complete).

from taking my figures and interpolating a partial differential equation and spitting its own figures back at me, to, well, these little uniquely rendered bastards are whining at me because the decor in the rooms lacks any art, and the food tastes like crap, and it’s stinky, INSIDE THE ROOMS I HAVE CARVED FOR THEM OUT OF AN ASTEROID.

the hamurabi model is one kind of gaming model, there are many, and what holds true here holds true in all of them: with the accretion of improvement in very small doses game to game, we’ve produced something my 80’s self would think of as quite impossible to do. nothing has changed, and yet everything has, in little tiny, at times almost imperceptible, bursts of micro-change.

change, for temporary and local good or temporary and local bad, very often works just this way.

about 20 years ago, i read a book of “evolutionary epistemology”. it collected about 200 years of key papers that formed the seminal documents of this emerging discipline, what most of us would now call evo-psych. (how do u get 200 years of papers in an emerging science? by retroactively designating older work as belonging to your affinitiy. this, too, was ever thus, and has created many distortions in cultural history.)

what was remarkable to me about that book of papers was there in the selected lineage, in two forms.

  • first, it was that tho the gap between the first paper and the last — the richness and power of the understanding — was notable and significant, the gap between any two papers neighboring in time was pathetically small. some thinker would write a paper elaborating or replacing one phrase in a paper that had appeared 20 years before. just one phrase. and so it went, paper after paper, for 200 years. (if memory serves, it may have been more like 300, not sure.)
  • second, it was that the selection of papers drew a straight line from the past to the present, when in fact there is no straight line that works that way.

if it’s winter, take a look at the nearest hardwood tree and study its shape. pick the tallest branch-tip and trace it back until it disappears into the ground.

notice some things as you do this: 1) how hard it is to do. 2) how many branches get excluded from your tracing. 3) how not tree-like your resulting mental sketch. 4) how temporally bound your assessment of tallest is. 5) how much harder it would be to do with a tree that had leaves on it. 6) how very very very much harder it would be to do if you were *in* the tree.

this is us, fellow coaches, fellow geeks, fellow managers, fellow thinkers, fellow *humans*. we’re in that tree, in a time, tracing out its shape from top to bottom, and we are fundamentally, permanently, ineluctably caught there. we live there, and we can’t live anywhere else. (to complicate the picture even further, and break my metaphor all to shit, it’s very rare in going from tree-top to root to find a branch that has two or more super-branches, but is ridiculously common in the target of my metaphor.)

what, you may well wonder, is geepaw up to, here? how does any of this have anything to do with coaching or geeking or managing or shipping software? it has everything to do with everything, because its a description of every human enterprise that is any way centered upon the life of the mind.

so? my advice to you, today, right now, if you’re geeking out, or managing, or coaching, or shipping software, is to stop for a second, take a deep breath, and just *marvel*.

for my closing epigraph, i turn to benjamin zander’s marvelous rule number 6: “don’t take yourself so goddamned seriously”. …

Optimizing A Program (And Programming)

A Simple Optimization Problem

Hey, it’s GeePaw!

Got a little optimization issue for you. Let’s look at some code. It’s easy stuff, so I’ve just used pseudocode here. Take a look.

The program starts by entering a loop that runs 50 times. Inside that loop, it calls the Read method. It then enters another nested loop, this time for 100 times. And in that inner loop, it calls the Scan method. Finally, we exit both loops, and we call the Write method. And then, the program is over.

Now, each of those three methods — Read, Scan, and Write — eats about 100 milliseconds per call. The total run time for this particular app is a little bit under eight and a half minutes or so.

Now, suppose we were setting out to improve that performance. We wanted to take much less than eight minutes.

Given what we’ve learned just so far, and nothing else, let me ask you this — where would you start? Would you investigate the Read method, the Scan method, or the Write method first?

I didn’t think this puzzle was going to give you much trouble. Of course, you’d go charging right after that Scan method, wouldn’t you?

Our Rationale

I know. Most of you got that instantly. But in case, you know, you were distracted and had to look away from the screen, let’s explain the logic. See, the Scan method is called way more times than the Write method, 4,999 times more, in fact.

Suppose we went to the Write method and shaved off 10 milliseconds of its run time. How much overall time would we have saved? 10 milliseconds. Write’s only called one time. Any savings we get by fixing Write, we only get one time.

If we shaved 10 milliseconds off the Scan method, we’d save almost 50 whole seconds of runtime — not quite a minute — in one shaving. Because that 10 milliseconds isn’t saved once, it’s saved 5,000 times. Of course, Read is somewhere in the mix, too, with only 50.

So our priorities would be to squeeze hard first on the Scan, then on the Read, and finally on the Write.

So you got a simple optimization problem. Now, I’m going to tweak this just a little bit, and I’m going to repeat it.

Optimizing Programming?

This time, instead of a program, let’s talk about programming. In programming, we scan code, we read code, and we write code. And it’s my absolute conviction we do way more scanning than reading. And we do a lot more reading than writing.

Of course, it’s not linear or mathy. It’s, after all, work going on in our flickery, mind-moving heads. But the linearity, remember, isn’t what matters. What matters is the relative proportion. If we are to optimize programming, instead of just a program, we need to start by optimizing for the scanning operation.

What’s Scanning?

What, even, is scanning? Well, it’s a kind of high-speed, tight-filtered way of seeing. The two properties, speed and filter, can’t be divorced from one another. It’s fast because it’s filtered. It’s filtered so it’ll be fast.

The filters can be different. Sometimes, we’re scanning a whole package of some kind, just scrolling quickly through the code. We’re filtering for the skeleton, the bones of the thing, like an X-ray. At other times, we’re bouncing between packages, and our filter is in the shape of a simple question whose answer we need for whatever next step we’re doing inside our main thread.

Now, did you ever try Where’s Waldo? Or, if you’re not in North America, you might know it as Where’s Wally? Waldo is that doofus with the red-and-white-striped shirt and stocking cap, and he’s got the distinctive round glasses. He’s usually waving. He’s hidden somewhere in a rich visual forest of people and things, many of which are themselves possessed of red and white stripes and little round things that might, in fact, be glasses.

When you’re solving a Where’s Waldo?, you are scanning.

Your mind becomes a Waldo detector. It makes a Waldo filter on the fly, and it’s jumping around the picture in quest of something– anything– that triggers that filter.

When the filter lights up, we focus tightly and quickly decide whether we have the real Waldo or not. Waldo or Wally puzzles are actually quite a bit of fun. True story, while prepping this video, I did, in fact, spend about two hours solving Waldo puzzles on my web screen, and I had a great time when I should have been working on the video.

Martin Handford, the creator of Waldo, is a master of creating things that are hard to scan. He’s the anti-scanner. But when we jump back to our optimization problem, that’s the exact opposite direction that we want to go in. That direction, the opposite of a Waldo puzzle, is what we call “optimizing for code scanning.”

Optimizing For Scanning

So how do we optimize for scanning? Well, the three most important aspects of easy scanning are sizing, grouping, and naming. And these factors aren’t really separable. Each one relates to and drives the others, and no one of them is quite good enough to solve the problem all by itself. If that seems really complicated, it’s because we’re talking, here, about design. Design is just about the most complicated thing a person can do.

OK, the first thing about a Waldo puzzle is its size. Each puzzle is a double-page spread in a large book. Off the top of my head, I’m guessing there’s something like 80 by 40 possible places Waldo could fit. The sheer number of things you have to move your filter over is enormous.

Now, if the pages were smaller, there would be fewer places to look for Waldo. Ha! Easy peasy, right? Sizing is exactly as simple as that. And it’s exactly as difficult as that, too.

Well, think about it. Making a page smaller means making more pages. If we just published a Waldo puzzle by drawing a grid on top of the real one and splitting it up into little, tiny, arbitrary subpages we wouldn’t really have improved our performance much, if at all.

No. The pages have to be grouped. Grouping is dividing the pages along lines that, in some way, make sense. Grouping means putting related things on the same page and unrelated things on a different page.

And, of course, that’s going to lead us, in turn, right to naming. If all the things aren’t all on the same page, we need ways to clue us in about what things are on which page. Those clues are, overwhelmingly, the names that we give.

So we’ve got sizing the pages to make them smaller. We have grouping of the things that fit on a page so that closely-related things are together. And we have naming of the pages to give us the clues to help us get to the right page as quickly as possible.

Easy Scannability *Is* Design

The problem of making things easier to scan is the basic design problem. And look, there’s no way that’s going to fit in a dumb old GeePaw. video. In the meantime, though, let me offer you two pieces of advice, just to get you started.

First, notice. The next time you’re geeking out on a problem, watch yourself work. You will find yourself scanning. You’ll know exactly what we’ve been talking about here. You’ll scan and scan and scan, bouncing here and there, taking things in at a glance.

In particular, notice the times when a scan fails, when you intended to check something really quick and get back to your main work, but you don’t find it. Just notice this. And then think a little bit about what it means.

The second thing, change things. Do you know how Google came to be so effective as a scanning engine? Because that’s what it is. It’s a scanning engine. By changing its page rankings based on detecting when the scan worked well versus when it didn’t work. They changed their organization. When scanning doesn’t work, change the thing being scanned until scanning works better.

We have a name for this, by the way. We call it refactoring. We’re changing what the code says without changing what it does. When you refactor, you’re actually optimizing for scanability.

As programmers trying to ship more value faster, we want to optimize first around the part of the work that we spend the most time on, and that is scanning code– formulating little, tiny questions and flipping through the code looking for the answers. We optimize for scanning by focusing our attention on sizing, grouping, and, above all else, naming.

So there you have it. Now, get back to work. Well, I mean, as soon as you’re done looking at Where’s Waldo? puzzles on the web, get back to work. Go back out there, start noticing yourself scanning, evaluating how it goes, and changing the code to make it easier for yourself. I’m GeePaw, and I’m done.

Optimize Programming By

Optimizing For Scannability