TDD: Resist Integration Tests

the expense of an integration test can be extremely high. consider the contentment app. this app makes drawings 1) that  distribute across time, as if they were being drawn live in front of you, 2) that are generated stochastically, 3) with a “pixel-inaccessible” framework.

now, it’s important to understand that none of these problems are insurmountable. before you tell me how you’d surmount them, let me tell you how i could. 1) screw time. rig it so it draws as fast as possible and wait for the end to assert. 2) screw stocastic, rig your prng so that you control the number sequence. 3) screw pixel-inaccessible, treat the medium like h/w and put a tee in front of it, assert against the tee.

all eminently doable, at some expense. so should it be done?

i prattle on about the money premise, and i want to be sure you understand how it applies here. please do attend, as this is absolutely *central* to learning how to make TDD work for you.

the money premise says we’re in this to “ship more value faster”.

reminder: “value” could mean any number of things, including fewer bugs, better runtime, or more cases or features. all that matters about the definition of “value” for this purpose is that it is dependent on us changing even modestly complex logic in code.

suppose you surmounted the three difficulties above. what would you have to assert against at the end? depending on your technique for trapping the output, you’d have either an ASCII log with a bunch of labelled doubles in it, or a literal image snapshot. you could either eyeball a given run and say “that will do, don’t change this caught behavior,” which we call “bless”, or you could figure out approximate values for those doubles in advance and assert against them, which we call “predict”.

either way, you will have spent a great deal of money getting those assertions, yes? the three surmounted challenges are not cheap, and tho blessing is cheaper than predicting — we’re talking about a lot of data here — neither of those is very cheap either.

what do the literal assertions you write actually assert conceptually about the contentment app?

that, say, given a blank screen, a specific sequence from the prng, instructions to human-draw over time a line from one corner to the other, at the end there is indeed a line from approximately one corner to approximately the other.

wow. you haven’t proven very much. a real script involves dozens of such lines, as well as text. a real script uses a real prng. a real script is inherently stochastic beyond your control because it depends on the timing of the not-owned-by-you multi-threading in the rendering. (aside, not to mention the odds are good that your test is quite fragile, and will break when you change things in the code that do not actually matter to the user.) i could prove all the things you proved without any rig at all. i could write the script that does that and run it and look, in a fraction of the time, and know just as much as that automated integrated test tells me.

in order to get very high confidence that my code could be safely changed, i would need *thousands* of these extremely expensive tests, and thousands of hours of predicting or blessing work besides.

now, your app may not be graphical like mine, it may not be performing its main function distributed across time like mine, and it may not be stochastic like mine. or. well. maybe it *is* like mine. if you’re working database to browser in a huge sprawling database with thousands of users and hundreds of screens and complex workflow and the kind of bizarre business logic enterprise apps demand, maybe it is like mine.

writing an integration test for it would be investing a very great deal in proving a fraction of a fraction of a fraction of the program’s viability. selenium, a test server, the slow runtimes of the intranet, the golden master db, and most particularly the effectively infinite cyclomatic complexity of the app as seen entirely from the outside.

so?

don’t do that.

put simply, the money premise says that we do TDD because we want more value faster. integration tests in most complex apps do not provide more value faster. as a direct result, in TDD we write very few integration tests, and suggest them very rarely.

2 thoughts on “TDD: Resist Integration Tests

  1. Very well said.

    There’s this natural assumption people tend to have, that it’s “best” to exercise *all* the code, or as much of it as possible. And this will flush out unexpected bugs; maybe even the ones you wouldn’t think to test for.

    I can see the attraction, but it’s a pernicious way of thinking. Where the whole system is all wired together, to enable “full runs,” and the tests exist to *enforce* the view of the whole system as a monolith, except with a bunch of parts mocked out until you’re hardly sure what you’re actually testing.

    Every test should be able to answer the questions “what information do I gain if this test fails,” and “what information do I gain if it passes.” These kind of end-to-end-but-skipping-the-middle tend to be very hand-wavey about what those answers might be :-/

    • A mantra I try to teach and demonstrate: “You work for me, I don’t work for you.” I say that about the code, the process, the tools, and yes — very much — the tests. Too many folks forget to ask the questions about who the benefit is for, what it actually is, and what it costs.

Leave a Reply

Your email address will not be published. Required fields are marked *