Coyote Tracks » TDD

Testing Backward-Incompatible Changes

Kai MacTane — Mon, 12 Oct 2009 18:06:20 +0000

Let’s say one day you decide to add a feature to your software or service. For example, you need a new flag on user accounts, so that different types of users get different features. (These don’t even have to be tiered account levels; maybe accounts of type “music lover” get a widget in the sidebar with suggestions for bands they might like while “sports fan” accounts get a sports scores widget instead.)

So, following good software development processes, you first write a couple of tests:

UserTypeOne gets features A, B, and C
UserTypeOne does not get features D, E, or F
UserTypeTwo does not get features A, B, or C
UserTypeTwo gets features D, E, and F

Then you write some code that meets these tests, including creating a pair of mock users with the new flag set in both possible ways. You run the tests, and they all pass. Yay! Your new feature is ready!

Except that, of course, the moment you check this code in, you find that suddenly, nobody can log in any more. After all, the new flag is supposed to be mandatory, and that requirement is reflected in one of the changes you made to the login code. But the only users that have the NewFlag at all are the ones you just created in your fixture.

Sure, this will probably get caught before it goes live. Even if your entire QA department is asleep at the switch, if just one person in your organization tries to run even the most rudimentary tests, they’ll get back an error message. But “it didn’t get pushed to production” is only the most minimal level of success; I’d like to look at how you, the coder, could have prevented this problem. How could you keep this error from even getting checked in (and thus breaking the build and flat-lining your co-workers’ productivity for a few hours)?

The way a real QA department would spot this problem is regression testing. But it’s not reasonable to ask every coder to do a full suite of regression tests before every commit.

The real failure, of course, is that you made a change that you didn’t test. But saying “force programmers to write better tests” is not a viable answer; people are imperfect. You might as well mandate “developers may not check in any code that contains bugs” as one of your organization’s rules.

I don’t have a good answer to this problem. Asking developers to also be QA people is completely impractical. The current solution seems to be “developers sometimes check in code which breaks the build, and blocks people’s productivity for a while”. (Distributed SCM can help mitigate this.)

Is there something QA people know that developers don’t, that would make an easy solution that we could adopt?

Notes on LJ Content Sieve

Kai MacTane — Sat, 04 Jul 2009 21:07:42 +0000

My latest project is something I call “LJ Content Sieve”: a Greasemonkey script to filter out content on one’s Livejournal views based on nearly any attribute of a post or comment.

However, Livejournal is very customizable. It has 31 different “layouts”, each of which can then be further “themed” by application of CSS. This means that a user viewing a given journal, community, or single post, at a given time, may receive an HTML document containing any of 31 different DOM structures.

This means that finding a post’s author, or title, or even just a single post or comment, is not a straightforward process. I’ve set myself up a Ruby script that downloaded my “friends” view, a representative entry with lots of comments, and my own journal view, thus giving me a set of 93 fixtures that I can test against. The Ruby script also tweaked each fixture in the process of spooling it to my hard drive, by adding the proper calls to JSUnit files.

At the end of each document’s element, there’s a call to jsUnitCore.js. Then, at the end of each , I add a call to the lj-content-sieve.user.js script itself, as well as a set of test files that depends on which fixture this is. Every fixture gets an ljcs-global-tests.js call added to it — that file contains tests that should work anywhere, regardless of what sort of page you’re on.

Then all the “friends” page fixtures get an ljcs-friends-tests.js file, which tests operations that should happen on every friends page. For example, determining which entries need to be deleted. (In contrast, single-entry pages get ljcs-entry-tests.js, and the page from my own journal — which stands in for a view of a community — gets ljcs-self-tests.js.)

Finally, each fixture page gets a test file based on its layout: the “3 Column” layout gets ljcs-3 Column-tests.js, while “Cuteness Attack” gets ljcs-Cuteness Attack-tests.js. (Hey, I didn’t write or name the layouts; I just have to make sure LJCS works with all of ‘em.) These files will test that the actual DOM manipulations work properly.

Without test-driven development and automated testing to ensure that each layout and page-type is being handled properly, I don’t think this project would be manageable at all.

TDD and Peace of Mind

Kai MacTane — Mon, 04 May 2009 16:53:10 +0000

Let’s face it, we’re not perfect. As much as I might realize that automated testing is a good practice, it still feels like a chore sometimes. In my latest round of personal-project development, just setting up a decent set of test fixtures and a working test framework turned into something of a hassle, as it’s my first attempt both at Greasemonkey scripting and at building a script that will act on Livejournal pages. (Since LJ users can customize their views with any of 36 basic “styles”, this means quite a few fixtures.)

So it was awfully tempting to say “screw this!” and just start writing code — you know, “code that works, code that gets stuff done”. Actual program logic, instead of testing tools. “Why don’t I just get something built to begin with,” I asked myself, “and then I can try to test that?”

Ha, ha. Of course, we all know that starting off without any tests just makes it easier to continue without them later. So I took the virtuous road, forced myself to get tests working, but allowed myself to skimp by only setting up fixtures for 3 of the 36 styles. The other styles are something I will have to go back and fill in before I can release, so there’s no chance I’ll “give myself permission” to blow them off.

When you really want to be writing code (because let’s face it, you consider yourself a coder, not a tester), it’s pretty annoying to write tests instead. But writing tests is at least a form of writing; if you’re not even writing tests, but rather setting up a test framework and fixtures, that’s almost excruciating.

But it all became worth it the moment I made a change in one of my basic data structures. I had a structure (what a Perlist would call a hash of arrays) that held information about how to identify which LJ style the page uses. Then I realized I didn’t want to have a separate structure for how to manipulate the page; instead, all information about LJ’s styles and DOM structure should live in one place.

So I altered the structure, then altered the functions created so far that rely on it. And that’s where I would then have to wonder, “How badly did I break everything?” Instead, I just ran my JSunit tests again. And seeing them pass was instant peace of mind. I don’t have to worry that there’s some hidden flaw waiting there, ready to be exposed by a user doing something unexpected. And as I add the other styles, I can easily be sure my code works for them, as well.

I’m still fairly early in the development of this code, and while unit testing has cost me a bit of time, it’s given me back peace of mind. The time profit? I have no doubt that will come later. (And had I made any errors in my data-structure change, the unit tests would have helped me find them more quickly than the usual debugging methods.)

I’m Getting Tired of Yak-Shaving

Kai MacTane — Fri, 19 Dec 2008 03:59:42 +0000

It seems like every time I try to get around to actually writing some code, I discover yet another thing that I have to do in order to make that more possible. Today, it was: Get PHPUnit installed on Lórien, so I could run my unit tests locally during development, rather than having to shove everything back up to Finrod every time I wanted to run a single test.

The yak-shaving odyssey that this necessitated involved upgrading my local copy of PHP (which originally came with InstantRails) from 4.whatever to 5.2.8, so that I could develop PHP5 code at all, then installing PEAR so that I could use it to get PHPUnit, and also resolving the weird extension-loading bugs on PHP for Windows (for example, you have to load php_mbstring.dll before php_exif.dll, or else both of them will fail and complain. Ditto with php_pdo.dll and php_sqlite.dll. None of this is documented in the manual; you have to Google on it and get links to the bug database. Joy.)

But I can now run PHPUnit test on the stuff that I’m working on, locally rather than remotely. So… now can I get some code written?