See the problem? Really, there are a few ways this can go wrong. To a quick glance, the only clue that line 14 (“interpolate a comma”) is part of a conditional is its indentation. The indentation is important to a human reader — but absolutely irrelevant to the PHP interpreter, which simply treats the next line after the conditional as the conditional’s block. Regardless of how it’s indented, and regardless of what else is around.

The way it looks to a human is not the way it looks to the machine.

What happens if someone wants to add some logging? What if they add it after the comma line?

Now the log claims the code has added a comma, even when it hasn’t. But still, it could be worse! What if you decided to add your logging before the other line?

Now it adds a comma no matter what — even the first time through the loop, when the string is empty. So instead of a string like '123,124,125', $itemId will now have a leading comma: ',123,124,125'. Since this value is getting stitched into a SQL query later on, it means your app will blow up with a SQL syntax error.

This is why Python makes whitespace significant to program flow. The way the indentation makes it look like the logical structure is, is how the structure actually is.

And this is also why Perl — of all languages, one that normally errs on the side of letting you leave out anything that can be inferred from context — Perl insists in its syntax documentation that in cases like this, “the curly brackets are required — no dangling statements allowed.” (It then says, in typically Perlish fashion: “If you want to write conditionals without curly brackets there are several other ways to do it.”)

If you’re working in one of those languages that lets you omit curly braces around a single-statement conditional — DON’T DO IT! The potential maintenance and debugging problems are not worth the fun of saving two keystrokes (or just one, if you work in an editor that auto-closes your braces for you).

(It turns out the “now they have two problems” construction didn’t even originate with Jamie; it can be found — slamming awk instead of regexes — in the .sig line of a Usenet post in comp.sys.ibm.pc.rt in 1988, and perhaps even further. Maybe this is just another instance of how, as Jamie puts it, people only remember the stupid stuff he says?)

Anyway, as witty as this line is, it’s wrong. Like any other tool, regular expressions are designed to solve a particular class of problems. And, like every other tool, there is a class of problems that they’re just not so good at handling.

Regexes are excellent for doing pattern matching. If you need to ensure that a text string is, for example, a standard ISBN, or a postal code from a particular nation, or a valid username for a given spec. In any of these (fairly standard and common) validation scenarios, regexes will save you a lot of time and trouble.

To take a fairly trivial case, suppose you want to know if a given string is a valid US ZIP code. If you don’t have regexes available (either because you’re working in a language that doesn’t support them, or because you haven’t learned them), your first thought might be:

But this doesn’t work, because “2134″ is not a valid ZIP code — although “02134″ is, and can be found in Boston. Plus, this conditional won’t handle 9-digit ZIP codes. What you really need to test for is: “This string has 5 digits in a row. Then, optionally, it has a dash, and then 4 more digits.”

If you don’t have regexes, this is where you wind up writing something painful like:

if (strlen(input) != 5 && strlen(input) != 10) {
    return false;
}
for (int i = 0; i < strlen(input); i++) {
    int ascii = chr(substr(input, i, 1));
    if (i = 5 && ascii != 45) {
        return false;
    } else if (ascii < 48 || ascii > 57) {
       return false;
    }
}
return true;

In fact, this isn’t some contrived example I whipped up to make the process seem more awkward than it has to be. This is the best solution I can find without regexes, and it relies on the fact that “the digits from 0-9″ is a contiguous range of ASCII characters and values. If I’d instead wanted to allow “upper- and lower-case letters, but not the characters ` [ \ ] or ^” (a fairly reasonable restriction), I’d have to use two non-contiguous ranges, with a corresponding increase in complexity.

Now suppose you’ve decided usernames must be 4-20 characters long, and may contain letters, numbers, hyphens, underscores, and periods. If you look at this from a user’s perspective (where details of implementation are irrelevant, and all that matters is the actual functionality), this makes perfect sense: it lets users use all the characters they’re most likely to want. They can have names like John.Doe, mary_roe, curly-moe and b_hills90210.

But if you look at this as a developer who’s going to have to implement the filter (without regexes), it looks like a nightmare. “Do we really have to let them use underscores?” you whine (after a quick glance at the ASCII table). To which the answer is, pretty simply, “Yes.” Even people who never heard of an underscore five years ago now expect to be able to use one in their usernames, and if it’s not allowed, they’ll perceive your service as cheap and poorly-constructed.

But really, that isn’t the end of it. Taking a cue from DNS, you should probably ban the “separator” characters (period, underscore, and hyphen) from the initial and final spots in usernames. This means your complete function to test for username validity looks like this:

If you’re not familiar with regular expressions, this is where you’re probably saying something like “Yeah, that looks like a fairly decent, and reasonably flexible, solution.” But the regex solution is so simple, it isn’t even worth its own function:

if (! matchRE(/^[a-z0-9][a-z0-9_\.-]{2,18}[a-z0-9]$/i, \
input)) {
    errormsg = "Requested username is not valid!";
}

The people who already do understand regexes, on the other hand, might have noticed how the three is_...() helper functions in my earlier code sample are groping their way towards regular expressions’ character classes.

If regular expressions are such a powerful and useful tool, why did Jamie Zawinski dis them so harshly? One possible reason is simply “because he could” — the opportunity presented itself, and there’s a level on which wittiness is its own excuse.

But it’s also true that no matter how wonderful a tool is, it won’t give you any help if you use it on the wrong type of problem.

And regexes have been misused an awful lot, on a general class of problem that they’re simply not equipped to handle: text parsing is a much bigger problem than mere text matching — but they’re not completely unrelated; the second is an important part of the first. Which is why lots of people keep trying to parse text (like chunks of HTML) with nothing but regexes.

Which is like trying to build a cabinet with nothing but a screwdriver. A screwdriver is a nice tool, but when you turn it around and use it to drive nails, you’ll find that it leaves something to be desired. When you try to use it as a plane or an adze, you can’t help but be frustrated by its inadequacies, and when you need a saw, your screwdriver can’t even begin to do the job.

Blaming the screwdriver does not help. And deciding that screwdrivers are “another problem” and throwing away yours simply leaves you without a screwdriver when the time comes to drive (or remove) a screw.

(Of course, a company like that isn’t about to hire any QA testers, so you folks haven’t got the option of working for them. And I’m not a QA tester, I’m a developer. So the rest of my advice is pretty much aimed at fellow devs — but that doesn’t mean I don’t respect you QA folks. Seriously, y’all deserve a lot more respect than you get, and I love it when you make my life easier by finding my bugs for me.)

The skills, talents, and basic mindset that make a good developer are entirely different from the ones that make a good QA person. Asking one to do the other’s job is a mistake as fundamental as expecting graphic designers and accountants to swap places. Let me explain:

Developers hate repetition. We hate having to repeat anything more than once or twice; that’s why some of us become developers in the first place: because we can write programs that automate repetitive drudgery, and hence banish it from our lives.

It’s not just repetitive tasks, either. A good developer will also have a strong aversion to repetitive code and data. If two procedures differ by only one or two minor details, a good developer will combine them into one, and parameterize it, or otherwise figure out a way for the machine to make the decision. (A poor coder, by contrast, will produce “copy-and-paste code” that’s scorned by more advanced devs, because it’s more brittle as well as being less elegant.) If data must be stored somewhere, the prevailing wisdom is to keep it in just one place — “Don’t Repeat Yourself” is the maxim.

But a good QA checker has to deal with repetition all the time. It’s a fundamental part of their job. Run this test plan — this particular series of actions — over and over again, on each of the platforms we need to test on. Log every step of it, so that if something does go wrong, we can reproduce it.

Then, once the coder has checked in the changes that (hopefully) will fix the bugs you found… do it all over again. Just the exact same way. On every platform.

This is a coder’s nightmare. A developer’s natural response to this situation is to write some kind of automated QA-testing software. Of course, that software itself will then have to be checked for bugs before you could possibly rely on it… and you can’t trust your software to find its own bugs.

In the absence of such software, asking a developer to do bug-checking means asking them to do something that goes against their very nature. You can force them to do it, but they’ll never be really good at it.

Of course, all of that is totally leaving aside the obvious point that asking someone to find the flaws in their own work means asking them to see past their own mental blind spots — a developer’s test plan will be based on the same assumptions as their code, without the coder even realizing it. And that’s to say nothing of the obvious implications once you put the human ego into the picture. Human beings are flawed, and too many see “find the bugs in your own code” as equivalent to “show me all the ways you suck”.

Don’t ask a developer to do QA. Development and QA testing are fundamentally different disciplines, with completely different skill-sets and personality requirements. A developer can no more cover for a QA tester than a QA tester could fill in for a developer.

And if you’re a developer at a company that wants you to do your own bug-checking? Your employer thinks what you do is so similar to bug-checking that you’ll be good at it. They think they can cut corners on quality, and pay you one salary to do two jobs at once (including one job that it is in your nature to hate). If you can’t make them see reason, it’s time to brush up your résumé.

Because of the way JavaScript’s search() method works, you can do:

Try running this, and it will correctly claim that “Your URL is http://kai.mactane.org” — even though there’s a ! at the beginning of the test. What gives?

That exclamation point looks like it’s saying: “If the search for http://kai.mactane.org within the given string fails…” But what that test is really saying is: “If the search for http://kai.mactane.org returns Boolean false — or anything that evaluates as Boolean false…”

What that search() call actually returns is the character index at which the needle is found, or -1 if it isn’t found. Since the needle is found at the very beginning of the haystack — at character index 0 — search() returns zero. Which evaluates to false in a Boolean context.

If the needle isn’t found, the search will return -1 — which evaluates as true in a Boolean context!

Effectively, the ! operator is reconciling the disagreement between search()’s idea of falsehood and if()’s idea of falsehood: search() returns -1 for a false result (a failed match), but if() considers -1 to be true, not false.

This trick only works with needles that you are sure will only occur at the beginning of the haystack.

Once again, you should never, ever actually write code like this for any serious purpose or in anything you intend to deploy in the real world. The maintainability problems are not worth the amusement of confusing all your co-workers with a test that looks reversed. Instead, use a positional anchor in your regex, and explicitly test against -1…

…like any sensible JavaScript coder. (Yes, that last bit is intended to be ironic. I wrote something once about the silliness of having to add that extra test; I’ll have to see if I can find it and republish it here. )

So, following good software development processes, you first write a couple of tests:

UserTypeOne gets features A, B, and C
UserTypeOne does not get features D, E, or F
UserTypeTwo does not get features A, B, or C
UserTypeTwo gets features D, E, and F

Then you write some code that meets these tests, including creating a pair of mock users with the new flag set in both possible ways. You run the tests, and they all pass. Yay! Your new feature is ready!

Except that, of course, the moment you check this code in, you find that suddenly, nobody can log in any more. After all, the new flag is supposed to be mandatory, and that requirement is reflected in one of the changes you made to the login code. But the only users that have the NewFlag at all are the ones you just created in your fixture.

Sure, this will probably get caught before it goes live. Even if your entire QA department is asleep at the switch, if just one person in your organization tries to run even the most rudimentary tests, they’ll get back an error message. But “it didn’t get pushed to production” is only the most minimal level of success; I’d like to look at how you, the coder, could have prevented this problem. How could you keep this error from even getting checked in (and thus breaking the build and flat-lining your co-workers’ productivity for a few hours)?

The way a real QA department would spot this problem is regression testing. But it’s not reasonable to ask every coder to do a full suite of regression tests before every commit.

The real failure, of course, is that you made a change that you didn’t test. But saying “force programmers to write better tests” is not a viable answer; people are imperfect. You might as well mandate “developers may not check in any code that contains bugs” as one of your organization’s rules.

I don’t have a good answer to this problem. Asking developers to also be QA people is completely impractical. The current solution seems to be “developers sometimes check in code which breaks the build, and blocks people’s productivity for a while”. (Distributed SCM can help mitigate this.)

Is there something QA people know that developers don’t, that would make an easy solution that we could adopt?

Luckily, this time, Twitter seems to have updated its signup page with some nice AJAX that constrains the user’s options, and provides helpful feedback. So, for anyone else who needs this information in the future, here’s the scoop:

1. Letters, numbers, and underscores only. It’s case-blind, so you can enter hi_there, Hi_There, or HI_THERE and they’ll all work the same (and be treated as a single account).
2. There is apparently no minimum-length requirement; the user a exists on Twitter. Maximum length is 15 characters.
3. There is also no requirement that the name contain letters at all; the user 69 exists, as does a user whose name I can’t pronounce.

If you want a regex to match on this, /[a-zA-Z0-9_]{1,15}/ would be nice and safe for use in both POSIX and Perl-style regex syntax. (If you’ve got Perl-compatible regexes, /\w{1,15}/ is quick and easy.)

And yet, here I am with Hummingbird, which is totally dependent on Twitter’s bandwidth. (In my own defense, I can only point out that: a) I wrote it because I needed the functionality; and b) I’m not building a money-making company around Hummingbird. I’m just giving it away.)

In the past couple of weeks, I’ve noticed that Twitter can sometimes take an astonishingly long time to provide an update. Hummingbird works by requesting URLs like http://twitter.com/statuses/user_timeline/kmactane.xml?count=25, and (up until now) it worked on the assumption that Twitter couldn’t possibly take longer than 10 or 15 seconds to respond to such a request.

That turns out not to be the case.

In fact, my recent tests have shown that Twitter can sometimes take over 5 full minutes to finish responding to such a request. This is a problem, because the default installation of Hummingbird tries to update its cached data once every 5 minutes. And since I assumed the request would be fulfilled in, at most, 10% of that time, I didn’t bother building in any concurrency checking.

I’ve just completely reorganized most of Hummingbird’s architecture. The interface that a blog owner sees (in the sense of the configuration variables in hummingbird.php, and the calling syntax for blog pages and cron jobs) is still the same. However, all of the code for retrieving data from Twitter has been spun off into a new hummingbird-cache.php file, which can be launched into the background by the rest of Hummingbird, so that it can patiently take as long as it needs to in order to update the blog’s tweets cache.

The Major Change This Causes

In Hummingbird’s previous incarnations, it would freshen an out-of-date cache before displaying it. The assumption was that every once in a while, someone might have to wait a few more seconds before seeing your blog page. However, everyone would always see an up-to-date record of your tweets. (Where “up-to-date” means “no more than 5 minutes old, absolute tops”. If you tweet so relentlessly that that’s a problem, you probably don’t have enough time to keep up your blog…)

Now that we know that Twitter can take over 5 minutes to give you the data needed to freshen your tweets cache, that design is not acceptable. Instead, Hummingbird is now committed to showing the contents of your tweets cache as quickly as possible, and only then does it fork the cache-update process into the background.

This means that, if you’re not using cron or some other automatic job-scheduling facility to run Hummingbird every 5 minutes, visitors who arrive at your blog after a period of inactivity will see an out-of-date, stale listing of your tweets.

If you have a cron job set to keep Hummingbird fresh, you’ll be fine. (A page view will also trigger an update, so if you get so much traffic that there’s never a 5-minute period during which your page doesn’t get hit, you’ll also be fine.)

However, Livejournal is very customizable. It has 31 different “layouts”, each of which can then be further “themed” by application of CSS. This means that a user viewing a given journal, community, or single post, at a given time, may receive an HTML document containing any of 31 different DOM structures.

This means that finding a post’s author, or title, or even just a single post or comment, is not a straightforward process. I’ve set myself up a Ruby script that downloaded my “friends” view, a representative entry with lots of comments, and my own journal view, thus giving me a set of 93 fixtures that I can test against. The Ruby script also tweaked each fixture in the process of spooling it to my hard drive, by adding the proper calls to JSUnit files.

At the end of each document’s <head> element, there’s a call to jsUnitCore.js. Then, at the end of each <body>, I add a call to the lj-content-sieve.user.js script itself, as well as a set of test files that depends on which fixture this is. Every fixture gets an ljcs-global-tests.js call added to it — that file contains tests that should work anywhere, regardless of what sort of page you’re on.

Then all the “friends” page fixtures get an ljcs-friends-tests.js file, which tests operations that should happen on every friends page. For example, determining which entries need to be deleted. (In contrast, single-entry pages get ljcs-entry-tests.js, and the page from my own journal — which stands in for a view of a community — gets ljcs-self-tests.js.)

Finally, each fixture page gets a test file based on its layout: the “3 Column” layout gets ljcs-3 Column-tests.js, while “Cuteness Attack” gets ljcs-Cuteness Attack-tests.js. (Hey, I didn’t write or name the layouts; I just have to make sure LJCS works with all of ‘em.) These files will test that the actual DOM manipulations work properly.

Without test-driven development and automated testing to ensure that each layout and page-type is being handled properly, I don’t think this project would be manageable at all.

Do I really want to just write the whole thing, and then start profiling it to see where the hot spots are, and then possibly have to re-design the whole thing? That seems like the complete opposite of “work smarter, not harder”. Then again, it doesn’t matter if you write an O(n²) or even O(n!) algorithm if n is always going to be small… and in this application, I expect low-to-middling n values.

Of course, even if n will always be small, and increasing CPU power is my friend… even if it performs fast enough to make users happy, I’ll still know there’s a problem down there in the details. That may be the deciding factor.

So it was awfully tempting to say “screw this!” and just start writing code — you know, “code that works, code that gets stuff done”. Actual program logic, instead of testing tools. “Why don’t I just get something built to begin with,” I asked myself, “and then I can try to test that?”

Ha, ha. Of course, we all know that starting off without any tests just makes it easier to continue without them later. So I took the virtuous road, forced myself to get tests working, but allowed myself to skimp by only setting up fixtures for 3 of the 36 styles. The other styles are something I will have to go back and fill in before I can release, so there’s no chance I’ll “give myself permission” to blow them off.

When you really want to be writing code (because let’s face it, you consider yourself a coder, not a tester), it’s pretty annoying to write tests instead. But writing tests is at least a form of writing; if you’re not even writing tests, but rather setting up a test framework and fixtures, that’s almost excruciating.

But it all became worth it the moment I made a change in one of my basic data structures. I had a structure (what a Perlist would call a hash of arrays) that held information about how to identify which LJ style the page uses. Then I realized I didn’t want to have a separate structure for how to manipulate the page; instead, all information about LJ’s styles and DOM structure should live in one place.

So I altered the structure, then altered the functions created so far that rely on it. And that’s where I would then have to wonder, “How badly did I break everything?” Instead, I just ran my JSunit tests again. And seeing them pass was instant peace of mind. I don’t have to worry that there’s some hidden flaw waiting there, ready to be exposed by a user doing something unexpected. And as I add the other styles, I can easily be sure my code works for them, as well.

I’m still fairly early in the development of this code, and while unit testing has cost me a bit of time, it’s given me back peace of mind. The time profit? I have no doubt that will come later. (And had I made any errors in my data-structure change, the unit tests would have helped me find them more quickly than the usual debugging methods.)