A lot of people have been, effectively, pointing and laughing at Stoll’s failed prediction. I’d rather consider it a cautionary tale: The man who was so totally wrong wasn’t just a random pundit who didn’t know what he was talking about. He was Clifford Stoll — author of The Cuckoo’s Egg, a man who had been online for 20 years at a time when most people were just beginning to hear that there was a such thing as the World-Wide Web, and the man who traced German cracker Markus Hess through umpteen layers of insecure computer systems and networks.

In short, the man knew what he was talking about. He wasn’t a Senator Ted Stevens. If he could be so wrong, how much faith can I place in my own predictions about where the Internet’s headed?

But wait, there’s more — how wrong was he? Sure, Stoll claims that “no online database will replace your daily newspaper”, which has turned out to be completely false. But how about “no CD-ROM can take the place of a competent teacher”? No matter how interactive the CD, it can’t substitute for a good human teacher’s ability to guide and nurture a student’s intellect. (Not without AI, which is still at least 20 years away — just like it’s been for the past 50 years.)

And maybe you think it’s obvious that CD-ROMs can’t replace real teachers. But there have been, and there still are, people who claim their CDs are just effective as face-to-face teaching methods — or even more effective.

Stoll gripes about the problems of Usenet. Okay, the main bulk of Internet conversation has moved to blogs and comment threads, but his words still apply: “Every voice can be heard cheaply and instantly. The result? Every voice is heard. The cacophony more closely resembles citizens band radio, complete with handles, harassment, and anonymous threats.”* There’s a reason why many of my friends say, “Never waste your time reading the comments.”

I’ve written before about the effects of “harassment and anonymous threats”. That was just last year, and I doubt that all the anonymous threateners have suddenly left the Internet.

It’s very easy to look at Stoll’s rant and get distracted by the petty details: “Usenet? Hah! How archaic!” But that’s just a way of trying to remain comfortable, and ignoring the ways in which things haven’t changed one bit. Because the problems of anonymity aren’t technological problems; they’re problems of human nature.

But even in the places where Stoll was wrong — demonstrably, ridiculously wrong — it does me no good to simply point and laugh. Instead, I’d rather ask myself, “Could I have done any better?” Stoll claimed that online shopping would never take off:

[Internet hucksters promise that] We’ll order airline tickets over the network, make restaurant reservations and negotiate sales contracts. Stores will become obsolete. So how come my local mall does more business in an afternoon than the entire Internet handles in a month? Even if there were a trustworthy way to send money over the Internet — which there isn’t — the network is missing a most essential ingredient of capitalism: salespeople.

He’s wrong on almost every particular; the only one that might possibly be an exception is “negotiat[ing] sales contracts”. Aside from that? I’ve lost track of how many restaurant reservations I’ve made online and how many airplane tickets I’ve bought online; over the past ten years, I’m quite positive that I’ve done those things far more often online than by “traditional” methods. Even if you don’t consider PayPal to be quite trustworthy, online credit-card transactions are now safe and secure.

And salespeople? Most of the time, I can do without them. I can’t help but remember the last time my girlfriend and I visited a Victoria’s Secret; after being approached by 5 different salespeople in as many minutes, we left the store in frustration at being interrupted and distracted so much — to the point that we couldn’t even browse the merchandise in peace!

Great, so Stoll was totally wrong about online shopping. If you’d asked me about it in 1995, what would I have said? Would my predictions have been any better?

And more to the point: What am I predicting right now? And how wrong am I? And how can I learn from Stoll’s mistakes — or my own — and make better predictions?

That’s the real take-away from Stoll’s article. And those are question I don’t have the answers for yet. (If you’ve got answers, leave a comment and tell me!)

* All quotes from Newsweek edited to fix simple spelling mistakes.

It might be a welcome trend, if the replacement widgets people were building had more functionality than the OS-native ones that are available for free in any other context. But usually, the widgets I see in these frameworks have less than half the functionality of the things they try to replace. I’m going to pick on scroll bars for now, because I’ve seen them horribly mangled too many times.

To start with one aspect of scroll bars that we often take for granted: Can you guess which of these text boxes contains more text?

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Quisque est est, luctus quis suscipit et, lacinia ac odio.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco.



Of course you can; you don’t even need to guess. The one on the left has more, and you can tell immediately because the movable slider on its scrollbar is smaller. This is a feature you get for free from the native OS widget, and one I have never seen implemented in a Flash, AIR, or similar interface.

Here are all the things a native scrollbar does. I absolutely guarantee you, if you’ve reinvented a scrollbar, your implementation is missing at least two of these features:

1. As mentioned above, the slider object changes size automatically, to give a general impression of how much content the scrollable area contains.
2. The arrows at the top and bottom of the scrollbar are clickable, and clicking them once will move the content up and down by a small amount (typically 1 or 2 lines).
3. Clicking and holding the top and bottom arrows will cause the content to scroll more rapidly.
4. Clicking and dragging on the slider will scroll the content smoothly to any position the user chooses. (Yes, I have seen scrollbar implementations in which the slider was not a draggable item!)
5. Scroll wheel support: If the user’s hardware includes a scroll wheel, scrolling it will move the content up and down. This is many users’ preferred method of scrolling content these days, and yet it’s totally ignored by over half the Flash interfaces I’ve encountered in the past year. I’ve bolded it because it’s the one that gets so egregiously ignored.
6. The “gutter” in between the arrows and slider is also clickable. Clicking on it once will scroll the content up or down by, roughly, the size of the visible area (a “page”, as it were).
7. Of course, clicking and holding in the “gutter” area will repeatedly and rapidly scroll the content. (This is rarely apparent unless you’re dealing with a tall scrollable area and a fairly long document. Try loading a 25-page or longer document into a word processing program, and set the program to full-screen.)
8. When the content is scrolled (by whatever method), the slider will move up or down to indicate how far from the beginning or end you are. (Essentially the converse of item #4: Moving the slider scrolls the content, and scrolling through the content moves the slider.)
9. The arrow buttons and the gutter area give a visual indication when they’ve been clicked, so the user has better feedback. (This is one of those little things that makes the user experience so much better. It’d be so easy to do, and yet I’ve seen about 2 Flash scrollbars in my life that have bothered with that feature.)
10. If you set your cursor focus on the scrollable area, using the up and down arrow keys will scroll the content. Either the Home and End or Ctrl+Home and Ctrl+End keystrokes will move directly to the beginning and end of the content. The PgUp and PgDown keys may also move the content by an amount similar to clicking in the gutter.
11. The scrollbar — including its arrows, gutter and slider — looks just like every other scrollbar on the user’s computer. The user can immediately identify the items and finds them familiar.

Even if you have done a smashing job of replicating the first 10 points on this list, you simply can’t get the final one in an AIR, Silverlight, or Flash interface. If your whiz-bang interface platform provides some other benefit, then dropping that last feature might be worth it.

But it’s no excuse for dropping all the other stuff. Most non-native scroll bars I’ve seen support only 3 or 4 of these features, which means anyone who normally uses any other ways of interacting with scrollbars winds up cursing and deciding that this broken implementation is cheap and chintzy.

Which it is. If you’re going to reimplement the wheel, at least make it something that will roll smoothly.

So, following good software development processes, you first write a couple of tests:

UserTypeOne gets features A, B, and C
UserTypeOne does not get features D, E, or F
UserTypeTwo does not get features A, B, or C
UserTypeTwo gets features D, E, and F

Then you write some code that meets these tests, including creating a pair of mock users with the new flag set in both possible ways. You run the tests, and they all pass. Yay! Your new feature is ready!

Except that, of course, the moment you check this code in, you find that suddenly, nobody can log in any more. After all, the new flag is supposed to be mandatory, and that requirement is reflected in one of the changes you made to the login code. But the only users that have the NewFlag at all are the ones you just created in your fixture.

Sure, this will probably get caught before it goes live. Even if your entire QA department is asleep at the switch, if just one person in your organization tries to run even the most rudimentary tests, they’ll get back an error message. But “it didn’t get pushed to production” is only the most minimal level of success; I’d like to look at how you, the coder, could have prevented this problem. How could you keep this error from even getting checked in (and thus breaking the build and flat-lining your co-workers’ productivity for a few hours)?

The way a real QA department would spot this problem is regression testing. But it’s not reasonable to ask every coder to do a full suite of regression tests before every commit.

The real failure, of course, is that you made a change that you didn’t test. But saying “force programmers to write better tests” is not a viable answer; people are imperfect. You might as well mandate “developers may not check in any code that contains bugs” as one of your organization’s rules.

I don’t have a good answer to this problem. Asking developers to also be QA people is completely impractical. The current solution seems to be “developers sometimes check in code which breaks the build, and blocks people’s productivity for a while”. (Distributed SCM can help mitigate this.)

Is there something QA people know that developers don’t, that would make an easy solution that we could adopt?

However, Livejournal is very customizable. It has 31 different “layouts”, each of which can then be further “themed” by application of CSS. This means that a user viewing a given journal, community, or single post, at a given time, may receive an HTML document containing any of 31 different DOM structures.

This means that finding a post’s author, or title, or even just a single post or comment, is not a straightforward process. I’ve set myself up a Ruby script that downloaded my “friends” view, a representative entry with lots of comments, and my own journal view, thus giving me a set of 93 fixtures that I can test against. The Ruby script also tweaked each fixture in the process of spooling it to my hard drive, by adding the proper calls to JSUnit files.

At the end of each document’s <head> element, there’s a call to jsUnitCore.js. Then, at the end of each <body>, I add a call to the lj-content-sieve.user.js script itself, as well as a set of test files that depends on which fixture this is. Every fixture gets an ljcs-global-tests.js call added to it — that file contains tests that should work anywhere, regardless of what sort of page you’re on.

Then all the “friends” page fixtures get an ljcs-friends-tests.js file, which tests operations that should happen on every friends page. For example, determining which entries need to be deleted. (In contrast, single-entry pages get ljcs-entry-tests.js, and the page from my own journal — which stands in for a view of a community — gets ljcs-self-tests.js.)

Finally, each fixture page gets a test file based on its layout: the “3 Column” layout gets ljcs-3 Column-tests.js, while “Cuteness Attack” gets ljcs-Cuteness Attack-tests.js. (Hey, I didn’t write or name the layouts; I just have to make sure LJCS works with all of ‘em.) These files will test that the actual DOM manipulations work properly.

Without test-driven development and automated testing to ensure that each layout and page-type is being handled properly, I don’t think this project would be manageable at all.

Do I really want to just write the whole thing, and then start profiling it to see where the hot spots are, and then possibly have to re-design the whole thing? That seems like the complete opposite of “work smarter, not harder”. Then again, it doesn’t matter if you write an O(n²) or even O(n!) algorithm if n is always going to be small… and in this application, I expect low-to-middling n values.

Of course, even if n will always be small, and increasing CPU power is my friend… even if it performs fast enough to make users happy, I’ll still know there’s a problem down there in the details. That may be the deciding factor.

So it was awfully tempting to say “screw this!” and just start writing code — you know, “code that works, code that gets stuff done”. Actual program logic, instead of testing tools. “Why don’t I just get something built to begin with,” I asked myself, “and then I can try to test that?”

Ha, ha. Of course, we all know that starting off without any tests just makes it easier to continue without them later. So I took the virtuous road, forced myself to get tests working, but allowed myself to skimp by only setting up fixtures for 3 of the 36 styles. The other styles are something I will have to go back and fill in before I can release, so there’s no chance I’ll “give myself permission” to blow them off.

When you really want to be writing code (because let’s face it, you consider yourself a coder, not a tester), it’s pretty annoying to write tests instead. But writing tests is at least a form of writing; if you’re not even writing tests, but rather setting up a test framework and fixtures, that’s almost excruciating.

But it all became worth it the moment I made a change in one of my basic data structures. I had a structure (what a Perlist would call a hash of arrays) that held information about how to identify which LJ style the page uses. Then I realized I didn’t want to have a separate structure for how to manipulate the page; instead, all information about LJ’s styles and DOM structure should live in one place.

So I altered the structure, then altered the functions created so far that rely on it. And that’s where I would then have to wonder, “How badly did I break everything?” Instead, I just ran my JSunit tests again. And seeing them pass was instant peace of mind. I don’t have to worry that there’s some hidden flaw waiting there, ready to be exposed by a user doing something unexpected. And as I add the other styles, I can easily be sure my code works for them, as well.

I’m still fairly early in the development of this code, and while unit testing has cost me a bit of time, it’s given me back peace of mind. The time profit? I have no doubt that will come later. (And had I made any errors in my data-structure change, the unit tests would have helped me find them more quickly than the usual debugging methods.)