My Coding Philosophy
I try to avoid “holy wars” about things like editors, operating systems, hardware platforms, coding practices, and so on. But I do have some fairly strong opinions about how to write good software, and what makes software good (or bad).
It all starts with utility. You write a piece of software to be useful. When someone tries to run your program, they won’t care whether you used emacs or vi, or whether you used K&R or Allman indent style, or what your politics are — all they care about is that your program works. Ideally, it should even make their lives easier somehow. If it’s no easier than whatever way the person was using before, then there’s no reason why they should use your program.
It shouldn’t just get the job done; it should get it done well, and be a joy to use.
Of course, usefulness is never in conflict with code elegance. In fact, I think one of the surest signs of elegance in a program — or a tool of any kind — is that its users go, “Wow! This thing really saves me time/effort/money/frustration/whatever!” This level of elegance is something that can be seen even without looking at the code. Google is one great example: even without reading their description of their ranking algorithm, users flock to Google in droves because: “It just works. You enter your search terms, and it finds the stuff you’re looking for.”
That’s the kind of utility I try to strive for in my creations. I think a Google-ish level of accuracy, usability, and life-enhancement is a great target to shoot for in any software project.
Make the Machine Do the Grunt Work
I don’t believe that humans should have to do any kind of repetitive, tedious activity that can be accomplished by a computer (or some other machine). Accordingly, I think it’s ridiculous to make a person jump through hoops to interact with your program. If the program has to jump through hoops to interact with its users, that’s okay; CPU cycles are cheap, while human time is not.
This attitude is displayed pretty well in the syntax for quiz files to be read by QuizMaster, my CGI quiz engine. (QuizMaster was a project I did back in 1998. It is now completely outdated and obsolete, and I’m not maintaining it or working on it any more. However, the quiz file syntax makes a variety of useful examples in this essay — especially as it illuminates my own coding practice — so I’ll refer to it a few times.)
The computer is not about to form a labor union and complain of overwork. Make it do the work, not people!
QuizMaster’s quiz file syntax includes a variety of free-form text values, such as QuestionText and ExtraText, and the type definitions on these say things like: “As many lines as desired of free-form text, including optional comment lines. QuizMaster will automatically end the value at the next command.”
This sounds pretty blithe in plain English — and it’s supposed to; this is one of the things that happens when you write the docs first (which I discuss at greater length below). But it actually hides a fair bit of complexity, because getting QuizMaster to take such free-form input requires it to go through a minor nightmare in the process of parsing a quiz file.
So what? The Perl interpreter isn’t about to complain that it’s having nightmares, or that I’m making it work too hard. It likes to work hard. QuizMaster isn’t about to agitate for a lighter work-load, or paid time off. But if I’d required users to remember to terminate their free-form texts with some randomly chosen special character, the users would be sure to complain when they kept forgetting to use it. (And it wouldn’t be available for use in the texts themselves, and — most importantly — the users would rightly see that doing it that way was a crock and a kludge, caused by the programmer’s laziness.)
Sure, I also had to go through a minor nightmare once, to write the parsing routines in the first place (and then a second time, during debugging). But that’s okay — that means the work got done once, the right way, and could then be considered over-and-done-with, rather than forcing everybody else to do work every time they wanted to interact with QuizMaster.
Sometimes, of course, you need to require a certain amount of precision from your users, to specify their intent precisely enough for the computer to figure out what they want. But if you can possibly get the computer to grok it for them, do so! And, in general, the more technical your users are, the more precision you can ask them to supply. (If your users are typical end-users working in a GUI, of course, you can always guide them into providing the required precision by using appropriate controls and text on your dialog boxes.)
But even if your users are system administrators, there are limits to how far you can go in requiring them to talk to your program in its own language. Sendmail is a wonderful example of how not to write a configuration interface. Even sendmail.cf isn’t really the true configuration file; for that, you have to look into m4 (and honestly, neither of the two is really human-readable). The moment Sendmail’s authors found themselves creating a second-level abstraction that parsed m4 into sendmail.cf, they should have known, beyond a shadow of a doubt, that it was getting too complex. (And they should have had strong suspicions long before that.) Luckily, more modern MTAs such as Qmail and Postfix have much clearer configuration sytems... and isn’t it funny how people have been switching away from Sendmail?
Not only does Sendmail’s awkwardness and user-hostility make it the target of endless complaints and invective by anyone who has to deal with it (and having users complain about having to use your program is never a good sign!), it also results in rampant misconfiguration issues. These invariably result in lost or misdirected mail, open relays or otherwise insecure servers, and even causes administrators to avoid updating their configurations, just to keep from breaking anything. The severe costs of giving Sendmail an arcane, impenetrable configuration syntax have had very real effects in the real world, and far outweigh any time savings for the developers. It’s a terrible mistake, and well worth learning from when designing your own configuration and command syntax.
Reading the User’s Mind
If you try to make your program (and hence the computer it’s running on) think like a human, your program’s users won’t have to try to think like a computer just to talk to your program. They’ll be happier, and they’ll be better able to communicate their desires to your program, so that it can better serve them.
My usual trick in getting a computer to figure something out is to ask myself, “Well, how do humans figure that out?” It works surprisingly well. For example, I once had to write an MS Word macro to tidy up formatting of text copy-pasted from Web pages. In the pasted text, things sometimes came out double-spaced; there were always paragraph breaks after every line (instead of just at the ends of paragraphs). Some sites distinguished paragraphs from each other by an extra line-break; others used a first-line indent. Sometimes the text would all be indented by a certain number of spaces (because of HTML margin settings); other times, only some of the text would be indented (where it had wrapped around images on the original Web page).
One of my greatest dilemmas was that of distinguishing a first-line paragraph indent from an overall page indent. When I asked myself “How do humans do it?”, I started looking at text all around me, and quickly saw that we distinguish new paragraphs by their deviation from the previous text — if all the text on the page is indented by five spaces, then the few lines that are indented ten spaces must be initial lines of new paragraphs. From there, it was a simple task to do a few statistical scans on the text at the beginning of the macro’s execution, to find out what the “baseline” parameters were, and then have it start looking for deviations from that baseline.
Making the computer think like a human is a lot of work, but it’s fun work, and work that only needs to be done once — thus freeing your users from having to do work every time they use your program.
Write the Docs First
Note, added December, 2008: This essay was written in early 2004, before I'd had any exposure to Agile practices or iterative development. The views expressed in the following section no longer reflect my current thinking on software creation. I'm leaving it as a snapshot of my views at the time. In particular, I still stand by many of my comments about what constitutes an interface, and what sort of thinking a developer should do about adapting the interface to the users' needs.
Ever since I first heard of it, I’ve had a fascination with the idea of writing the documentation before the code. Since I first tried it, I’ve made it a firm part of my programming practice, and I’ve found that it brings a variety of powerful benefits:
- The docs actually get written.
- Your interface is better adapted to your users (and so is everything else about your program!).
- The docs can then serve as a framework and a specification for you to write your code from, making the actual writing easier.
These concepts are all too involved to be dealt with in just one or two sentences each, so let me expand on them all.
The Docs Actually Get Written
This might better be stated, “the docs don’t get completely shafted”. All too often, when someone tries to document their own project after they’ve finished the coding (and debugging!), the author is simply worn out. (At least, I know I am.) At that point, writing the docs just feels like another piece of drudge-work, one that can often be intolerable. Is it any wonder that docs like that — the docs stereotypically produced by coders for their own projects — are terse, cryptic, and ultimately unhelpful?
If you write the docs at the beginning of the project, though, you’re riding on that initial wave of adrenalized joy-in-creation. You can take the time to write them right, pouring into them all your enthusiasm for the software you’re about to create. Including more details isn’t a chore; it’s an advertisement, a promise to yourself and your users: “Yes, I will include this, as well!”
Interface and Program Better Adapted to Users
Every piece of software has its intended “target audience”. For a programming API, the audience is other programmers. For a system daemon, the intended users are sysadmins. Then there are CGI scripts, intended for Web developers, and there are programs intended for standard end-users, such as office suites, and email and chat clients.
When you start writing your docs, the simple act of choosing a basic tone, style, and jargon level for your docs implies a choice about who your target users are. A Unix man page (such as you’d include with your system daemon) has a very different layout from the Windows Help file you’d include with your email program — everything, from the basic structure to the writing style and vocabulary level used, is designed to help communicate with a different set of users.
Put simply: writing your documentation, when done with a bit of mindfulness, forces you to consider your users. No longer can you simply whip up whatever interface you like and force them to use it. You have to think about their needs and what will serve them best. Which is as it should be. If you find yourself writing, just for example: “The values in the config file must be terminated by such-and-so character” (aside from newline), that should be a sign that you’re making things too hard on the user.
That was a mistake I found myself making in the QuizMaster docs: I started telling users to put special characters at the ends of their texts. I read it back to myself, and it sounded ugly. That was when I realized I had to make QuizMaster smart enough to understand free-form texts — and started asking myself, “Well, how would I figure out where a free-form text ends?” (The answer turned out to be: “When the next line begins with any command word,” which was easily amenable to beginning-of-line regex searching.) This is also the sort of mistake that Sendmail’s authors made (and then compounded repeatedly), but didn’t see as a mistake — hence the unwieldy and torturous m4 configuration that I talked about earlier.
A user reading your docs should see a set of instructions that make simple, intuitive sense, not a bunch of strange, arcane edicts and strictures. If your docs are simple and make sense, then the interface they’re describing will — almost perforce — be simple and sensible as well.
An interface is any way that your users interact with your program. Even the choice of -f <config file name> or -c <config file name> as a command-line switch is an interface design decision!
And when I say “interface” here, I mean “every way that the user interacts with the program”. This goes far beyond simply “CLI or GUI”; consider a Unix daemon such as crond, sshd, sendmail or named. It’s started on the command line, possibly with some kind of arguments. It might also read one or more config files (perhaps specified by a -f or --config option), and these files undoubtedly require some specific syntax for their internal structure. This daemon should respond to standard IPC signals like HUP, INT, ALRM or USR1 (to say nothing of TERM). It might even read input of some sort on STDIN, or write to some standard file or files.
All of these things are part of the program’s interface: the command-line arguments, the config file syntax, the location and names of files used, the specific names and mnemonics of command-line arguments — all of it. These are how the user (in this case, most likely a sysadmin) relates to your program.
Docs Serve as Framework and Spec
Here’s the secret that exposes the problem with the entire concept of a dichotomy between coding and documenting: writing docs is not a separate activity from coding, and it’s certainly not the “waste of good coding time” that some people make it out to be. It’s an integral part of the coding process, one that can make your code — from the overall high-level design and architecture, right down to the line-by-line details — cleaner, tighter, more efficient, more robust, and more elegant.
Writing those initial docs is your first opportunity to clarify your own thoughts about your project. Use it! Don’t just ask yourself who your target audience is, and what kind of interface will best serve them. Ask yourself how your program will manipulate and process the data it gets through that interface. For example, suppose your config file syntax includes a “foo” value. What happens if the config file contains lines “foo=bar” and “foo=baz”? Does the first line take precedence? The last one? Does the program notice the problem while parsing the config file, and generate an error message? Or might it be legitimate for foo to have more than one value in this context? (In which case, maybe it should be a Perl array instead of a scalar, or a C struct of some sort rather than a char[]...)
These are the kinds of questions you’ll need to ask yourself anyway, when you go to write your config file parsing routines. But if you ask them now, you can put them in the docs, so that your users know exactly what behavior to expect.
Not only that, but you’ll have specified in advance exactly how the program is supposed to deal with inputs of all types. Half your coding will essentially be done, and you’ll be able to write from your own docs, using them as a specification.
Time spent writing good docs before coding is never wasted.
A Final Point on Docs Before Coding
Since I already know I’ll be writing my docs before I do any coding — especially on any project of any size or complexity — I can simply have the docs be part of my proposal to the client. Clients love it; they’re often a bit bewildered at first to receive such detailed and thorough proposals, but after the initial raised eyebrow, they realize that it’s a great way of ensuring that they and I agree on what’s to be done.
It also gives me a good way to gauge how long the project will take, and that becomes part of my schedule and cost estimate. By documenting things before I start, everybody wins.
Object Oriented “Versus” Procedural Programming
Object-oriented programming used to be all the rage (as in, “practically mandatory”) and it does have great utility. It’s especially valuable for encapsulating data in large projects — indeed, it becomes more and more essential as the project’s size grows. But it’s not the only way to do things. Java is notable for forcing object orientation on everything; even a “Hello, world!” applet has to be its own class.
But smaller projects don’t require the kind of overhead involved in objectifying everything. There’s still a place for procedural programming, and this is one reason why Perl is still favored for rapid design and prototype development — it doesn’t force you to choose one style over another.
Most crucially, it should be understood that OO and procedural styles are not in opposition. Using one for some things doesn’t mean you can’t use the other for other tasks. I can’t recall how many times I’ve written a short, basically procedural, Perl script that makes use of an object-oriented module from CPAN.
Indeed, Eric Raymond made the cogent point once that “the structured-programming wars eventually blew over with the realization that both sides were wrong” (“considered harmful” Jargon File entry). Luckily, more and more coders seem to be realizing that insisting on one programming style is throwing away too many valuable tools. (And hopefully, the next generation of programmers will have learned from both the structured-programming wars and the OOP overenthusiasm the next time a new programming paradigm starts making the rounds... I note with some optimism that “extreme programming” is not becoming nearly as much of a fad as structured or OO programming were, though that may be partly because the name sounds like an MTV marketing buzzword.)
Indent Style and Tabs
I’m somewhat agnostic about brace style — I used to prefer Allman style, because I tended not to notice the opening curly brace in K&R format. Somewhere along the way, after much exposure to K&R, I got used to it and now prefer it. So I can switch back and forth from one to the other pretty easily. I also use a four-space indent. But I consider these to be mere stylistic matters, not things that should ever be carved in stone. (Indeed, the concept of “One True Brace Style” makes me laugh, regardless of what particular style it might be applied to.)
Of more importance, to my mind, is the idea that the indentation should be done with tabs rather than spaces. This allows anyone reading the code to easily change their tab width setting to whatever value they most prefer. And above all, tabs and spaces should not be mixed! Otherwise, the result of changing the tab width is utter visual chaos. (If I recall correctly, I acquired this part of my philosophy while hacking on Majordomo code. Functional, but visually rather messy.)
CGI Output
Note, added December, 2008: The following section was getting somewhat obsolete even when I wrote it, nearly five years ago. It is now completely irrelevant to modern coding practices, which is a good thing for all concerned.
For a while now, I’ve been convinced that the output from CGI programs should not be hard-coded into the programs themselves. Doing that sort of thing makes it very hard for a non-programmer to tweak the appearance of the output. And part of the definition of a CGI program is that it will be used on a Web site, where it should be tweakable to match the style, look, and feel of the rest of the site.
Of course, the best way out of this dilemma is a hypertext preprocessor, such as PHP or HTML::Mason. But sometimes you may need a CGI script, for historical, architectural, or other reasons. This part of the essay is focused on that situation, because it shows some of my thinking about how different types of data should be handled.
It’s pretty unreasonable to require a programmer to grovel through the code, looking for hard-coded HTML sections in print statements, just to make the CGI script’s output match up with everything else on the site. In fact, for most organizations, it would actually be much more time-efficient and sensible to have the Web designers handle this sort of thing. But if the output statements are buried in program code, they’ll be very annoying for the designers to find and work with.
Don’t make Web designers read through Perl, Java, C or even shell code. And don’t make programmers write HTML.
My solution to this problem is usually to segregate the output into its own HTML files, completely separate from the program file itself. You can see this architecture at work in mlm-command, where all output is performed by spooling a given “display file” to STDOUT. Exactly which one depends on the DisplayFile value passed in from the calling form, which provides even greater flexibility. Rather than just spooling the file statically, the error_msg subroutine scans through the file looking for any DisplayFile tokens to substitute values into, providing a system that combines the power of allowing the script to say what it needs to with the flexibility of allowing anyone who speaks HTML to easily tweak the formatting of the script’s output. The mlm-command installation package comes with a variety of sample output-file sets, just to try to display a bit of that power and get users’ minds thinking about what else they could do with it.
Indeed, I’ve found that error_msg routine so useful that I renamed it HTMLspew and have since incorporated it into a variety of one-off projects and works in progress.
But that’s not always the best way to do things. (Indeed, I’m firmly against getting too canalized; the solution should always fit the problem at hand, not “what I did last time”.) For QuizMaster, this option wouldn’t have worked; QuizMaster itself generates entirely too much of the HTML, based on the specifics of each question in the quiz file. Instead, I gave users the ability to tweak various <body> and <font> tag attributes, to generally match up with the rest of a site’s color scheme... then left it at that.
Please note that I wrote QuizMaster back around 1998, when tags like <font> were officially deprecated, but CSS compliance in browsers was a joke, and there really were no better options. If I were writing QuizMaster today, the entire output system would be completely different, and would probably involve at least one option to place a given stylesheet <link> tag in every generated <head> section.
CGI Security
Note, added December, 2008: In the following section, rather than "CGI" or "CGI script", please read "any network-accessible software". With such a substitution, the basic thrust of the argument is still relevant.
All CGI authors should also keep in mind that their creations are going to be deployed on the Internet — that is, in a nasty, hostile environment, where thousands of crackers and script kiddies are no more than 500 ms away. All input should be checked, not just for the possibility that some user made a typo or forgot to fill in a field, but for the possibility of pathological and malicious entries. Non-standard characters should be stripped from the input.
And, though it’s not really an issue with Perl, there are languages used for CGI scripts for which it must be said: there is no excuse for buffer overflows! It’s very easy, when writing your program (and confronting the horror of doing proper dynamic memory allocation in C), to simply say, “No user is going to need more than 15 digits to put a phone number into” and declare int[15] phoneNum;. But there’s a hidden assumption in that sentence: No normal, legitimate user is going to need more than 15 digits. What about the abnormal, illegitimate users? It’s not enough to consider what a normal user needs when your program is open to attack by the entire Internet; you have to consider what will happen when someone with too much time on his hands decides to feed 4K of “interesting” control characters into your 15-byte buffer.
This doesn’t mean Perl writers can just slack off; either. If you’re writing in Perl, there is equally well no excuse for writing any script that takes network input without the taint and warning flags on, and usually with use strict enabled as well. This won’t stop every possible problem, of course, but it will go a long way toward getting you started on thinking about security.
My caveat about use strict is that I — like any programmer facing a non-trivial problem — sometimes write code that makes heavy use of references. It can easily get to the point where it makes more sense to start the script with “use strict 'vars'; use strict 'subs';” and leave it at that than to keep toggling 'refs' off and on all the time.
Naturally, if you’re using taint checking, you should actually consider what input is legitimate, instead of just blindly untainting everything with $var =~ /^(.*)$/; If you do that kind of thing, there’s nothing I or anyone else can say that will get the point across.
In PHP, there isn’t (to the best of my knowledge) any system analogous to Perl’s taint checking. But writing your code as if you had taint checking turned on is still a great start.
Conclusion
Software should be written for the people who will use it. It shouldn’t be designed to make things easy for the computer, or even for the programmer. Instead, the programmer should put time, effort, and brainpower into figuring out how to make the program as useful and usable as possible.
All the functionality in the world is no good unless people can actually make use of it. The entire point of computers — like any other technology — is to make life easier. Not to force them into new, unwieldy ways of doing things; not to show off one’s coding prowess; not to fill out your résumé. (It may also fulfill either or both of the last two points, but these should never be the primary purpose of your code!)
Finally, a point that Eric Raymond makes in The Cathedral and the Bazaar, and one that’s often served as a guide for me, in recognizing good design and architecture: “Any tool should be useful in the expected way, but a truly great tool lends itself to uses you never expected.” (From Chapter 7, “Fetchmail Grows Up”) I’ve found that my better tools tend to surprise me by becoming useful in ways I hadn’t expected or planned for.