Scott Leberknight's Weblog

This year has gone by so fast it's not even funny. It's been a great year, with the best part being the birth of my first daughter, Sarah, and all the good things that come along with that. Hope all of you have a great New Year's and that the new year brings good things.

I came across this article on IT World in one of the emails I get from them. It reminded me of something I think about a lot, which is that writing software is not much different than writing in general - that is, writing books, articles, presentations, etc. When you write a book, an article, a presentation, a blog entry, or compose even a short email, you are constantly rewording, reworking, restructuring your words...in software parlance you are refactoring your prose. I am doing it right now as I write this entry.

Why then, do so many people in the software industry assume that software is exempt from this reworking and restructuring, and that by "proper design" you can somehow avoid rework and restructuring? This problem is mostly found in managers and other like-minded people, who want to make sure you "get it right" the first time so there is no going back to any code you've already written. They want to check off the task in their project management tool that says your domain objects are done, and then make sure you don't touch them ever again because to do so would throw off the schedule and their ability to manage. Interestingly, most of these people would never release project documentation without it going through multiple reviews and revisions. Sadly, they fail to see how this is the same as reviewing and revising code as you write it and add more and more functionality throughout the development of a system.

I love IDEs. They make developing software much more enjoyable and productive. But how much work do you let your IDE do? At what point are you fighting the IDE instead of being productive? Every IDE and tool you'll ever use will have some things you wish it would do or do differently. But there are certain things that I am not willing to let my IDE do for me. I think probably the most important thing is builds. Certainly in Java the de facto build tool for all sorts of Java applications is Apache Ant, but every single IDE out there has its own internal build tool and process. Most times you end up clicking through more than a few screens to configure it just right, and then find out it is really difficult or impossible to share all those settings or to easily view what the build process and settings actually are! Even worse, once you've got your IDE's build process mastered, you find you are locked into that IDE. So there is no way to share your build process with other team members who are not using the same IDE as you are.

Many organizations "solve" this problem by mandating the same IDE for all developers. I won't get into that argument, but I'll just say that I think there are positives and negatives to that approach. So, back to the build problem. Assuming all developers have the same IDE, is it a good idea to use their build configuration? I don't think so because the IDE build process is typically neither flexible nor transparent. Once you get beyond simple applications and into the arena where your build must be able to target multiple environments (e.g. development, test, production, etc.), IDE build processes quickly break down.

A simple but very relevant example is Ant's <filterset>, which I use all the time to replace tokens for different build environments among other things. To illustrate, most teams provide local instances of a database for each developer, one or more test databases where integration testing and user testing occurs, and a production database. Each of these databases is probably going to have a different URL, username, password, etc. Using the Ant <filterset> you can very easily replace tokens during the build with specific values for the target environment. Ant makes this very, very easy. You could simply place the varying properties in different properties files, e.g. dev.properties, test.properties, and prod.properties, and then use a -D property to determine which environment to build, e.g. ant -Dbuild.env=test would build the application for the test environment. Whereas this is very simple in Ant, IDEs I've used don't provide this type of customization. Why not? It seems very narrow for an IDE to assume there is only one target deployment environment.

There are other things I don't want my IDE doing for me as well, but there are of course a ton of things I really want my IDE to do for me, like code generation of getters and setters, refactoring, and being able to run unit tests within the IDE while I'm developing to get a really fast build, test, build iterative cycle going. Mainly those are things that are similar among IDEs but might have a different menu command or keystroke. IntelliJ, Eclipse, and JBuilder all can generate my getters and setters for me just fine, so what is the differentiator between IDEs? I think for me it is to what extent the IDE helps me versus hinders me, and what things it does that automate things I do all the time. I can be productive in any of those three aforementioned IDEs, but I prefer IntelliJ because it seems like it is more of an extension of me than a tool I am using. It just feels "smooth". But still, I won't let even IntelliJ do my builds for me!

There are some other relatively minor things I might avoid relying on an IDE for, but overall builds seem to be the most important aspect of development to keep out of your IDE's reach, regardless of whether you are developing in Java or something else.

Does you organization have a test department that still use a gigantic Word document or Excel spreadsheet to document all paths through a system and makes its human testers manually click through everything on a web site or rich client application? Several people I work with mentioned that on a recent project they used a test department that demanded a complete script of what buttons they need to click, what data to enter, in what order, and what results to expect. They then set the testers loose. Apparently all these testers did was to follow the "happy trail" and magically the application passed with flying colors! Of course the reality was much uglier, since they really didn't exercise the application either in terms of executing all potential paths thought the user interface, entering erroneous data, or load testing the application.

I submit that any testing team that insists on having humans manually click through a scripted session and manually type in the results is absurd, a waste of valuable time and money, and ultimately adds zero value to the software development effort and probably adds negative value.

There are tools out there to automate many of these repetitive tasks. The only one I ever worked with was Rational's Robot product, and that was back in 2001. It allowed you to record the interactions with an application and perform assertions on the results of those interactions. That allows you to automate regression tests in the same way developers can re-run all their JUnit tests at the touch of a button. From a brief visit to the Robot web site it appears to be able to do a lot more now, including web applications. I also found this article about using the Rational Robot automation framework support (RRAFS) and another link off that page to the Software Automation Framework Support project on Sourceforge.

In any case, please do your company a favor and let someone know there are lots of tools to automate many aspects of functional testing!

Ok, since I've been slacking so much since the November 5-7 No Fluff Just Stuff conference, I am going to write about the last five sessions I attended in this single entry. Each session gets one (hopefully short) paragraph.

Howard Lewis Ship, creator of Tapestry and HiveMind, gave an introductory session on HiveMind. This is another Dependency Injection/Inversion of Control container similar to Spring. Looks pretty interesting, especially the ability to configure separate modules and give them versions. It also contains the capability to define configuration points for plugging in your own extensions. However, with this capability also seems to come some pretty hefty complexity, from the example that Howard showed during the session. Like Tapestry, HiveMind has line-precise error reporting, which is always nice. But by far the coolest thing is HiveDoc, which thoroughly documents the HiveMind configuration in a JavaDoc-like web page. Someone mentioned a that Spring was going to introduce a similar feature but I haven't seen or heard anything about it yet. Overall, HiveMind looks pretty cool but for now I'm staying with Spring!

The first session on Sunday morning was "Hard-code Multi-threading in Java" given by Neal Ford. Overall this was a good session with lots of live examples showing the thread debugger in JBuilder and OptimizeIt, which are both pretty cool. He also showed using JDB to debug at a very low level. I suppose sometimes writing web apps is nice since you don't normally need to worry about threading - well, you actually do since servlets are by nature multi threaded, but you get to deal with threading at a much more basic level than worrying about deadlocks, lock starvation, etc.

The next session was "Ant Hacks" by Erik Hatcher. Erik did his usual bang-up job and showed some really cool new things in the latest version of Ant. A cool feature is the <image> task to do things like write the version number onto the splash screen when building, or creating image thumbnails. Next was the <import> task which provides the ability to import another Ant script to mix-in the imported build files, override targets in an OO-like fashion, and define abstract targets which must be overridden. Another really cool new task is the <subant> task, which recurses a directory tree and can operate in one of two modes. The first mode executes the same build file against each directory, which would be really useful if you have subprojects within a large project that all follow the same directory structure. The second mode is to use <subant> to execute a collection of build files, e.g. run all build.xml files in the directory tree. The <presetdef> and <macrodef> tasks look really useful for eliminating duplication in build files. And the <scriptdef> task could be really useful sometimes by allowing you to write script in your builds using one of several languages, such as JavaScript, Python, BeanShell, and Groovy. New stuff for Ant. All useful.

After the Ant session, went to another session by Erik on Subversion, a potential CVS-killer. Actually after earing this talk I believe it is a CVS-killer and I plan to start using it soon. Some of the cooler features are atomic commits, true version history across copy and rename operations, versioned metadata, directory versioning, and offline operations like status, diff, add and remove! Go see for yourself. Oh, and apparently all the Apache projects are migrating to Subversion...that ought to say something.

Ah finally. The last session. "Top 10 Security Vulnerabilities Developing Web Applications" by Neal Ford. They are in a nutshell: unvalidated input; broken access control; broken authentication and session management; cross-site scripting flaws; buffer overflows (though not in Java; injection flaws (e.g. SQL injection); improper error handling; insecure storage; denial of service; and insecure configuration management. One interesting thing Neal talked about was Stinger, an open-source tool that validates HTTP requests against an XML rule set. Another cool toy he mentioned is WebScarab by the Open Web Application Security Project (OWASP). This tool allows you to "record the conversations (requests and responses) that it observes, and allows the operator to review them in various ways", like trying out illegal values and seeing how your application behaves. This session could have been improved a lot if Near had shown using these and other tools to demonstrate the security vulnerabilities he talked about, but overall was informative and interesting.

Whew! Done, and it only took me another month and a half after the conference. :-(

It's been a month since I went to the Reston No Fluff Just Stuff and I still haven't written all the sessions. Maybe next time I'll blog the sessions while I'm there. Anyway, went to a good session given by Dave Thomas on the Ruby programming language. It seems there has been a lot of discussion lately by Stu Halloway, Dave Thomas, and others about metaprogramming and how certain languages don't need the notion of patterns simply because the language supports the "patterns" as part of the language.

Some of the cooler things Ruby supports include closures, blocks, iterators, mixins, and the ability to add and remove methods dynamically from core classes like String. In Ruby, everything is an object, unlike Java and C# which have primitive and reference types. And Ruby does away with the special new keyword in favor of a special initialize method which is invoked when you call new to create an object:

myBook = Book.new("Programming Ruby")

Another cool thing is that Ruby provides native support for defining attributes and specifying whether they are readable and/or writeable using a shortcut notation:

attr_reader :title
attr_accessor :artist

The above defines a read-only attribute named title and a read/write attribute named artist. Even better you can directly access these attributes using dot notation, e.g. myBook.title and myBook.artist = "Dave Thomas" without needing to declare mundane Java-style getters and setters. The above are shorthand for a def block which is the formal notation for defining attributes. For example, the attr_accessor is equivalent to the following:

def artist
  @artist
end

def artist=(val)
  @artist = val
end


In Ruby, instance variables are always private and are prefixed with an @ symbol, e.g. @artist. That's kinda cool. No more project code convention arguments whether instance variables should be prefixed with an underscore, or always prefixed by the this keyword, or any other contrived mechanism for unambiguously identifying instance variables in code.

Ruby support for blocks and iterators is really cool. For example, since everything is an object, you can write code like the following:

3.times { puts "Hello, Ruby" }

myHash.each { |key, value|
  puts "#{key} -> # {value}"
}

IO.foreach("/my/file") do |line|
  process(line)
end


The first line in the examples above prints "Hello, Ruby" three times, as you would expect. Pretty nice. The next example shows iterating over a hash and printing the keys and values. The last example shows iterating over the lines in a file and calling some method process to handle each line. In Ruby a block is simply a "chunk" of code attached to a method. This has the nice side effect that you can pass around blocks of code to be executed in predefined methods. So you could define a method called do_twice and define it as such:

def do_twice
  yield
  yield
end


Then if you called the do_twice method using the line do_twice { puts "Ruby is cool!" }, the output would simply be "Ruby is cool!" printed twice.

There was a lot more to Dave's presentation including examples of scraping a web page, connection to a database, and information on Ruby web and persistence frameworks. All in all, Ruby seems like a pretty cool language, and I'd like to try it out on a few projects. Oh wait, I just remembered that requires having some free time...

Check out Dave's Programming Ruby book.

The Spellbound Spellchecker for Firefox simply rules! Why anyone continues to stick with Internet ExploDer I have no idea, and would encourage everyone to try out Mozilla Firefox. A couple of good articles on Firefox are:

• Ten reasons to try Firefox
• Firefox Secrets
• More Firefix Secrets

After a few hours I guarantee you'll never want to use IE again.

Today I was discussing software development with several colleagues. We started talking about unit testing and one developer mentioned he simply didn't have time to write unit tests and needed to get the code written and shipped. I won't bother trying to argue whether unit testing saves time and money and produces higher-quality software in the long run or not (of course I think it does but that's not the point here). My point is simply this: I think unit testing is a part of a developer's job in the same way that using version control is part of a developer's job. Most projects nowadays would never consider not using version control. Using version control is simply an accepted part of software development that adds significant value during software construction. I think unit testing should have the same status, such that projects simply consider unit testing part of every developer's normal job. There should not be separate schedule items for unit testing like you so often see on project teams. When I write code now I literally do not feel confident about it if I don't have corresponding test cases.

Probably the best session at No Fluff is Dave Thomas' "Herding Racehorses, Raching Sheep" talk. It isn't really a technical session and could be applied not just to software development but to many other industries. The basic premise is simply that the software industry does not have enough people who are at the Competent level or above on the Dreyfus Model of Skills Acquisition, and that this has had a negative effect on the software industry in terms of the quality of products produced. The Dreyfus Model basically states that there are fundamental differences in how people at different levels perceive the worlds, how they solve problems, how they create mental models, how they acquire new skills, and what affects their performance. The Dreyfus Model has five levels: Novice, Advanced Beginner, Competent, Proficient, and Expert. A person can be at different levels for different things. For example, Dave gave the example of how he began learning to fly as a novice, the differences in the way you think and perceive, and how you progress to higher levels.

The title "Herding Racehorses, Racing Sheep" derives from the way the software industry treats individual software programmers, which is to say many naive companies think all developers are interchangeable and should be treated the same. This is extremely common in the industry, and the companies that do best understand this sentiment of uniformity and interchangeability is simply wrong and is actually counterproductive. Many studies have shown that programmer productivity differs by orders of magnitude between beginners and experts.

What can be done? Dave suggests we must "encourage competence" through better methods of training, keeping experts in development jobs rather than "promote" them to management, and make payscales match the actual skills and more importantly productivity of programmers. Trying to change the mentality of companies will be a difficult challenge at best, and an onging one. But one sure thing individual developers can do is to, as Dave puts it, "invest in their own Knowledge Portfolio" and continue to learn new things throughout their careers. Yes, many low-level development jos are moving offshore and will continue to do so, but developers who continually maintain and enhance their skills and knowledge portfolio will thrive.

My own philosophy has been to always be learning, always upgrading my skills and evaluating myself against others. This is the same mentaility as in the medical, law, and accounting indistries where those professionals are constantly learning, going to conferences to gain new knowledge, etc. For some reason, though, a large percentage of software programmers do not this. They feel no need or desire to learn anything new and continue to stagnate. I have worked with many developers who simply do not care about improving their skills or learning anything new. In the past these people had no problems retaining a job, because programmers were in such demand. The big difference now is that the demand is still there but their jobs are leaving to go overseas. After all, why pay someone here in the United States three or four times as much for the same productivity and skill level than someone half-way across the world who has the same skills and knowledge but costs much less? In fact, Dave mentioned at one point that he though about 30% of developers should be fired, to weed out those who haven't learned anything new since they graduated college. My only question is whether the percentage shouldn't actually be more like 50%.

Carl linked to a really good article on how Microsoft may change the world. This will be really interesting to track, to see if they succeed and if they do, how much the large system integrators like IBM and EDS will start to squirm. Far too long have these types of companies been allowed to bungle hundred-million dollar contracts and then get awarded the next one to do it all over again. Having worked on a multi-year $500 million waste of taxpayer money for a year and a half of my life in 2001-2, it would be nice to see someone (even if it is Microsoft) teach those companies how software development should be done. On a 200 person project, you should see the looks you get when you tell managers you could take 10-20 really good developers and have it done not only faster but also better. But that's really not that difficult since most of the time those types of contracts never deliver anything at all. That project I worked on? It started in 2001, is still going, and still hasn't delivered anything at all. But the contractor is sitting pretty with $500 million in revenue, so what do they care? I would have delivered working software many times over by now, and gone on to the next challenge while the big integrators sit on the same contract for 20 years maintaining the status quo.