Manual testing

Posted on December 20, 2004 by Scott Leberknight

Does you organization have a test department that still use a gigantic Word document or Excel spreadsheet to document all paths through a system and makes its human testers manually click through everything on a web site or rich client application? Several people I work with mentioned that on a recent project they used a test department that demanded a complete script of what buttons they need to click, what data to enter, in what order, and what results to expect. They then set the testers loose. Apparently all these testers did was to follow the "happy trail" and magically the application passed with flying colors! Of course the reality was much uglier, since they really didn't exercise the application either in terms of executing all potential paths thought the user interface, entering erroneous data, or load testing the application.

I submit that any testing team that insists on having humans manually click through a scripted session and manually type in the results is absurd, a waste of valuable time and money, and ultimately adds zero value to the software development effort and probably adds negative value.

There are tools out there to automate many of these repetitive tasks. The only one I ever worked with was Rational's Robot product, and that was back in 2001. It allowed you to record the interactions with an application and perform assertions on the results of those interactions. That allows you to automate regression tests in the same way developers can re-run all their JUnit tests at the touch of a button. From a brief visit to the Robot web site it appears to be able to do a lot more now, including web applications. I also found this article about using the Rational Robot automation framework support (RRAFS) and another link off that page to the Software Automation Framework Support project on Sourceforge.

In any case, please do your company a favor and let someone know there are lots of tools to automate many aspects of functional testing!

Five Sessions in Five Minutes (NFJS)

Posted on December 20, 2004 by Scott Leberknight

Ok, since I've been slacking so much since the November 5-7 No Fluff Just Stuff conference, I am going to write about the last five sessions I attended in this single entry. Each session gets one (hopefully short) paragraph.

Howard Lewis Ship, creator of Tapestry and HiveMind, gave an introductory session on HiveMind. This is another Dependency Injection/Inversion of Control container similar to Spring. Looks pretty interesting, especially the ability to configure separate modules and give them versions. It also contains the capability to define configuration points for plugging in your own extensions. However, with this capability also seems to come some pretty hefty complexity, from the example that Howard showed during the session. Like Tapestry, HiveMind has line-precise error reporting, which is always nice. But by far the coolest thing is HiveDoc, which thoroughly documents the HiveMind configuration in a JavaDoc-like web page. Someone mentioned a that Spring was going to introduce a similar feature but I haven't seen or heard anything about it yet. Overall, HiveMind looks pretty cool but for now I'm staying with Spring!

The first session on Sunday morning was "Hard-code Multi-threading in Java" given by Neal Ford. Overall this was a good session with lots of live examples showing the thread debugger in JBuilder and OptimizeIt, which are both pretty cool. He also showed using JDB to debug at a very low level. I suppose sometimes writing web apps is nice since you don't normally need to worry about threading - well, you actually do since servlets are by nature multi threaded, but you get to deal with threading at a much more basic level than worrying about deadlocks, lock starvation, etc.

The next session was "Ant Hacks" by Erik Hatcher. Erik did his usual bang-up job and showed some really cool new things in the latest version of Ant. A cool feature is the <image> task to do things like write the version number onto the splash screen when building, or creating image thumbnails. Next was the <import> task which provides the ability to import another Ant script to mix-in the imported build files, override targets in an OO-like fashion, and define abstract targets which must be overridden. Another really cool new task is the <subant> task, which recurses a directory tree and can operate in one of two modes. The first mode executes the same build file against each directory, which would be really useful if you have subprojects within a large project that all follow the same directory structure. The second mode is to use <subant> to execute a collection of build files, e.g. run all build.xml files in the directory tree. The <presetdef> and <macrodef> tasks look really useful for eliminating duplication in build files. And the <scriptdef> task could be really useful sometimes by allowing you to write script in your builds using one of several languages, such as JavaScript, Python, BeanShell, and Groovy. New stuff for Ant. All useful.

After the Ant session, went to another session by Erik on Subversion, a potential CVS-killer. Actually after earing this talk I believe it is a CVS-killer and I plan to start using it soon. Some of the cooler features are atomic commits, true version history across copy and rename operations, versioned metadata, directory versioning, and offline operations like status, diff, add and remove! Go see for yourself. Oh, and apparently all the Apache projects are migrating to Subversion...that ought to say something.

Ah finally. The last session. "Top 10 Security Vulnerabilities Developing Web Applications" by Neal Ford. They are in a nutshell: unvalidated input; broken access control; broken authentication and session management; cross-site scripting flaws; buffer overflows (though not in Java; injection flaws (e.g. SQL injection); improper error handling; insecure storage; denial of service; and insecure configuration management. One interesting thing Neal talked about was Stinger, an open-source tool that validates HTTP requests against an XML rule set. Another cool toy he mentioned is WebScarab by the Open Web Application Security Project (OWASP). This tool allows you to "record the conversations (requests and responses) that it observes, and allows the operator to review them in various ways", like trying out illegal values and seeing how your application behaves. This session could have been improved a lot if Near had shown using these and other tools to demonstrate the security vulnerabilities he talked about, but overall was informative and interesting.

Whew! Done, and it only took me another month and a half after the conference. :-(