Scott Leberknight's Weblog

Today I was discussing software development with several colleagues. We started talking about unit testing and one developer mentioned he simply didn't have time to write unit tests and needed to get the code written and shipped. I won't bother trying to argue whether unit testing saves time and money and produces higher-quality software in the long run or not (of course I think it does but that's not the point here). My point is simply this: I think unit testing is a part of a developer's job in the same way that using version control is part of a developer's job. Most projects nowadays would never consider not using version control. Using version control is simply an accepted part of software development that adds significant value during software construction. I think unit testing should have the same status, such that projects simply consider unit testing part of every developer's normal job. There should not be separate schedule items for unit testing like you so often see on project teams. When I write code now I literally do not feel confident about it if I don't have corresponding test cases.

Probably the best session at No Fluff is Dave Thomas' "Herding Racehorses, Raching Sheep" talk. It isn't really a technical session and could be applied not just to software development but to many other industries. The basic premise is simply that the software industry does not have enough people who are at the Competent level or above on the Dreyfus Model of Skills Acquisition, and that this has had a negative effect on the software industry in terms of the quality of products produced. The Dreyfus Model basically states that there are fundamental differences in how people at different levels perceive the worlds, how they solve problems, how they create mental models, how they acquire new skills, and what affects their performance. The Dreyfus Model has five levels: Novice, Advanced Beginner, Competent, Proficient, and Expert. A person can be at different levels for different things. For example, Dave gave the example of how he began learning to fly as a novice, the differences in the way you think and perceive, and how you progress to higher levels.

The title "Herding Racehorses, Racing Sheep" derives from the way the software industry treats individual software programmers, which is to say many naive companies think all developers are interchangeable and should be treated the same. This is extremely common in the industry, and the companies that do best understand this sentiment of uniformity and interchangeability is simply wrong and is actually counterproductive. Many studies have shown that programmer productivity differs by orders of magnitude between beginners and experts.

What can be done? Dave suggests we must "encourage competence" through better methods of training, keeping experts in development jobs rather than "promote" them to management, and make payscales match the actual skills and more importantly productivity of programmers. Trying to change the mentality of companies will be a difficult challenge at best, and an onging one. But one sure thing individual developers can do is to, as Dave puts it, "invest in their own Knowledge Portfolio" and continue to learn new things throughout their careers. Yes, many low-level development jos are moving offshore and will continue to do so, but developers who continually maintain and enhance their skills and knowledge portfolio will thrive.

My own philosophy has been to always be learning, always upgrading my skills and evaluating myself against others. This is the same mentaility as in the medical, law, and accounting indistries where those professionals are constantly learning, going to conferences to gain new knowledge, etc. For some reason, though, a large percentage of software programmers do not this. They feel no need or desire to learn anything new and continue to stagnate. I have worked with many developers who simply do not care about improving their skills or learning anything new. In the past these people had no problems retaining a job, because programmers were in such demand. The big difference now is that the demand is still there but their jobs are leaving to go overseas. After all, why pay someone here in the United States three or four times as much for the same productivity and skill level than someone half-way across the world who has the same skills and knowledge but costs much less? In fact, Dave mentioned at one point that he though about 30% of developers should be fired, to weed out those who haven't learned anything new since they graduated college. My only question is whether the percentage shouldn't actually be more like 50%.

Carl linked to a really good article on how Microsoft may change the world. This will be really interesting to track, to see if they succeed and if they do, how much the large system integrators like IBM and EDS will start to squirm. Far too long have these types of companies been allowed to bungle hundred-million dollar contracts and then get awarded the next one to do it all over again. Having worked on a multi-year $500 million waste of taxpayer money for a year and a half of my life in 2001-2, it would be nice to see someone (even if it is Microsoft) teach those companies how software development should be done. On a 200 person project, you should see the looks you get when you tell managers you could take 10-20 really good developers and have it done not only faster but also better. But that's really not that difficult since most of the time those types of contracts never deliver anything at all. That project I worked on? It started in 2001, is still going, and still hasn't delivered anything at all. But the contractor is sitting pretty with $500 million in revenue, so what do they care? I would have delivered working software many times over by now, and gone on to the next challenge while the big integrators sit on the same contract for 20 years maintaining the status quo.

Dave Thomas at it again, giving a really interesting presentation on what exactly Open Source development is, how it works, and what can be learned from they way it is written. I think the main points in this presentation were that open source is driven by a need (e.g. a developer scratching an itch), is measured the value or utility it provides, and is controlled by evolutionary pressure. This last point is interesting, since it basically translates to survival of the fittest. The evolutionary pressure ensures that only the best open source software thrives, the average open source projects survive, and the worst die off and are never heard from again. This formula makes it "likely to produce high-quality, applicable products." As a result, many developers prefer the best open source products over the best commercial offerings.

Dave mentioned several interesting tidbits that jive well with the survival of the fittest in the open source world. Specifically, that open source projecs are not "utopian communes of loving developers" and are instead characterized by dictator-driven meritocracies where the best rise to the top; brutal honesty between developers; and a focus on delivering value. What a shame most companies refuse to learn from this model.

The last two sessions last Friday at NFJS were spent listening to David Geary explain the wonderous nature of JavaServer Faces (JSF). He started by giving a brief history of Java on the server-side focusing mainly on Servlets, JSP, and Struts. David is a member of the JSF expert committee, and so obviously likes JSF. I, on the other hand, do not like it at all.

According to Geary, JSF is both a "better Struts" and as "the standard web application framework". I think neither is true. First, in many ways JSF is similar to Struts. That's not surprising since Craig McClanahan, Struts' creator, and David Geary, the 2nd Struts committer, both helped to define JSF. That doesn't mean it's better though. Second, how is JSF "the" standard web application framework? Perhaps in Sun's distorted view it is, but I doubt it is to WebWork or Tapestry users, or how about .NET users? For example, though EJB is a standard, how many people have been running away from it at full speed towards lighter-weight frameworks like Spring and HiveMind?

JSF was designed by a committee, and it looks like it. JSF tries to satisfy a ton of requirements. It tries to define a flexible and extensible model for creating UI components. It tries to separate GUI rendering from the actual GUI components, to allow a component to be rendered into different view technologies. It has an event model, a validation framework, a navigation framework, and supports i18n. It doesn't seem to do any of these things particularly well, however, and seems overly abstract and convoluted right now. Perhaps it'll get better in the future but I'm not interested right now.

One thing about JSF that really irks me is the lack of useful GUI components, such as a DatePicker component or a powerful Table component that supports sorting, pagination, etc similar to DisplayTag. According to Geary, the JSF committee was a year and a half behind schedule because of infighting between Apache and Sun over the future of Struts versus JSF. After they resolved their differences they actually began doing something but then were pressured to "get something out". Due to this their strategy was to provide a flexible and extensible component model from which you can build complex component types. While it's nice to have that flexibility, I think it is absurd that they simply left out some of the most useful and needed components. I'd have gladly waited another year for them to put JSF out if they had included useful GUI components out of the box.

Another thing that is annoying is that JSF was designed with tools in mind, and in fact was designed (more or less) for vendors to create point-and-click GUIs such as Sun Studio Creator. Designing a framework specifically to permit tooling is not necessarily a bad thing, but I think a framework should be not painful to use without tools. JSF development without tools seems painful to me. Sun's position, as stated at JavaOne and by Geary at NFJS, is to use JSF and the supporting tools as a way to lure millions of Microsoft VB developers to Java development. Is that really a good thing?

The first session I attended at the Reston No Fluff Just Stuff conference was "State Machines: Moving Logic into Data" by Dave Thomas. It was, as pretty much all of Dave's presentations, very informative and well done. Dave started with a very simple state diagram and some Java code that represented the various states, transitions, and events. He showed how quickly you can get yourself into trouble using a code-driven approach rather than a data-driven approach, with masses of conditionals within conditionals, switch statements, etc.

We then saw some serious refactoring into a much more practical table-driven approach where the state machine is described in a database table. We then moved into a discussion of the GoF State Pattern and showed how that is not really the best way to implement state machines, and how you end up with some serious code bloat using the State Pattern. We then moved into State Machine Compilers, the specific example being SMC, the State Machine Compiler available on Sourceforge. Interestingly, SMC generates code that follows the State Pattern! Of course, since the code is generated for you, it doesn't matter all that much, precisely since the code is generated for you.

Dave ended the session with four hints. First, make life easy when possible by using strings rather than numbers for states and events. In addition, add debugging hooks on state transitions. Second, states are long lived, meaning state can be stored in a database for long-running business processes. A state is a "stable, quiescent place" until an event occurs which triggers a transition to another state or back to the same state. Third, state machines are business logic. If you put the logic in data not code it is easier to change the logic. This data-driven approach can also allow business users to change the logic by modifying data rather than requiring a code change and redeployment. Fourth, isolate and test. Specifically, separating your state machine from event generation permits separate unit testing of individual transitions using mock events.

Just got home several hours ago from the Reston No Fluff Just Stuff conference. It started on Friday and ended Sunday afternoon, and as usual was packed with high quality technical content and excellent speakers. I was too lazy to write any blog entries during the conference sesions, plus I probably would have gotten behind in the sessions trying to learn and write simultaneously! But over the next couple of days I'm going to try and write down descriptions of some sessions and conference highlights.

I'll start by mentioning the keynote that Dave Thomas (of the Pragmatic Programmers) gave on Friday night was really interesting. It was entitled "The Art in Programming" and talked about whether programming is art or engineering. He gave lots of really interesting examples of famous artists, showing their "sketches" and the final products. It was intriguing to see sketches made by Da Vinci, Michelangelo, and others as they tried to gain an understanding of a problem, and then see the final painting. The important point is that you cannot just start off doing the final product; you first have to try some things out, make some preliminary "sketches", figure out difficult parts of your problem before you can dive into creating the final product, whether it be software or a painting. He noted that in the software industry, though, developers are somehow expected to go straight to the final product without doing any sketches, without following "tracer bullets" to get closer to the target, without gaining a better understanding of their problem. This is exactly why most software projects fail. Many managers and the general culture in the software industry believes it is wasteful to do any "sketching" and that developers are supposed to be "professional" and "get it right the first time".

Another suggestion he made was the duality that without art, there is no engineering; while without engineering there is no art. I for one agree and enjoy the art part of software development just as much as the engineering part. It is really satisfying to start with a blank editor buffer, type some text, press compile, and then a program spring into existence which can perform all kinds of useful functions. I am in no way doing justice to Dave's presentation. Suffice it to say it was a great presentation. If you get the chance to see Dave speak, jump at the chance. He speaks regularly at the No Fluff conferences around the country.

One project I'm working is using Hibernate as the persistence framework. We are writing the Hibernate XML mapping files and then generating the POJO model classes. We initially began writing the mapping documents using the Hibernate built-in types like "string", "integer", and "float". A potential gotcha to watch out for is using primitives rather than the wrapper types for numeric fields. The reason is simple - there is no way to set a primitive to a null value since they always assume default values in Java. So if you have a table in your database with a numeric column that allows nulls and a corresponding POJO that uses a primitive to represent the field, when Hibernate tries to hydrate an object from a row in that table it will get really upset because you cannot set a primitive to null!

When generating the POJOs from the Hibernate XML mapping files, we found that some fields in the POJOs were generated as primitives instead of the wrapper types. I am not really sure how exaclty Hibernate decides whether to generate a primitive or a wrapper type, and it doesn't seem consistent as to whether a primitive or wrapper type is generated. Maybe it depends on the database dialect or something like that. In any case, our solution was to set the type attribute of the property elements to the fully-qualified class name such as java.lang.Integer and java.lang.Long rather than integer and long to force Hibernate to use wrapper types no matter what. For example:

<property name="quantity" column="quantity" type="java.lang.Integer"/>

This ensures that the field in the generated POJO is declared as a java.lang.Integer and can handle null values. I've read the Hibernate reference documentation, JavaDocs, and Hibernate in Action, and none of these resources seem to mention (that I've found) that you can do this. There is only one example in Hibernate in Action of using the fully-qualified class name as the type for property elements, and that is showing the output of using Middlegen to generate Hibernate mapping files from a database. I would think this would be a common issue and addressed in the books and documentation but apparently not.