The recalcitrant engineer

2003-11-05T18:06:00.000-08:00

301: This blog has moved permanantly

2003-11-04T16:36:00.000-08:00

The "too complex" rule: if you can't figure it out in a day or two, it shouldn't be done

There are a few things that have been bouncing around in my head the last few days with regard to software. The first is an old adage, from I think Brian Kernighan, that says (roughly) "you must be twice as smart to debug code than you need to be to write it". If you use all your cleverness to write the code, then by definition you aren't smart enough to debug it. The other thing is a quote from Albert Einstein that (paraphrased) is "If you really understand something, you should be able to explain it to your mother."

I've been thinking about this a lot, both due to the inherent conceptual complexity of a web framework I've been working with and the difficulty trying to explain it. The other thing that's been bouncing around is trying to figure out a really complex web UI if you're a backend java person.

Real world problems are hard. Really really hard. A commercial webapp that needs to actually scale to thousands of users and cluster, that has hundreds of pages and probably 50-100 tables is a beast. Super complex. Not necessarily in any part, but as a whole, it's a freakin' monster. When you try and build a framework that can be a platform for such a thing, you're up against a very big, very tall cliff. There's almost this tradeoff: by the time you get your framework complete enough to actually work for the app, it's going to be so complex in its details and behavior that it won't be understandable without a huge amount of ramp up time. There's a lot to be said for good theoretical backing, orthogonality, good general design, etc, but that won't make things necessarily crystal clear. I know that ultimately, if its a good design, it will have that clarity, but I am wondering if anyone is smart enough to do it, or if its just going to take a lot more time and a lot more baby steps before we can get there.

The other half of this is complex user interfaces. I'm currently working with a UI where developers have implemented extremely clever JavaScript, which, when it explodes, blows the minds of everyone expect perhaps 1 or 2 people. Such a thing is really not workable, as the long term maintenance is not going to be either cheap or possible, depending on your perspective.

I'm starting to wonder if the only thing that actually works is really, really dumb code. Not necessarily bad, but forgoing a lot of things like complex user interfaces, skinnability (micrositing), massive performance throughput, etc, in order to just get something that a normal human can understand and maintain. It's depressing that microsoft access is looking like a good answer for business applications.

2003-10-27T09:47:00.000-08:00

Naming Conventions: Language versus Technology Holy Wars

One thing I've been thinking about a lot is poor design decisions, and how they cripple software. I'm probably going to write a good bit about this over the coming week or so as I serialize out my musings and try and extract something worthwhile to read. There's a lot of angles on this, and I'll try and write down all the things that have been floating around in my head on them.

Today, we see a lot of API and framework designs on top of existing languages, rather than a lot of new language designs. The main driver here is install base on the lanaguage side, but as Ralph Johnson told me several years back, APIs and frameworks themselves constitute a meta language, and are every bit as hard to design as the language itself. Sometimes it seems like this point is lost on the people that are doing the actual design.

Anyway, what I was thinking about is people that are doing APIs for a specific technology that exists in multiple languages. The classic (old) example of this is CORBA: When you build your IDL (interface description), it defines your object API in every language (with a single naming convention). The notable exception to this is in Smalltalk, where the language binding for CORBA actually fixes the naming conventions from "foo_bar" to "fooBar" at IDL compile time so the code you write still LOOKS like Smalltalk. In my opinion, this is another case where the Smalltalkers made the absolute correct decision.

A modern example of this is the APIs for acessing all the various X* technologies (DOM, XSLT, XPath, ...) and their language bindings. If you use the W3C standard APIs for Java (TRAX, JAXB, etc), you end up using these weird NodeList objects and other things that make your Java code awkward, ugly and difficult to maintain. However, if you use something designed for Java, like dom4j, you end up with code that looks, reads and feels like Java. And, as such, is amazingly cleaner and easier to maintain.

The disconnect here, as I see it, is the hubris of the people driving the technology, and their focus on a single language (usually C++). The assumption is that the technology will be far bigger than any single language, so the goal is to have code in the various languages look basically similar. This makes it so developers can move between languages and immediately be able to work on code that is driven by technology X.

I think this highly misguided, and misses the point of how real programmers work. In the real world, Java programmers write Java code. It has a set of conventions that are well defined, and the more Java looks like Java, the easier it is for people to come in and work on it. I will probably never work on DOM trees in C++, and if I did, I would have to go back and read the documentation anyway to find out all the language specific nuances of the APIs, so this convoluated API isn't buying me anything.

Having missed this point, many standards boards (such as the W3C) end up burdening other languages with these funky APIs, forcing bifurcation and duplication of effort as people work to reinvent what the API should have been in the first place, after the standard implementation is complete.

2003-10-24T14:37:00.000-07:00

How can is it possible that BEA's JMS is so slow?

A while back, I wrote a large amount of code that used the JMS implementation in Weblogic 6.1 for internal messaging. The system was a user interactive system, where timely deliveries of messages were critical for user display. Ideally, you wanted the user input to trigger a message, have the system act upon it, and have the user interface update with the result in milliseconds. What I expected when I began was that things would run at or near method invocation speeds, as long as I was in the same JVM.

The result I found was that things would stay magically "in the queue" for 3-5 seconds (that's seconds, not milliseconds) before the response was heard, each time a write was done. If you had to do a JMS write, then a JMS read to service a single page request, the result was an additional 6-10 seconds PER PAGE, on top of your normal computation and page rendering time.

You can say that a lot of things, such as removing the filters (I was using filters of moderate complexity) might have helped. You can say "Gee, everybody knows Weblogic JMS is slow." I, honestly, don't really care. The Weblogic JMS server is basically unusable for user interactive applications. I had to turn around and write my own internal messaging system, developing my own filter mechanism, etc. But once I was done, it ran at the speeds that I expected, and is in heavy production use today.

The honest truth behind it is that JMS has a lot of guarantees on the message delivery. In fact, it makes TCP seem like a total weakling in comparison. And yes, doing a lot of these things probably requires a lot of effort on the server's part. What I think BEA is honestly missing is the ability to relax what the server guarantees you in terms of reliability in order to recover the speed you need. Most people and most applications honestly don't require all the fancy pants stuff that JMS offers, at least most of the time. But for Weblogic to really make good on some fraction of JMS's potential is going to require it to become much more flexible and lightweight. It's a great mechanism for scaling computations, if it was fast enough to deploy.

2003-10-24T11:33:00.000-07:00

TimeZones: It's not just a java problem

I was sent this link this morning about one man's pain with daylight savings handling under Win32. If you read this, it's basically a dump on how the internals of the time zone computation mechanisms in windows are as weird and flaky as Java when it comes to DST handling.

2003-10-22T17:02:00.000-07:00

Who thought of the J2EE configuration mechanisms?

I think I'm something of a rare bird, in a way: I've worked as a system administrator for large, enterprise systems, and I've worked as the architect developing those systems. Sometimes it seems like the distance between the two is a chasm. The disconnect between the people that write software and the people that install and maintain it is amazing.

Case in point: Configuration files. What a system administrator wants is a flat file that lives somewhere (preferrably in /etc or /var somewhere) that clearly and concisely allows them to tune all the knobs of the system. This thing should be reloadable either via a SIGHUP, or, at worst, via a JVM restart. Either way, the file is readily accessible and easily changed.

J2EE actually puts the config files (deployment descriptors) INSIDE the application BINARY. I don't care that war and ear files are "just zip files". I, as a system administrator, do not want to disassemble the application to find the configuration file. Honestly, as a developer, I won't even manually attempt to take a war apart and then put it back together, because I almost always break it.

When I get in the config file, I want it to make logical sense to someone that doesn't have the source code in front of them. Don't mix the config options in with the nuts & bolts of what makes the application run (such as the servlet mappings).

This frustration has actually led me to abandon the j2ee config mechanisms entirely and implement my own system that is actually accessible to system administrators, doesn't require a GUI, and automatically detects and reloads itself. The response from the people using the applications that I've developed have been amazingly more positive now that they can see config options without trying to disassemble and decipher the deployment descriptor.

2003-10-16T19:22:00.000-07:00

Taglibs and internal state

So I just discovered the hard way that you can't safely store internal state in a taglib.

My current application sports a fairly large taglib set that lets you do many powerful things, such as iteration, value display, automatic form element rendering and lots more. The thing I found is that if you have tags that do things like looping, when they nest, many appservers reuse instances of the Tag class for the nested calls to the tag. The problem here is when the tag saves instance variables: each nested invocation clobbers the state of the parent. If you're doing something like looping tags, when the innermost loop completes, the whole nested iteration completes because the state about being "at the end" of the loop just got clobbered by the inner most tag.

Instead, you have to push the state that you stored in the taglib member variables as attributes in the pageContext, which are scoped by the "depth" of the nesting of the tags. The depth of the nesting has to be computed by incrementing on doStartTag and decrementing right before you return SKIP_BODY. This "stack depth" gives each nesting a namespace in which it can declare its context variables and save its state.

2003-10-14T15:27:00.000-07:00

Time Zone Representations & Software

I've built a few systems in java that required doing lots of TimeZone conversions for end user date display. It's a pretty standard problem for any software that has to work outside of a single site, and generally falls into the i18n/l10n bucket of problems that have to be solved when you're building the app. Which, in my experience, means it rarely gets solved well.

I'm not going to start down the standard path of complaining about the implementation of java.util.Date; it's safe to say that horse is dead. What I did find fascinating, though, is just how complex time zones actually are. The first thing to realize is the timezone list that the JVM ships with is totally inadequate. It changes depending on the platform, and is fairly incomplete. You can't rely on it to build a real application. This generally means you have to make your own subclass of java.util.TimeZone which you then populate from a database (or something) that you ship with the product.

When you go to implement your own subclass of java.util.TimeZone, you'll find you need to provide a lot of values to the constructor of SimpleTimeZone to create the time zone, such as the day of the week that daylight savings time starts, the day of the week in the month it ends, etc. I still find these confusing to some extent, but thankfully I was provided with a list of the data so I just had to implement it.

The tricky part is: how do you actually refer to a time zone? The typical thing I've seen is "GMT-6" or something like that. What you find out when you delve into the details is, "GMT-6" is highly imprecise. All it says is "-6 hours from GMT", which maps to an actual SET of time zones. For -6 hours offset, you could have a set of different timezones with different DST rules.

The next natural question is: What is this set and where can I get the canonical list? The short answer is: It doesn't really exist. The crux of the problem is, time zones are a political creation. If you look, there is no ISO standard for timezone names (at least that I could find). These things are highly fluid; they change at the whim of legislatures and are not well defined to start with. Quite honestly, it's a complete mess.

If you look in /usr/share/zoneinfo on a UNIX machine, you'll see the "state of the art." One of the things that poking around at this data showed me is that timezones have a "historical" dimension, which is 100 times worse. This adds the problem of "from 1971-1979, daylight savings time started on the last Thursday of the month." If you're curious about what doing it "right" entails, read the manpage for tic, which is the timezone info compiler

The best answer I found to all of this is to flatten it into a namespace you define and pick the timezones you need to support from your userbase. Also, discard the historical dimension; not only is it confusing, Java doesn't support it.

The naming convention that I found most useful for naming timezones is "Country/City" like "America/Los_Angeles" because it actually maps to the political entity that created the timezone, and then show that string to the user to let them chose their time zone.

2003-10-13T11:06:00.000-07:00

Key Conversion Step Through: The Mechanics

I had to actually finish the job of getting the PVK file (mentioned below) into a JKS keystore today, and it was non trivial. I did not mean to mislead my loyal readers by portraying the PVK to PEM conversion as the last step in the process, as it most certainly is not.

The whole story is: The PVK format is known as "Microsoft Authenticode" format for a key, and is commonly used by COM (read: VB) developers for signing controls. For actually doing the signing, there's actually two parts that you need: The PVK file, containing the private key, and the SPC file, which is the actual certificate. Both of these must be used together and imported into Windows in order to export to a neutral format so they can be used. The SPC file is downloaded from Thawte after they complete the signing process.

The tool you need for the import is here, and it's called pvkimprt. Once you get that, you follow the directions, do the import and then (interestingly enough) the key shows up in Internet Explorer. In IE, choose Tools -> Internet Options -> Content -> Certificates... and you'll see your key under the Personal tab.

The thing to know about this next step in the process is you need to export the certificate with the private key, and the only way you can do that here is in the PKCS12 format (a well supported standard) , which the JVM (>= 1.3.1) can read correctly. Select Export.... Make sure you export the private key, accept the defaults and safe the cert to a PFX file somewhere.

Once you have a PKX file, you simply need to convert that to JKS format for maximum ease of use with jarsign. As usual, Sun displayed plenty of foresight and doesn't provide a tool to do this. Thankfully, someone has done the legwork for us. They go into much more detail about how to use the key in other environments, but for our purposes, all you need is KeystoreMove (which I have mirrored, but is available from the original page above as well). Compile this, and then run:
java -cp . KeystoreMove pkcs12 yourkey.pfx yourpass jks yourkey.jks yourjkspass Which then leaves you with yourkey.jks that you can use with jarsign.

2003-10-09T12:27:00.000-07:00

PVK, PEM and JKS: Code signing key conversions

Ok: So if you go to thawte and buy a universal code signing key, you've just spent a very small amount of money to be able to run trusted code on pretty much everything everywhere. It's a great deal, but I won't comment on the security implications. It scares me silly. Suffice it to say, you can sign microsoft code (COM controls), java jars and anything else you need.

Anyway, the key you get is in "PVK" format, which is msft proprietary. Don't despair, this CAN be used for Java. What you are trying to do is get it into JKS format for ease of use with the JVM. What you need to do is first convert the "PVK" format to "PEM" (which is a standard) and then convert the "PEM" file to "JKS" using keytool.

I'm not going to explain how to use keytool here, as that's a lot of work. What I will say is the missing link is how to get the damn "PVK" file into "PEM". Thankfully this kind person reverse engineered the key store format and provides a downloadable utility to get the PVK file into PEM. From there, you can use BouncyCastle JCE (see below) to convert the PEM file into JKS with keytool.

2003-10-09T12:04:00.000-07:00

Keystores, keytool, JCE and the state of Java encryption

Not everyone has had to do this, but when you deal with signing jars and lots of crypto in java, things can very quickly get very complex. And you also quickly discover that most of the crypto stuff that's out there is actually broken and flaky. Most important is that the default JCE that Sun ships with the VM is anemic and buggy. In particular, before 1.4, the keytool that you use to manipulate keystores that comes with the JVM could only manipulate a custom proprietary keystore format (jks) unless you swapped the JCE provider. The problem is, none of the key signing agencies actually deal with anything but standards like PEM encoded keystores.

But what is one to do? RSA charges a truckload for their BSAFE libraries, and having them in your app doesn't do much for you except to be able to say "Yep, there's BSAFE in the classpath." OpenSSL (the de facto C standard) can be used for some things like doing command line operations and key generation, but it doesn't do you any good from Java.

The best thing I've managed to find is The Legion of the Bouncy Castle. They make a great JCE provider that actually reads and writes standard formats, along with a lot of other great crypto tools, like OpenPGP compliant encryption, etc. And it's all BSD license, it works and it's a full cleanroom implementation.

2003-10-09T10:48:00.000-07:00

Self Documenting Builds & Ant

As I've been thinking more and more about ant, I've rememberd a lot more tricks that I've found useful. I didn't intend to trickle all this out over a week or so, but I suppose having several articles about ant is more interesting than one gigantic huge article.

There are really two ways to approach telling your users about how to run the build: send them a big document that describes the build, or just have the build itself guide them. In my experience, nobody reads documentation anyway, and it gets rapidly out of date. So it's better to just use what ant provides. In this case, your biggest boost is setting the description on each task. This is done like:
<task name="webapp" description="Build the main deployment ready web application. This makes the war file."> .... </task>
Once you do this, everyone can see what the build targets do with a simple call of:
ant -projecthelp
Which shows you all the targets defined, and their descriptions.

Also, It is frequent that builds need things like Weblogic to be on the host to be able to actually build the project. Usually, this is done by requiring a user to have a local properties file that defines paths for their machine so that the build can get running. The powerful thing here is ant's unless and if attributes on tasks.

The easiest way to do this is to do something make a task like:
<task name="weblogic.config.check" unless="weblogic.home"> <fail message="Please define weblogic.home in your local.properties file to point to your weblogic server home"/> </task> <task name="compile" depends="weblogic.config.check"> .... </task>
This way, you know that unless the user defines thier weblogic.home, the build will fail, and the build will tell them exactly what they need to do to make the build run. This tends to reduce ramp-up time a lot for new hires.

2003-10-08T09:15:00.000-07:00

One more ant tip: paths and multiple build scripts

The reason I wrote the chunk of ant advice below originally was I was trying to remember all the stuff I had learned about advanced usage of ant that was seriously difficult to figure out to pass it on to someone. The problem was, I forgot the thing that I found the weirdest and that actually inspired me to start writing the blog entry.

Anyway, when you're splitting a large build up into multiple files, you really want all those files to have a single global classpath that they share for varius parts of the build. The problem is, normal path blocks are actually local to the xml file that declares them. They are not inerhited across <ant> invocations, even if you set inheritAll=true. What you instead have to do is flatten the path into a property that is colon separated and use that as a classpath rather than a classpathRef.

If you flatten the path in the global build script into a property, then all the invoked scripts can see it correctly and all is bueno. The magic to do this is:

<path id="base.path">
....
</path>

<property name="base.path" refid="base.path"/>

Note that the property and the path exist in separate namespaces, so there's no collision here having them both named base.path. For clarity, it might be better to name the property version base.path.property.

2003-10-07T10:40:00.000-07:00

Binary XML & Bandwidth

I've been looking for a long time for a basic compression scheme that was standardized that I could use for XML. Today I found this, which describes a simple huffman-like encoding for XML. I had personally come up with something very similar a while back, but ran out of time to implement it. I'm glad to see such things are out there and standardized, so theoretically parsers should support them without me writing my own compressor.

2003-10-07T08:36:00.000-07:00

Scaling your ant build, and making it maintainable: Kris's tips for ant

I think most complex apps that get built these days, especially when you've got an appserver and (god forbid) ejbs in the mix, have, by necessity, a pretty complex build. I've often heard of places having dedicated build engineers (I've even worked at some), but almost always even when you have a person dedicated to it, the programmers are going to end up working on the build.

Almost all of the things that I'm writing here are obvious if you're an ant guru, but the one thing I've noticed about ant is the way the documentation is written, you constantly find more and more ways to make your build script that much better. The tricks I've found for making your ant builds scale:

Multi-file builds are absolutely necessary. If you keep your huge multi-target build in one file, you're going to end up with a mess. A java class with 10,000 lines is impossible to understand, and so is an ant script.
Dealing with multi file builds is easiest done with a build script per deployment scenario. That is, if you have a target for websphere, a target for weblogic, etc, and each of these builds its own ear, its best to keep them in separate files.
Ant is not a programming language. Complex "logic" (as best as it can be called) is always brittle, and difficult to debug if it causes subtle things to go wrong. (yes, ant is duct tape).
Code sharing between ant scripts (such as a common set of build scripts shared by multiple apps in the same product suite) are problematic at best and a disaster at worse. This mostly comes into being because of subtle differences between the needs of each product.
A product should never require a user to drop jars in their ant/lib directory. This was necessary in earlier versions of ant, but can now be accomplished by passing a classpath to the custom task declaration and keeping the jars in the source tree. Having your product work with a vanilla ant install is a huge boost to getting people up and running.
Use custom tasks with extreme caution. Not many people understand how ant works all that well, and custom tasks even less so. You can quickly make your build impossible to understand if you have lots of custom tasks with sparse documentation. However, the boost to your system by bringing things together in a single build with custom tasks is also huge, so it is a double edged sword.
Use "-" as a prefix to all internal task names. Internal tasks are those that are intermediate build steps, and should not be called from the command line. The "-" prefix actually makes it impossible to call them from the command line at all.
Use property files instead of individual property statements. That would look like: <property file="build.properties"/> rather than:
```
<property name="foo" value="bar"/>
<property name="baz" value="quux"/>
<property name="blah" value="blahblah"/>
....
```
repeated a million times. Also, name your properties file build.properties. This is becoming standard and I'm starting to see editors (like eclipse) that actually know about the build properties.
Filesets are a powerful tool for setting up your classpath. In particular, doing something like:
```
<path id="app.class.path">
    <fileset dir="${lib}">
        <include name="**/*.jar"/>
    </fileset>
</path>
```
lets your programmers just drop jar files in the lib directory and they automatically become part of the app. This can lead to headaches later when you have to construct things like ejb jar manifest Class-Path entries that need to explicitly list jar names, but that can also be handled with the right set of ant incantations. The alternative to this (which I've also done) is explicitly listing the jars in the classpath in the build, which creates a maintainance nightmare as jars are added and upgraded and you try to recall which files contain the jar names.
Put the version numbers of your jar in the jar name. I can't emphacize enough how critiical I think this is, and I'm also floored that other people don't do it. Having a file named classes12.zip versus having a file named oracle-jdbc-8.1.7-1.2.jar is just no contest to me. At some point, some day, you're going to need to swap out those jars and upgrade them, and when you do, knowing what jar versions you're using is going to be absolutely imperitive.

When you're doing you're build, adding a datestamp, build host and time, etc, to some file that is in the output (such as html comments in your login screen) quickly helps with customer diagnosis of what version of the code the person is running. Ant can do this for you automatically with the copy task and filterset. What this looks like is:

<copy todir="${output}">
  <fileset dir="${jsp}">
    <include name="version.jsp"/>
  </fileset>
  <filterset>
    <filter token="BUILDDATE" value="${builddate}"/>
    <filter token="BUILDCOMPILER" value="${build.compiler}"/>
    <filter token="BUILDUSER" value="${user.name}"/>
    <filter token="BUILDSCRIPT" value="${ant.file}"/>
    <filter token="BUILDOS" value="${os.name}"/>
    <filter token="BUILDOSVERSION" value="${os.version}"/>
    <filter token="BUILDARCH" value="${os.arch}"/>
  </filterset>
</copy>

and then adding a block like this to version.jsp:

<!-- Built By: @BUILDUSER@ -->
<!-- Build Script: @BUILDSCRIPT@ -->
<!-- Build Compiler: @BUILDCOMPILER@ -->
<!-- Build OS: @BUILDOS@ @BUILDOSVERSION@ @BUILDARCH@ -->

The productivity boost for your programmers for having both an exploded build target and a packaged build target is well worth the effort of maintaining them. This is J2EE specific, but I think highly useful. There is a large difference in the deployment times for the various appservers if they are deploying a zipped ear file versus an exploded file on disk that just has to have the classes reloaded. Often this can save minutes per build, which is huge for the developers.
This is just a general guideline, and not ant specific, but you should always keep your build fast for development. That means it runs in seconds, not minutes. I've had ant builds that ran ejbc for development (adding 2-3 minutes to the build), and productivity cut by a factor of 2 immediately.

2003-09-30T16:07:00.000-07:00

HTTPUnit: Unit test your web apps!

One thing I've always missed when developing web applications is unit testing. One can always unit test portions of the app, which usually amounts to testing the layers underneath the actual user interface code, but that's really not the place where the biggest disconnects happen. The place where you have real problems is in the user interface talking to the model code via HTTP.

This is something the windows interface people have been dealing with for years, and they have a lot of decent solutions out there for automated UI testing, but most of them are fairly sensitive to things like widget placement, etc. There are also products like Silk that will both do functional and load testing of your app. But almost all of these are targeted at QA folks, which are far too late in the game to actually let you catch your errors. Usually by the time things get to QA as regressions, you have to go through a bug process, etc, and it's just a huge pain. There was an old Rational ad that I always think about when I'm unit testing: Is a mistake a mistake if nobody ever sees it?

Anyway, HTTPUnit seems to me to be the clostest thing to the missing link I've yet seen. It simulates the servlet container and the browser (including javascript, frames rendering, etc) and lets you unit test servlets, jsps, etc, along with write java code that actually walks click paths in the application, sets form values, does posts and verifies UI integrity. All for the developers. All without having to start Weblogic. I'm currently working on integrating it with my application platform to ensure that it serves our needs, but the promise so far is huge.

2003-09-29T16:53:00.000-07:00

Hiring & Women in Software

I'm currently in the middle of trying to bring in an engineer to my current company to help with a product we have in the works. I've plowed through hundreds of resumes, we've done phone interviews, etc, culling out all but around 10-12 of these people. I've been through this process many, many times, and the one thing that really jumps out at me is: When you get down to the short list, I'm starting to notice more and more women in tech that are really really qualified. My wife always says that to be a woman or a minority and to make it as a software engineer, you have to be twice as good as the average person to get noticed.

What I find encouraging is that after the dotcom crash and the whatever the hell we're in now, it seems like the numbers for women showing up in the hiring pool that are actually really qualified and solid candidates are really looking up. It gives me some fairly large measure of hope for fixing the gender gap in software, I suppose.

2003-09-29T16:50:00.000-07:00

The Evils of Downcasting, or Why I Miss Smalltalk

When I learned how to develop using OO, I learned that any downcasting was a "funny smell", and could always be eliminated. My experience of developing this way led me to develop code that was extremely tight and easy to maintain. I had found my development nirvana. All I had to do was follow this pattern, and all was well.

Then I started working in Java. Did I mention I learned how to properly do OO using Smalltalk? All of my Java code always seemed like a hack. There were static "helper" functions cropping up that constantly made me ill. And most importantly, I was writing code that did conditional downcasts that I couldn't work around.

Then it dawned on me: in Java, if I need it, I can't add a method to string to clean my code up. Swing is fixed. All my commericial libraries have fixed APIs. In Smalltalk, if you need to add a method to Boolean, you just go right ahead and do that. The libraries and your code merge into one sort of blob that are both honed to each other. This is a clean, elegant blob that works perfectly, but it also means that you incur the penalty of maintaining all that gargantuan amount of code you've just "tweaked" to make cleaner.

I'm really not sure which is a better tradeoff, but the idea of tweaking the JDBC drivers and the appserver to really make my code smooth, and then incurring the weight of maintaining those 250-500kloc of code that I could previously write off as someone else's problem doesn't sound all that hot. Maybe I'll live with my Java hack code and simply remember OO nirvana like a sweet, pleasant memory.

2003-09-22T14:22:00.000-07:00

Bonkers: Mate for the Masses

In the last few months, I've started drinking Yerba Mate instead of coffee as my primary caffeine source. It's a much happier buzz, as it doesn't leave you feeling all shitty and jittery after 12 hours of slugging it down while coding. After having been off coffee for a while and then going back to it briefly, coffee to me feels like some sort of toxin I'm bringing into my body, while mate feels cleansing.

What I discovered today was Bonkers, which is a non-loose Mate in tea bags. The genius thing there is they have super caffeinated Mate, which is like making a good thing 10x better. It's called WIRED & CRAZY MATE; I encourage you to go find some at your grocery store (I shop at the Piedmont Grocery, which is Divine, and carries all the Bonkers you need).

2003-09-22T11:08:00.000-07:00

NTFS Maximum Path Name Length

So a useful tidbit for windows users: NTFS, at least on XP (and probably everywhere), has a maximum path name length. When I found this out, it really felt like being tossed back into UFS circa UNIX System 3. The maximum length of a path on Windows (at least from the command line) is 256 characters.

What this means from a practical perspective is that if you have deep path structures in your webapp, and you have your appserver deeply structured (such as WebSphere), you'll run into stray messages like:
(The system cannot find the path specified)

The problem is, WebSphere puts the app (by default) at C:\Program Files\WebSphere\AppServer\installedApplications\hostname. When you tack this onto the deep, nice file structure you have in your app, suddenly you're looking at paths far in excess of 256 characters long. And the beauty of it is, WebSphere fails silently and tells you that it deployed correctly. You only see the above message when you try and copy the files manually into the deployment directory using something like ant.

The solution is to tell the appserver to deploy the app (when you're pushing it in the deployment tool) into a shorter path, like C:\temp\ws. This way, it will actually deploy correctly when it says it did.

2003-09-22T09:45:00.000-07:00

Perl is the Yugo of Programming Languages

I, like 99% of the rest of the world, saw this article on Java being the SUV of programming languages. I'll spare you the lawyer jokes (note the url).

The part that really blows me away is that people actually believe this sort of junk and then parrot it to the rest of the world. How is Java too powerful? It has OOP that works and wasn't tacked on as an afterthought? It helps you structure your code so you can actually not end up with a total mess at the end of things?

My only retort to this article is that scripting languages should be used like duct tape. They seal up big cracks in software systems so they work, but they're almost always a hack, and when you try not to make them a hack, it's a hell of a lot more work than it would have been to actually write it using a full featured language in the first place.

I'm not here to argue about strong versus weak typing. I'm here to encourage people to actually TRY the technology in the real world before they start becoming a pro or con evangelist. And, yes, before you ask: I've written more Perl for money than I care to recall. There's a reason why I use Java now, and it's not because it's fashionable.

2003-09-18T11:41:00.000-07:00

Ruby, C#, and Python: Whither the Software Wind Blows

I've been looking for quite a while as to what is going to be the "next big thing" that is going to take off in the software world. At the rate that most of us were used to in 1999, the "hot new things" in Java and XML are starting to grow whiskers.

Over the last 6 months, I've had the opportunity to put out ads and get resumes for various software positions in the bay area. 6 months ago, we would get 200 responses to a job posting in the first day, and about 500 in the course of a week. We put out an ad last week for a java/XML person, and got a grand total of 100 resumes in a week. I see this as a definite sign of real economic recovery. We also didn't get any resumes from the people that were applying for jobs way below their skill sets, which I find relieving. At one point, we were hiring a junior engineer and got a resume from one of the original multics designers. That was a depressing day.

Anyway, long story short: it seems economic recovery is upon us, in some sense (cross our fingers). With this recovery, I'm seeing a lot more mention of alternate technologies and people pursuing real project development in things like Ruby and Python. And, of course, C# is rapidly gaining momentum. There's a feeling of a breath of life and fresh air in the industry, like someone opened the attic and remembered that a long time ago we used to actually invent new things and not keep banging out junk using the same old tools, and maybe they should invest some time, money and effort in moving things forward.

Here's to the future.

2003-09-18T11:19:00.000-07:00

Patterns of Enterprise Application Architecture, or Working Around The RDBMS

I've recently been readin Martin Fowler's latest book, Patterns of Enterprise Application Architecture. This, honestly, is the best patterns book I've read since the original GoF patterns book. It had the same effect of both showing me names for things I already knew how to do, quantifying the tradeoffs of the various decisions and showing me new ways to do things that I hadn't thought of. I fully recommend everyone working on enterprise software (C# or Java, in particular) buys it and reads it cover to cover.

The thing that struck me most about the book is always the thing that struck me most about dealing with enterprise software in general: the RDBMS is like a giant boat anchor that's unfortunately necessary. It seems like a massive chunk of the book is dedicated to dealing with getting your clean object model into and out of the database while not sacraficing too much efficiency or cleanliness. To his credit, Mr. Fowler does a great treatment on the subject that's clear and easy to understand and implement. I thank my lucky stars that hibernate exists, and implements a large number of the patterns in the book cleanly and efficiently so I don't have to.

2003-09-12T11:06:00.000-07:00

Intellij "Aurora": Great new UI, and lots more great plugins

If you're an intellij user, I encourage you to download the latest "Aurora" release from the JetBrains EAP (Early Access Program). It has a great new UI and a killer set of features that just made things that much better. In particular, the new JUnit runner is just too cool for school. The thing that got me to download Aurora in the first place was the Auto Boxing Plugin, which is a plugin that lets you do:

Integer i = 12;

And will "auto suggest" that it be changed to:

Integer i = new Integer(12);

Which conveniently solves a major language shortcoming with just an IDE hook. It does a LOT more than that though, working around a lot of other Java foibles cleanly and gracefully.

2003-09-12T10:13:00.000-07:00

Test Driven Development with Databases

I, like I suspect many people, have always had a problem with test driven development for enterprise systems. The unit tests that you read in the books all have no side effects. They check that the code runs, and if they fail, no big deal. However, the reality for enterprise systems is that tests have side effects: they mutate the data in the database. If they fail in ways you don't expect, they can horque the data that they are depending on to run, and the whole system breaks down. My experience is that in this situation, unit testing gets pretty quickly tossed out the door, because as soon as you end up having to diagnose the data to get your tests to run after they've clobbered it, it becomes very labor intensive and breaks your train of thought and productivity rapidly. I had the opportunity to talk to someone that does a lot of test driven development against enterprise systems, and he offered me the nougat I was missing. Props to ThoughtWorks. The two necessary things are:

Data sets for the tests are as important as the test code, and are fixed. These are guarded with the same care that you guard the test code itself. This I had managed to figure out on my own.
Test code, when it runs, has a harness around it that always, when the execution is complete, rolls the transaction back. This way, even if the test fails mysteriously, it won't clobber the underlying data that the tests depend on.

The only downside to this that I see is that tests have to have all their data that they need defined as DML that is loaded into the system using something like sqlplus before the tests can run, because by definition, the tests themselves cannot modify the database.

2003-09-11T12:58:00.000-07:00

Anonymous no longer

Cameron mentioned me in his /dev/null blog, and pointed out that I was ranting anonymously, which I found both suprising and pretty amusing. I mean, someone is actually reading this.. that sets a much higher standard for me to live up to; delivering quality content. This is me on the left, and this was me while still in grad school. My name is Kris, and I can be contacted at kris at further dot com.

2003-08-12T17:29:00.000-07:00

WebSphere v5: A case study in how to design the worst development platform ever

I've had the opportunity to work with a large set of application servers and j2ee platforms, including Resin, Tomcat, WebLogic and WebSphere. The open source (Tomcat, Resin) servers, have, in my experience, always been the best platforms to work with, hands down. Configuration is simple. Applications deploy and run with minimal trouble. Things are buggy, but software is always buggy. The cursing you do about the appserver is generally good natured, and you get along. If you really get stuck, you grab the sourcecode and debug the stupid thing, which almost universally works and isn't terrible to do because, well, it's an http protocol translator in a fancy box.. how complex could it be really?

Enter WebSphere.

Really, all I'm looking for in a development platform is:

Stable: The platform has been used by a lot of people, and the job it is expected to do it does reliably. This means no crashing, even when i redeploy my application a zillion times.
Fast Development: You should be able to drop code into the appserver and see it run without reading the wall street journal between compiling and actually seeing the code run.
Non Developer Hostile Feature Set: In the interpretation of any standard, there are always things that are ambiguous, and can be interpreted in several ways. Often times one of these ways will allow developers some more latitude to do things that are not necessary or look slightly strange but really don't affect correctness, and the other way will do things extremely strict that will cause massive development problems because the person who wrote the tool is anal retentive.
Diagnosable: In the event that I do something wrong, or god forbid, the tool crashes, I should be able to figure out what the hell actually happened, and fast.

Websphere, amazingly, fails all 4 of these tests.

It routinely crashes on all of my development hosts and platforms just deploying my code. The crashes are non repeatable, and often silent. They usually appear to be instability in the jvm, but I don't give a damn, really. The appserver should eat generic webapps for lunch.
Deployment cycles are terrifyingly slow. The only way to deploy an application is a zipped war, and you have to use a webgui to update your application. This totally ruins any chance at keeping your train of thought. Updates of an application usually take 5 minutes from a code change to seeing it run, or more. I know there is a hot deploy mode, but it's hostile to use and has been severely unreliable.
I have never taken a jsp that works on another appsever, dropped it on websphere and seen it work. Websphere wont read mac line endings in jsps, but when it crashes, it tells you that your jsp tags are not properly closed. Complete crap. It also has the strictest, most retarded interpretation of the j2ee spec i've ever seen. Resetting the content type of your jsp page twice or more works perfectly on every other appserver. I know that's not right. I don't care, as long as its always set to the same string.
When it crashes due to bad file encodings, or simply using the console, it gives misleading error messagess or no error messages at all.

For shame. I previously held IBM in very high regard with respect to the quality of their java tools. My experience with websphere has totally ruined the IBM java reputation for me. They need to get their heads out of the sand and realize that the people working in the trenches USE their tool, and make buying decisions based on how hard it is to use the tool.

2003-07-19T23:29:00.000-07:00

File encodings suck, or dealing with localizing a real world app

If you're like me, you probably have to deal with bringing translated text (resource bundles) into your application. This means, you send your english property files off to another company, and they send them back to you in french, russian, spanish, korean and a truckload of other languages. The problem is, the translated files arrive from the (terrible) translators in any random file encoding that the translator felt like using.

That is, you get the files in some weird Latin-3 encoding and you need it in UTF-8. Except you don't know that it's in Latin-3, it just looks like a bunch of garbage because you don't have fonts to render greek on your box. What is one to do?

The two things you need, if you're dealing with java, are iconv and native2ascii .. native2ascii is the tool that converts from the binary utf-8 encoding to an ascii encoding with escapes, so it looks like \uXXXX whenever there should be a non-ascii chacter. native2ascii comes with the jdk, so you've already got that. iconv or piconv are available under cygwin or any linux distro, and both work equally well. Remeber, in the end, you're going to end up running the file thru native2ascii before you try it in the app, anyway, because that is the only format that java can read natively.

Remember, when you're working on a moden linux machine, vi is UTF-8 clean, and on windows, notepad is UTF-8 clean. So what you need to do is using iconv or piconv to convert the file from whatever encoding you guess it is to UTF-8, and then open it with vi or notepad. If it does not look corrupt, you're in business. At that point, run native2ascii -encoding < file > file.java.ready, where is the encoding you used with iconv to produce the file that looked ok in notepad or vi.

The homebrew heuristic to detect the corrupt files that were converted with the wrong -encoding switch with native2ascii, based only on experience, is as follows: Pick a file that has a sequence like this [ascii character][non-ascii character][ascii character]. Take the file and run it thru native2ascii, with the source set to a your encoding that you think is right. If the output looks like [ascii character]\uXXXX[ascii character] then you've found the encoding. However, if there are more than 1 \uXXXX sequences between the ascii characters, your encoding is wrong and you need to try again.

Of course, if you had a hex editor and a copy of the code pages on hand, you could determine this much less heuristically, but I can never find either of those when I need them.

2003-07-18T22:36:00.000-07:00

Relearn how to type, it's worth your effort; gesture keyboards rule

I've recently switched to using a Fingerworks gesture keyboard. It takes a lot of effort to re-learn how to type (how well can you type on a flat table?). Once you throw in the 3 weeks of retraining, however, the boost that the gestures support gives your development is amazing. Imagine making the "hang loose" sign with your left hand and your ide does autocompletion. Then you slide around your right hand, and without leaving the keyboard, you're jetting around using the mouse, arrow keys at lightning speed with your left hand, and simple gestures that trigger extemely complex application level macros.

The biggest challenge is not learning how to gesture or use your mouse, though, its learning how to type. I have yet to switch over to the fingerworks keyboard for my main device, just because my programming on it is like 1/3 speed of my normal development, which personally makes me want to throw the computer out the window. But I am getting faster, and I can see the day where my Kinesis will go up on ebay and I'll be that much faster and efficent when I work.

2003-07-18T22:29:00.000-07:00

"COTS" Open Source: Practically Delivering on the COTS vision

I've recently been putting in a lot of work building a custom web application framework for internal product work at my organization. Largely, this effort has focused on using off the shelf components, leveraging their strengths and implementing the custom framework as a layer of glue between those components to integrate them into a platform. Some would say "duh" and claim that is the whole point of using off the shelf software. And I do not entirely disagree.

The exciting bit here is that this is actually real, and the integration can happen in such a fashion that things are actually nicely coherant and integrated as a tools platform without having to spend a fortune on customization of the invidual tools from vendors, and deal with the fact that tools don't meet your requirements exactly. Instead, just having the ability to look inside the tools and work around issues, work around features that are getting in your way, you can very quickly assemble something that will rocket you ahead in your development effort.

The tools that I have glued together are jStateMachine (no longer publicly available), Hibernate and a collection of open source libs (dom4j, etc), using Eclipse UML as a charting tool. This has given us a system that lets you draw uml statecharts that describe your app, run them immediately in the appserver and cleanly drop in dhtml renderers for your data that are decoupled from the backend logic.

2003-07-08T15:48:00.000-07:00

Scrollable Results & Hibernate

I was looking for a method to do nice, database indedendent paged results without sucking the whole result set out of the database. Just found out from crazy bob that scrollable result sets don't work the way they "should", mostly because the drivers suck. So much for that idea.

However, it turns out that Hibernate (which I am looking to use) has an Oracle9 dialect that understands the pattern of using a nested select and the Oracle ROWNUM pseudo-column. Here's to the Hibernate developers for a clean solution.

2003-07-08T15:40:00.000-07:00

Plugging some amazing effort: Reality Interactive

I'd like to take this opportunity to plug Reality Interactive and my buddy Rob. He's done some amazing things with the 1.4 JDK, and, despite the fact that it looks like he rewrote a game that's several years old, he's done something infinitely cooler. The promise here (which he has shown proof that he can realize) is that simulations (i.e., games or things like flight simulators) can be written much faster, much more reliably in Java than they can in C++. Over and above the boost of using a better language that's more suited to the task, he's built a toolkit that brings a lot of the hard won experience of the enterprise java world to simulations. This lets people do things like focus on actually writing their physics engine rather than invent another architecture that will be discarded in 8 months.

3 cheers for some real innovation.

2003-07-08T15:35:00.000-07:00

Why did you make us wait for this?

To echo the sentiment of a thundering crowd: Is this really what we were waiting for?

2003-06-12T20:21:00.000-07:00

Weblogic, Container Managed Transactions, and Connection Pooling: Why you need the JTS driver

Long winded intro
So, to start this off: I've been working on a VERY large system for the last year or so that runs on BEA Weblogic. It uses EJBs in container managed transaction mode, because that's what "everyone" says to do. There are, of course, benefits to having transactions and avoiding a commit on each single db operation, so great, I said, let's use them. Did I mention this was my first weblogic application? Did I mention it's almost a quarter million lines?

So we built the system, and the whole time we were building it, it was always slow. Some of the lessons I learned about Oracle and database connections still leave my shaking my head. Anyway, we kept getting faster. And faster. But faster will still slow as dirt, quite honestly. I'd built a ton of apps that ran on other appservers, and they were always TONS faster than this. I was left scratching my head until I wore a bald spot in my hair. I mean, I'd tuned the code, and tuned it and tuned it. I was having halucinations of JProbe UIs, and it would not speed up.

So I was getting into quite a tight spot, as the customer was not able to actually USE the app, it was so slow. As it was more and more important that I actually fix the problem, I sat down one afternoon and read the "Container Managed Transactions" section of the weblogic docs yet again. And I noticed a weird footnote, that said something to the effect of "A transaction should use a single JDBC connection, as any connection that is used by the transaction, even if closed, remains associated with the transaction until it commits."

I sat straight up in my chair and grabbed the desk, as I thought I might faint.

The chase
Long story short: I was following the same pattern I (and probably lots of others) had always used, which is:

try {
    connection = dataSource.getConnection();
    connection.executeXYZ();
} finally {
    connection.close();
}

This would happen for each db operation I needed to do in the transaction, so what it turns out I was doing was using a HUGE swath of database connections (15-30 sometimes) for each transaction. Then, when I committed the outer TX, the Weblogic transaction manager would have to run a commit on each individual connection that was attached to the transaction. Which makes it yet slower. And the fact that those connections are unavailable to the server during the whole TX makes it yet slower still, as the connection pool was artificially exhausted by only a small number of users. The fix was to go from "connection per query" to "connection per transaction", which, if done the brute force way, is a massive code change (everywhere you currrently fetch a connection, you need to have it passed in). Turns out the fix was trivial if you do it using a vendor proprietary solution. In this case, proprietary saved my bacon.

The answer to my (and maybe your) ills is called the "Weblogic JTS Driver". You can read more about it here. What this does is change your API from using a DataSource to an old fashioned Driver, but it, under the hood, uses your connection pool and makes sure your single transaction always uses the same connection. Suddenly, my application went from crashing with 8 users, to handling a solid pounding by 60. And it was miller time.

2003-06-11T17:21:00.000-07:00

Why do open source projects ignore me?

So here's just a thought for a cloudy wednesday afternoon: I've written various chunks of code that I actually want to contribute back to open source projects over time, specifically to apache ant, and I've always gotten completely ignored. Now, if I were a neophyte and writing and sending in stupid hello world code snippets, I would expect this. However, I'm a professional, contributing back code that I use every day in production that could be useful to other people. And not even a "thanks but no thanks" email in response to my submission.

Normally, I think most people would feel shunned and walk away rebuffed in some way because these other "super cool" programmers "won't email me back". I, on the other hand, think that these projects are basically shooting themselves in the foot, because I (and I suspect, many other talented programmers) are getting ignored when they try and contribute, and so they just basically go "aw, fuck it anyway" and never bother again because the team is too lazy or buried in their own hubris to respond to code submissions. I guess this is just a bit of a lament, and, if any open source project leaders are listening, a warning: ignore people who are trying to contribute at your own (and the rest of the industry's) peril. Open source lives and dies on the goodwill of the developers, and if you don't breed good will, you can't grow.

2003-05-19T02:10:00.000-07:00

Insomnia yeilds cool IntelliJ tricks

First off, the IntelliJ plugin registry contains tons and tons of plugins. I do have to say the presentation is poor, and its difficult to tell what is a "real" worth my time plugin and what is basically a toy. Which is what took me so long to find this stuff. So the revolutions are:

Perforce Direct Plugin: This is the single greatest perforce integration I've seen thus far, tho I am not up on where p4eclipse is these days. This is a must if you use perforce with IntelliJ, and the only p4 integration for it (besides the one it ships with) that works out of the box.
Tabifier Plugin: It turns one someone else used to use emacs, and misses having their = signs lign up when they declare variables.
Log4j Monitor: Jump a few hoops and you can see and navigate your log4j output ala the chainsaw tool inside intellij, with the bonus of being able to adjust logging levels at runtime.

2003-05-18T22:33:00.000-07:00

Two finds for a sunday

The first thing is The Peon's Guide To Secure System Development, which I just re-read while surfing my bookmarks. This wasn't anything I didn't already know, but I definitely can say that I really like the way the guy writes and the message he gets across. He and I are of a single mind.

The other thing is: Cygwin users, it's time to stop using that damn DOS box!
The trick is:

Install/upgrade to the latest cygwin
When doing the update, in the setup.exe, look under Shells and select "rxvt"
After the app is installed, from a cygwin (or dos) shell, run C:\cygwin\bin\rxvt.exe -fn "Courier New-14" -fg grey85 -bg black -e (path to your shell) -l

I have this setup as a shortcut to start my shell on my box, and it's heavenly. I finally have my xterm back!

2003-05-16T21:18:00.000-07:00

Heuristic resource leak checkers

Lui Sha's Law: Engineering is about solving the problem that can be solved.

The above is a rough paraphrase of something Lui, my advisor and mentor in grad school, used to say reguarly. Lui is a damn smart guy, and that has really stuck with me over the years. If you try and solve the general problem of finding things like JDBC leaks, EJB leaks or any other technically "intractable" problem, you're just gonna end up stuck in the weeds digging for years with no answer. I'm certainly not smart enough to solve the halting problem, and my guess is you're not either.

What you can, instead, do is look for the pattern of how you fetch and release database connections in your system, using your frameworks and code standards. If something doesn't match the pattern, it's probably a leak, but if nothing else, it's definitely something odd that doesn't match the rest of the system and should be looked at.

I built an EJB leak checker for the system I'm currently building that scans for things that don't match how we fetch and release EJBs. Thus far, it's found something like 99.9% of all of our leaks, if not 100% (damn halting problem.. I can't tell!). Certainly the before and after of runninging it shows a much more reliable application afterwards under load. It works so well we integrated it into the build, and the build halts if there's a probable leak. If you can do this for your system, I fully encourage it. It saves on JProbe licenses :-)

2003-05-16T21:01:00.000-07:00

A moment to reflect on your ergonomics

Kris's Law: When you're in pain because of working, you won't want to work.
Kris's Law Corollary #1: If something is a hassle or complex to do, and you have to do it more than once, you won't do it, you'll do it wrong or very half-assed.

The general principal is pretty simple. I included the corollary just because I needed to write it down. From an ergonomics perspective, causing yourself pain can happen a myriad of ways. The way I'm currently fighting this is how to deal with correct monitor position on multidisplay machines. If you have it wrong, you cause yourself neck, shoulder and back pain as you twist your body in funny ways to look at things that are at some weird angle to your body and normal position. If this gets bad enough (and, yes, like everything else, I took this to a pretty distant extreme) you'll really fuck your body up.

So: If you start to hurt, see a doctor. And start moving around your monitor, chair, etc, to fix it. Don't let it go on, or you'll regret it later.

2003-05-15T23:58:00.000-07:00

Weblogic and exploded ejb deployments

Several things I discovered today:

If you deploy an exploded ear file, you must deploy all of the ejbs exploded or all of them jarred, otherwise the deployment will choke, as it is hard-coded to have all of them in the same packing format.
If you have some ejb jars pre-compiled and some that have not been, this will generate random results, since the first jar that ejbc happens to hit that is precompiled, it will stop. This can be worked around with an undocumented deployment order switch in the config.xml, but does not work reliably, and will typically have to fail once for you to generate the config.xml entries that you can then tweak so they work. The undocumented trick is to force the ejb jars that are not precompiled to be first in the deployment order, and the compiled ones all with the same priority and last.
Ant does string replacement based on literals, it does not normalize / and : in path-like strings. So if, for instance, you're using a < pathconvert> element that is trying to replace project paths with "" or other strings, you have to not use / or \ in project paths, but instead refer to ${file.separator} when building the output paths.

2003-05-10T11:05:00.000-07:00

Apache `mod_proxy`, your appserver and slow clients

Weblogic (and other java appservers) operate in a thread-per-client mode, which means that a single client that is getting a response from your appserver consumes a thread on the server side, and until the message is completely delivered, this thread cannot service any other clients. Thread per client is both a design choice by the original J2EE designers, and a necessity due to the fact that Java before 1.4 had to have a thread for each client because the way the I/O model was structured.

Under lab conditions where you have fast clients and very low network latency, having each client consume a thread on the server is totally fine, because the time to send the response to the client is negligable. This allows threads to quickly recycle after they finish their work cycle.

The problem is, this is actually the best case scenario. Under real-world workloads, where clients are slow and packet loss is rampant, what ends up happening is your server bogs down trying to send data to slow clients. So if one client is on a 56K modem and everyone else is on a T1, that 56K modem will actually consume a thread and impact performance for all the rest of the users on the appserver. The thread that sends the slow client event static content will stay stuck and unable to do useful work until the download completes. This has a very real impact on the server as a whole, because the thread pool will quickly exhaust and people will queue and wait for threads that can service them.

Originally, before I figured out what was going on, I was boosting the thread pool size on the server to rediculous levels (well over 150) just based on trial and error results from remote load testing across the internet. I had discovered that the higher the thread pool numbers, the faster things seemed to be for people. However, huge thread counts start causing the operating system problems, primarily in scheduler overhead trying to properly do the run queue.

This is where mod_proxy comes in. This apache module is written in C, has internal response buffers and is intended to deal with tons of slow clients without eating tons of critical OS resources, like threads (which, unfortunately, is not possible before JDK 1.4). If you put it in front of your application, suddenly, when the threads complete their work cycles, they return the data to the proxy and immediately are ready for service again. The throughput of your server suddenly skyrockets because the java side of things is actually spending time doing useful work, rather than buffering data to slow clients.

And, just for your boot notes, the Apache mod_weblogic that comes with the appserver does not release the appserver-side resources until the actual client completes. This is just based on observation of server operation and not empirical testing, and should be further verified, but there is STRONG evidence that mod_proxy makes much more efficient use of your appserver resources.

From the Apache mod_proxy documentation (thanks Igor): When a response is received which fits entirely within the IO buffer size, the remote HTTP or FTP server socket will be closed before an attempt is made to write the response to the client. This ensures that the remote server does not remain connected unnecessarily while the response is delivered to a slow client. A high value for the IO buffer decreases the load on remote HTTP and FTP servers, at the expense of greater RAM footprint on the proxy So long story short is: make your I/O buffers LARGE in your mod_proxy to take the heat off your appserver. Reference for mod_proxy here

2003-05-05T17:37:00.000-07:00

Disable and enable GC mechanisms

I just found out that when you enable a GC mechanism (for JDK 1.4) you have to explicitly disable the mechanism you were previously using. Thus, to actually have your app only use the ParallelGC mechanism, you must set TWO switches:
-XX:+UseParallelGC -XX:-UseConcMarkSweepGC
to both turn ON the parallel GC, and turn off the Mark/Sweep algorithm. Also, you need to adjust:
-XX:ParallelGCThreads=N
to match your system, where a "good" value of N is typically 8

2003-05-03T11:45:00.000-07:00

XDoclet for rapid EJB changes

If there's one thing I've learned in the past year, it's that any little barrier that you have between you and doing something makes it so it doesn't happen, especially in development when the timelines are short and the pressure is high. This includes having to call people that aren't in the office, and running static leak checkers on your code outside of the build. Or testing your code because it has to be inside the appserver to run.

Normally, to add or modify an EJB, you have to update 4 files: The Remote interface, the Bean implementation, and potentially the ejb-jar.xml and weblogic-ejb-jar.xml files. To create an EJB, you also have to make a Home interface. All of this is really quite useless from a productivity point of view. I've been through several revisions of code generators to try and solve this problem, and none of them really worked ALL that well. You may cut the 4 file edits down to 2, but 2 is still too many, and causes a huge pain when things get out of sync. This leads to your ejb code being a total mess.

Enter XDoclet: The correct approach. This is a code generator that is driven not from an external document, but from actual annotations in the sources. This lets you edit (or create) one file for your ejbs, and the system handles the rest. Combine this with an IntelliJ macro, and suddenly the business of making EJBs changes dramatically.

2003-04-30T19:33:00.000-07:00

Turn logging on and off at runtime

If you're using log4j like the rest of the java world seems to be now, there's two really great tricks:

Turn logging on and off dynamically: If you call PropertyConfigurator.configureAndWatch("log4j.properties", 20 * 1000); from a servlet that is set to load on bootstrap in the init method, every 20 seconds it will reload the log4j.properties properties file in the cwd of your program (which, for weblogic is the wlserver6.1 or weblogic81 directory). This file doesn't have to exist at startup, and will be read if it is created. This lets you ship a canned log4j.properties with your app, then drop a new file in this directory that turns on DEBUG for parts of your app that you need to diagnose at runtime. The only thing to note here is that you need to explicitly set things back to INFO to shut off DEBUG at runtime, simply removing the config line from the dyanamically loaded file will not reset anything.
Disable line numbers: This is a trick I found from someone else's blog, so I don't take credit for this one. The way log4j finds line numbers for log messages is by actually parsing a stacktrace from a Throwable. So if you want a major boost with logging on, remove the line numbers from your output formatters and things will pick up dramatically.

2003-04-30T19:23:00.000-07:00

Send your appserver temp dirs to a second hard disk

If you have a second disk in your machine, your appserver startup time can be dramatically reduced by sending the generated output to a second disk. There is a "bug" with weblogic where when you're running an exploded application that it copies all of the files out of the normal application directory into the .wlnotdelete directory during startup.

To do this you need junction from sysinternals, which lets you create directory hard links in NTFS filesystems. Create a directory on your second disk called wlnotdelete, and then junction the directory config/mydomain/applications/.wlnotdelete to the wlnotdelete on your second disk. I also did the config/mydomain/.wlstaging. This VASTLY reduced my startup time; when the appserver starts now, it just does a big cpu burst until it's started, instead of hanging doing tons of disk I/O for a long time. The same appears to be true for hot deploys.

One thing that holds a lot of potential is sending your the generated files to a ramdisk. Unfortunately, my application is far too huge (300M in my .wlnotdelete) to make this feasible, but if your app is small enough, this is supposed to be a huge boost. You can get a ramdisk for XP for free here

Post Script: A friend (hi GP) read this and suggested: "rather than doing it in the application, if you are running on solaris, you will make it go even faster if you turn off the UFS write throttle (or set the watermarks to something freaking absurd)." .. I am quite interested in trying this, and seeing if the general concept ports to other platforms.

Post Post Script:: The above trick, with basically altering the UFS write buffers to be HUGE, works great on Solaris, but there is no logical analog on Windows. One of the design constriants for NTFS is that there are operations that must be guaranteed to be persisted out to disk before the write() method returns to the caller, which is true for things that change filesystem metadata. So there's no buffer size that I can find to alter that makes any difference for this case.

2003-04-30T10:37:00.000-07:00

Another approach to the application server db throughput problem

Just was sent a link to c-jdbc by a coworker (thanks Igor, again), which is an open source jdbc solution to move an existing jdbc application to using a cluster of dbs. Right now, this is focused on open source solutions (JoNAS and PostgreSQL), where it makes an extreme amount of sense. This isn't as much of a great idea for Oracle, as the license fees for the extra dbs would kill you.

2003-04-29T23:29:00.000-07:00

More on weblogic hotdeploy: now that it's reasonably fast, make it work most of the time

So there's another hidden gotcha to weblogic hot deployment: it tends to crash the appserver. Great, huh? You dangle the carrot of not having to spend 5 minutes picking your nose every time you compile, then take it away. But don't give up hope, if you're running the right service pack, it is possible to patch your code (yes, your code must be patched in order to make their code work) so that hot deploy can actually work.

What I'm speaking to is mostly working around several bugs in weblogic 6.1. The issues are:

Weblogic 6.1 Service Packs < 4 often just don't work. There are crash bugs in the hot deployment in, I beleive sp3, that just make it not an option. Upgrade to sp4 on your development boxes. Hopefully 7 and 8.1 fix this.
There are various memory leaks in the hot deploy code. A way to work around this (or at least to delay it) is to set -XX:MaxPermSize=128m in your JAVA_OPTIONS in the startWebLogic.sh
Servlets don't un-deploy due to a crash in the Weblogic GenericServlet destroy method. This is where you get to change your code to deal with their bug. You need to edit all your servlet subclasses and override destroy() (if you are not already) and be sure that after whatever processing YOUR destroy does, it does not ever call super.destroy(). The easiest solution is just to implement it as an empty method.

2003-04-29T21:11:00.000-07:00

JCache clustered caches: they're not really a database

So something that's becoming hot, and is definitely coming as a mainstream j2ee development toolkit, is to use cluster-coherent caches in your j2ee apps. If not the oldest trick in the book, caching has to be the second oldest. The problem comes when your app is expected to run across a cluster of hosts. Caches that get out of sync and start returning inconsistent results are the things of my nightmares. This is where JCache comes in. You can read more about it in JSR 107, which is an attempt to make a vendor neutral JCache API.

So I've been using the JCache product from my friends over at Tangosol (shouts out to Cameron, Alex, Rob and all the gang). All told, great product. It can do all sorts of neat things, like call notification methods when you put data in the cache, etc. Great. And it can either replicate data across a cluster, or distribute it and keep a sortof L1/L2 cache.

The thing I learned today: Attempting to keep things so they MUST be in cache, and having your application be wrong if things are not in cache (such as if you run out of memory) is a terrible idea. It leads to bugs like you'd never beleive, because bad things happen to good people, like running out of memory. It's far better to sometimes be forced to go to the data store (SQL) and incur the network round trip penalty than it is to try and be clobbered when something expires.

This is not to say that JCache (especially Tangosol) don't do a servicable job at doing most of the things a database does (including distributed queries and indexes). However, if you use them as more than a best effort medium, you really have to hone your development model to using them so you don't get bit.

2003-04-29T13:43:00.000-07:00

Weblogic Hot Deployment: How to waste less of your day waiting to test your code

My biggest frustration, and that of many people (I suspect) is the terrible documentation that BEA put out with releases before 8.1 (and possibly even after, I haven't dug fully into those docs, tho they do look better), and actually understanding how the appserver worked enough to make it so that it didn't basically kill their development process. The problem I am referring to is hot deployment, and how the appserver, every time you change something, has to redeploy your code. There is infinite value in being able to immediately test the change you make. When you work with Weblogic, though, you have to get your code inside the container before it will run.. accessing the ejbs, etc. This basically is death for development, as it makes unit testing things painful, etc. I'll go into how I've learned to live with this over time later.

The thing that I've been toiling with today is how to speed up the hot deployment process, which is basically how you can get your code into the appserver without actually restarting it. Restarting means restarting database pooling, and 3,000 other things that means your code won't be able to be run for literally several minutes after the build completes. One of my coworkers said he reads a Wall Street Journal article every time he compiles and needs to test something. Obviously, I don't like having to pay people to read the newspaper. I should add up the amount of time we've wasted doing builds and back bill BEA for our wasted development time.

ANYWAY: The point of this all is if you deploy as an exploded directory structure if you don't want things to just BITE. There's detail on this here, which basically tells you how to unpack things.

There are two things about hot deployment:

Web Application Hot Deploys
EJB Hot Deploys

The first will work basically out of the box if your app is exploded. Change the JSP in the directory and hit f5 in IE and it's updated. Boom (thanks Igor!). Almost like you're working with resin, except 10x as expensive. The second one requires actually telling the server to do a forcible deploy, which involves touching a file called REDEPLOY in the META-INF directory of your EAR, which, on my PC, takes about 30 seconds and redeploys the whole app.

The thing I spent the time with today is looking to see if it was possible to just redeploy the ejb-jar. The fast answer to that was: Weblogic (6.1 thru 8.1 it seems) can only redeploy the whole app. So if you deploy as an ear, you have to redeploy the whole thing. If you deploy as an individual ejb-jar and a war, you can redeploy them individually. My next task is to try moving to separate deployments and see if it speeds up the ejb reload process.

Also, you may find info online that says touching the ejb-jar.xml file in the ejb META-INF directory will cause the ejb-jar to redeploy, bu this is outdated. The only thing that will cause a redeploy is touching the META-INF/REDEPLOY file.

2003-04-29T00:03:00.000-07:00

Boost your interactive (and enterprise) java performance

If you're a common user of interactive java applications, such as the IntelliJ IDEA IDE, there are some great VM flags that you can use to boost your interactive application performance. The two, in particular that have helped me the most, are:

-XX:CompileThreshold=5
-XX:+UseParallelGC

The -XX:CompileThreshold sets how many times the JIT has to run a method or block before it actually produces compiled code. By default this is a huge number (15,000). Cranking it down to something very small (ie, 5) forces most of the application to compile when the system starts up, which causes a boost of about 30-40% after an initial startup delay. The -XX:+UseParallelGC helps eliminate some of the GC hit (pauses) with JDK 1.4 by making it run concurrently. Both of these tips are actually really handy for enterprise applications as well, where runtime speed is important, but the startup cost can be easily paid ahead of time.

For IntelliJ, as well as any LaunchAnywhere system, you set these flags by editing a .lax file and changing the lax.nl.java.option.additional line. The lax file for starting intellij is in the IntelliJ-IDEA-/bin directory.

2003-04-28T17:02:00.000-07:00

Boycott netgear access points

Never, ever buy one of these access points: netgear me 102.
This thing has rarely worked, and now has maigcally changed its SSID to "DORA" and won't reset with the PC configuration utility.

I've been perfectly happy with my linksys befw11s4.

2003-04-28T16:51:00.000-07:00

Ok. This is blogspot try #2. I mostly wanted to start this blog so I could bitch about software, and tell people what I've learned so they can avoid the pain I've had to go thru. Ironic that the blogspot software screwed up and failed to create my blog properly. If you're wondering (my first hint), if you have an account on blogspot, agree to be a beta tester.. the beta test software works (while the "production" code crashed for me). Sad that that is my first blog entry.