The recalcitrant engineer
Turn logging on and off at runtime
If you're using log4j like the rest of the java world seems to be now, there's two really great tricks:
- Turn logging on and off dynamically: If you call PropertyConfigurator.configureAndWatch("log4j.properties", 20 * 1000); from a servlet that is set to load on bootstrap in the init method, every 20 seconds it will reload the log4j.properties properties file in the cwd of your program (which, for weblogic is the wlserver6.1 or weblogic81 directory). This file doesn't have to exist at startup, and will be read if it is created. This lets you ship a canned log4j.properties with your app, then drop a new file in this directory that turns on DEBUG for parts of your app that you need to diagnose at runtime. The only thing to note here is that you need to explicitly set things back to INFO to shut off DEBUG at runtime, simply removing the config line from the dyanamically loaded file will not reset anything.
- Disable line numbers: This is a trick I found from someone else's blog, so I don't take credit for this one. The way log4j finds line numbers for log messages is by actually parsing a stacktrace from a Throwable. So if you want a major boost with logging on, remove the line numbers from your output formatters and things will pick up dramatically.
Send your appserver temp dirs to a second hard disk
If you have a second disk in your machine, your appserver startup time can be dramatically reduced by sending the generated output to a second disk. There is a "bug" with weblogic where when you're running an exploded application that it copies all of the files out of the normal application directory into the
.wlnotdelete directory during startup.
To do this you need junction from sysinternals, which lets you create directory hard links in NTFS filesystems. Create a directory on your second disk called wlnotdelete, and then junction the directory config/mydomain/applications/.wlnotdelete to the wlnotdelete on your second disk. I also did the config/mydomain/.wlstaging. This VASTLY reduced my startup time; when the appserver starts now, it just does a big cpu burst until it's started, instead of hanging doing tons of disk I/O for a long time. The same appears to be true for hot deploys.
One thing that holds a lot of potential is sending your the generated files to a ramdisk. Unfortunately, my application is far too huge (300M in my .wlnotdelete) to make this feasible, but if your app is small enough, this is supposed to be a huge boost. You can get a ramdisk for XP for free here
Post Script: A friend (hi GP) read this and suggested: "rather than doing it in the application, if you are running on solaris, you will make it go even faster if you turn off the UFS write throttle (or set the watermarks to something freaking absurd)." .. I am quite interested in trying this, and seeing if the general concept ports to other platforms.
Post Post Script:: The above trick, with basically altering the UFS write buffers to be HUGE, works great on Solaris, but there is no logical analog on Windows. One of the design constriants for NTFS is that there are operations that must be guaranteed to be persisted out to disk before the write() method returns to the caller, which is true for things that change filesystem metadata. So there's no buffer size that I can find to alter that makes any difference for this case.
Another approach to the application server db throughput problem
Just was sent a link to
c-jdbc by a coworker (thanks Igor, again), which is an open source jdbc solution to move an existing jdbc application to using a cluster of dbs. Right now, this is focused on open source solutions (JoNAS and PostgreSQL), where it makes an extreme amount of sense. This isn't as much of a great idea for Oracle, as the license fees for the extra dbs would kill you.
More on weblogic hotdeploy: now that it's reasonably fast, make it work most of the time
So there's another hidden gotcha to weblogic hot deployment: it tends to crash the appserver. Great, huh? You dangle the carrot of not having to spend 5 minutes picking your nose every time you compile, then take it away. But don't give up hope, if you're running the right service pack, it is possible to patch your code (yes, your code must be patched in order to make their code work) so that hot deploy can actually work.
What I'm speaking to is mostly working around several bugs in weblogic 6.1. The issues are:
- Weblogic 6.1 Service Packs < 4 often just don't work. There are crash bugs in the hot deployment in, I beleive sp3, that just make it not an option. Upgrade to sp4 on your development boxes. Hopefully 7 and 8.1 fix this.
- There are various memory leaks in the hot deploy code. A way to work around this (or at least to delay it) is to set -XX:MaxPermSize=128m in your JAVA_OPTIONS in the startWebLogic.sh
- Servlets don't un-deploy due to a crash in the Weblogic GenericServlet destroy method. This is where you get to change your code to deal with their bug. You need to edit all your servlet subclasses and override destroy() (if you are not already) and be sure that after whatever processing YOUR destroy does, it does not ever call super.destroy(). The easiest solution is just to implement it as an empty method.
JCache clustered caches: they're not really a database
So something that's becoming hot, and is definitely coming as a mainstream j2ee development toolkit, is to use cluster-coherent caches in your j2ee apps. If not the oldest trick in the book, caching has to be the second oldest. The problem comes when your app is expected to run across a cluster of hosts. Caches that get out of sync and start returning inconsistent results are the things of my nightmares. This is where JCache comes in. You can read more about it in
JSR 107, which is an attempt to make a vendor neutral JCache API.
So I've been using the JCache product from my friends over at Tangosol (shouts out to Cameron, Alex, Rob and all the gang). All told, great product. It can do all sorts of neat things, like call notification methods when you put data in the cache, etc. Great. And it can either replicate data across a cluster, or distribute it and keep a sortof L1/L2 cache.
The thing I learned today: Attempting to keep things so they MUST be in cache, and having your application be wrong if things are not in cache (such as if you run out of memory) is a terrible idea. It leads to bugs like you'd never beleive, because bad things happen to good people, like running out of memory. It's far better to sometimes be forced to go to the data store (SQL) and incur the network round trip penalty than it is to try and be clobbered when something expires.
This is not to say that JCache (especially Tangosol) don't do a servicable job at doing most of the things a database does (including distributed queries and indexes). However, if you use them as more than a best effort medium, you really have to hone your development model to using them so you don't get bit.
Weblogic Hot Deployment: How to waste less of your day waiting to test your code
My biggest frustration, and that of many people (I suspect) is the terrible documentation that BEA put out with releases before 8.1 (and possibly even after, I haven't dug fully into those docs, tho they do look better), and actually understanding how the appserver worked enough to make it so that it didn't basically kill their development process. The problem I am referring to is hot deployment, and how the appserver, every time you change something, has to redeploy your code. There is infinite value in being able to immediately test the change you make. When you work with Weblogic, though, you have to get your code inside the container before it will run.. accessing the ejbs, etc. This basically is death for development, as it makes unit testing things painful, etc. I'll go into how I've learned to live with this over time later.
The thing that I've been toiling with today is how to speed up the hot deployment process, which is basically how you can get your code into the appserver without actually restarting it. Restarting means restarting database pooling, and 3,000 other things that means your code won't be able to be run for literally several minutes after the build completes. One of my coworkers said he reads a Wall Street Journal article every time he compiles and needs to test something. Obviously, I don't like having to pay people to read the newspaper. I should add up the amount of time we've wasted doing builds and back bill BEA for our wasted development time.
ANYWAY: The point of this all is if you deploy as an exploded directory structure if you don't want things to just BITE. There's detail on this here, which basically tells you how to unpack things.
There are two things about hot deployment:
- Web Application Hot Deploys
- EJB Hot Deploys
The first will work basically out of the box if your app is exploded. Change the JSP in the directory and hit f5 in IE and it's updated. Boom (thanks Igor!). Almost like you're working with
resin, except 10x as expensive. The second one requires actually telling the server to do a forcible deploy, which involves touching a file called REDEPLOY in the META-INF directory of your EAR, which, on my PC, takes about 30 seconds and redeploys the whole app.
The thing I spent the time with today is looking to see if it was possible to just redeploy the ejb-jar. The fast answer to that was: Weblogic (6.1 thru 8.1 it seems) can only redeploy the whole app. So if you deploy as an ear, you have to redeploy the whole thing. If you deploy as an individual ejb-jar and a war, you can redeploy them individually. My next task is to try moving to separate deployments and see if it speeds up the ejb reload process.
Also, you may find info online that says touching the ejb-jar.xml file in the ejb META-INF directory will cause the ejb-jar to redeploy, bu this is outdated. The only thing that will cause a redeploy is touching the META-INF/REDEPLOY file.
Boost your interactive (and enterprise) java performance
If you're a common user of interactive java applications, such as the
IntelliJ IDEA IDE, there are some great VM flags that you can use to boost your interactive application performance. The two, in particular that have helped me the most, are:
- -XX:CompileThreshold=5
- -XX:+UseParallelGC
The -XX:CompileThreshold sets how many times the JIT has to run a method or block before it actually produces compiled code. By default this is a huge number (15,000). Cranking it down to something very small (ie, 5) forces most of the application to compile when the system starts up, which causes a boost of about 30-40% after an initial startup delay. The -XX:+UseParallelGC helps eliminate some of the GC hit (pauses) with JDK 1.4 by making it run concurrently. Both of these tips are actually really handy for enterprise applications as well, where runtime speed is important, but the startup cost can be easily paid ahead of time.
For IntelliJ, as well as any LaunchAnywhere system, you set these flags by editing a .lax file and changing the lax.nl.java.option.additional line. The lax file for starting intellij is in the IntelliJ-IDEA-/bin directory.
Boycott netgear access points
Never, ever buy one of these access points:
netgear me 102.
This thing has rarely worked, and now has maigcally changed its SSID to "DORA" and won't reset with the PC configuration utility.
I've been perfectly happy with my linksys befw11s4.
Ok. This is blogspot try #2.
I mostly wanted to start this blog so I could bitch about software, and tell people what I've learned so they can avoid the pain I've had to go thru. Ironic that the blogspot software screwed up and failed to create my blog properly. If you're wondering (my first hint), if you have an account on blogspot, agree to be a beta tester.. the beta test software works (while the "production" code crashed for me). Sad that that is my first blog entry.
