Insomnia yeilds cool IntelliJ tricks

First off, the IntelliJ plugin registry contains tons and tons of plugins. I do have to say the presentation is poor, and its difficult to tell what is a "real" worth my time plugin and what is basically a toy. Which is what took me so long to find this stuff. So the revolutions are:

1. Perforce Direct Plugin: This is the single greatest perforce integration I've seen thus far, tho I am not up on where p4eclipse is these days. This is a must if you use perforce with IntelliJ, and the only p4 integration for it (besides the one it ships with) that works out of the box.
2. Tabifier Plugin: It turns one someone else used to use emacs, and misses having their = signs lign up when they declare variables.
3. Log4j Monitor: Jump a few hoops and you can see and navigate your log4j output ala the chainsaw tool inside intellij, with the bonus of being able to adjust logging levels at runtime.

  ¶ 5/19/2003 02:10:00 AM

 

Two finds for a sunday

The first thing is The Peon's Guide To Secure System Development, which I just re-read while surfing my bookmarks. This wasn't anything I didn't already know, but I definitely can say that I really like the way the guy writes and the message he gets across. He and I are of a single mind.

The other thing is: Cygwin users, it's time to stop using that damn DOS box!
The trick is:

1. Install/upgrade to the latest cygwin
2. When doing the update, in the setup.exe, look under Shells and select "rxvt"
3. After the app is installed, from a cygwin (or dos) shell, run C:\cygwin\bin\rxvt.exe -fn "Courier New-14" -fg grey85 -bg black -e (path to your shell) -l

I have this setup as a shortcut to start my shell on my box, and it's heavenly. I finally have my xterm back!  ¶ 5/18/2003 10:33:00 PM

 

Heuristic resource leak checkers

Lui Sha's Law: Engineering is about solving the problem that can be solved.

The above is a rough paraphrase of something Lui, my advisor and mentor in grad school, used to say reguarly. Lui is a damn smart guy, and that has really stuck with me over the years. If you try and solve the general problem of finding things like JDBC leaks, EJB leaks or any other technically "intractable" problem, you're just gonna end up stuck in the weeds digging for years with no answer. I'm certainly not smart enough to solve the halting problem, and my guess is you're not either.

What you can, instead, do is look for the pattern of how you fetch and release database connections in your system, using your frameworks and code standards. If something doesn't match the pattern, it's probably a leak, but if nothing else, it's definitely something odd that doesn't match the rest of the system and should be looked at.

I built an EJB leak checker for the system I'm currently building that scans for things that don't match how we fetch and release EJBs. Thus far, it's found something like 99.9% of all of our leaks, if not 100% (damn halting problem.. I can't tell!). Certainly the before and after of runninging it shows a much more reliable application afterwards under load. It works so well we integrated it into the build, and the build halts if there's a probable leak. If you can do this for your system, I fully encourage it. It saves on JProbe licenses :-)   ¶ 5/16/2003 09:18:00 PM

 

A moment to reflect on your ergonomics

Kris's Law: When you're in pain because of working, you won't want to work.
Kris's Law Corollary #1: If something is a hassle or complex to do, and you have to do it more than once, you won't do it, you'll do it wrong or very half-assed.

The general principal is pretty simple. I included the corollary just because I needed to write it down. From an ergonomics perspective, causing yourself pain can happen a myriad of ways. The way I'm currently fighting this is how to deal with correct monitor position on multidisplay machines. If you have it wrong, you cause yourself neck, shoulder and back pain as you twist your body in funny ways to look at things that are at some weird angle to your body and normal position. If this gets bad enough (and, yes, like everything else, I took this to a pretty distant extreme) you'll really fuck your body up.

So: If you start to hurt, see a doctor. And start moving around your monitor, chair, etc, to fix it. Don't let it go on, or you'll regret it later.  ¶ 5/16/2003 09:01:00 PM

 

Weblogic and exploded ejb deployments

Several things I discovered today:

1. If you deploy an exploded ear file, you must deploy all of the ejbs exploded or all of them jarred, otherwise the deployment will choke, as it is hard-coded to have all of them in the same packing format.
2. If you have some ejb jars pre-compiled and some that have not been, this will generate random results, since the first jar that ejbc happens to hit that is precompiled, it will stop. This can be worked around with an undocumented deployment order switch in the config.xml, but does not work reliably, and will typically have to fail once for you to generate the config.xml entries that you can then tweak so they work. The undocumented trick is to force the ejb jars that are not precompiled to be first in the deployment order, and the compiled ones all with the same priority and last.
3. Ant does string replacement based on literals, it does not normalize / and : in path-like strings. So if, for instance, you're using a < pathconvert> element that is trying to replace project paths with "" or other strings, you have to not use / or \ in project paths, but instead refer to ${file.separator} when building the output paths.

  ¶ 5/15/2003 11:58:00 PM

 

Apache mod_proxy, your appserver and slow clients

Weblogic (and other java appservers) operate in a thread-per-client mode, which means that a single client that is getting a response from your appserver consumes a thread on the server side, and until the message is completely delivered, this thread cannot service any other clients. Thread per client is both a design choice by the original J2EE designers, and a necessity due to the fact that Java before 1.4 had to have a thread for each client because the way the I/O model was structured.

Under lab conditions where you have fast clients and very low network latency, having each client consume a thread on the server is totally fine, because the time to send the response to the client is negligable. This allows threads to quickly recycle after they finish their work cycle.

The problem is, this is actually the best case scenario. Under real-world workloads, where clients are slow and packet loss is rampant, what ends up happening is your server bogs down trying to send data to slow clients. So if one client is on a 56K modem and everyone else is on a T1, that 56K modem will actually consume a thread and impact performance for all the rest of the users on the appserver. The thread that sends the slow client event static content will stay stuck and unable to do useful work until the download completes. This has a very real impact on the server as a whole, because the thread pool will quickly exhaust and people will queue and wait for threads that can service them.

Originally, before I figured out what was going on, I was boosting the thread pool size on the server to rediculous levels (well over 150) just based on trial and error results from remote load testing across the internet. I had discovered that the higher the thread pool numbers, the faster things seemed to be for people. However, huge thread counts start causing the operating system problems, primarily in scheduler overhead trying to properly do the run queue.

This is where mod_proxy comes in. This apache module is written in C, has internal response buffers and is intended to deal with tons of slow clients without eating tons of critical OS resources, like threads (which, unfortunately, is not possible before JDK 1.4). If you put it in front of your application, suddenly, when the threads complete their work cycles, they return the data to the proxy and immediately are ready for service again. The throughput of your server suddenly skyrockets because the java side of things is actually spending time doing useful work, rather than buffering data to slow clients.

And, just for your boot notes, the Apache mod_weblogic that comes with the appserver does not release the appserver-side resources until the actual client completes. This is just based on observation of server operation and not empirical testing, and should be further verified, but there is STRONG evidence that mod_proxy makes much more efficient use of your appserver resources.

From the Apache mod_proxy documentation (thanks Igor): When a response is received which fits entirely within the IO buffer size, the remote HTTP or FTP server socket will be closed before an attempt is made to write the response to the client. This ensures that the remote server does not remain connected unnecessarily while the response is delivered to a slow client. A high value for the IO buffer decreases the load on remote HTTP and FTP servers, at the expense of greater RAM footprint on the proxy So long story short is: make your I/O buffers LARGE in your mod_proxy to take the heat off your appserver. Reference for mod_proxy here  ¶ 5/10/2003 11:05:00 AM

 

Disable and enable GC mechanisms

I just found out that when you enable a GC mechanism (for JDK 1.4) you have to explicitly disable the mechanism you were previously using. Thus, to actually have your app only use the ParallelGC mechanism, you must set TWO switches:
-XX:+UseParallelGC -XX:-UseConcMarkSweepGC
to both turn ON the parallel GC, and turn off the Mark/Sweep algorithm. Also, you need to adjust:
-XX:ParallelGCThreads=N
to match your system, where a "good" value of N is typically 8  ¶ 5/05/2003 05:37:00 PM

 

XDoclet for rapid EJB changes

If there's one thing I've learned in the past year, it's that any little barrier that you have between you and doing something makes it so it doesn't happen, especially in development when the timelines are short and the pressure is high. This includes having to call people that aren't in the office, and running static leak checkers on your code outside of the build. Or testing your code because it has to be inside the appserver to run.

Normally, to add or modify an EJB, you have to update 4 files: The Remote interface, the Bean implementation, and potentially the ejb-jar.xml and weblogic-ejb-jar.xml files. To create an EJB, you also have to make a Home interface. All of this is really quite useless from a productivity point of view. I've been through several revisions of code generators to try and solve this problem, and none of them really worked ALL that well. You may cut the 4 file edits down to 2, but 2 is still too many, and causes a huge pain when things get out of sync. This leads to your ejb code being a total mess.

Enter XDoclet: The correct approach. This is a code generator that is driven not from an external document, but from actual annotations in the sources. This lets you edit (or create) one file for your ejbs, and the system handles the rest. Combine this with an IntelliJ macro, and suddenly the business of making EJBs changes dramatically.  ¶ 5/03/2003 11:45:00 AM