I didn't know you could do that

We have a big pile of machines for running bulk jobs on. One of those jobs, which I just picked up, is our regression test suite. I got in this morning and discovered that the machine running the regtest was a bit, um, unhappy: about four hours into the test, it started leaking processes, at the rate of about 300 an hour, and slurping up huge amounts of memory. The system load peaked around 1000 at 5 AM, then dropped down to about 700 as the machine refused to do anything but swap. I finally wound up rebooting the machine a little before 11 AM when it started affecting other things.

Now, to figure out what the heck just happened...