I have a Java program that must do the following 3 things:
- Download a file from a website.
- Run the file trough testA and testB (both in java)
- Delete the file and save the test results on the disk.
That is done for roughly 1,000,000 different sites. It was supposed to be a fairly simple task, since I simply glued parts from other programs: testA
and testB
both already executed separately for dozens of millions of different pages without trouble, and the routine that download the pages has also been executed for a million or so pages some times, and never had any problems either. They are all being executed on a Ubuntu 10.4 machine.
However, when doing those 3 one after the other, whichever disk the files were being written to crashes. The first time I ran it on a External USB HD, which I had to manually disconnect and reconnect for it to resume operation (Linux wouldn't recognize it otherwise). Next time, on an Internal HD, the whole system went down, and I had to manually restart it. The same happened when writing to a Ram Disk.
The problem is that I can't really isolate the problem. It takes way too long for a crash to occur (about 50 hours or so, but it is quite random), so testing takes too long, and there are no system logs of the failure indicating where/how it happens. The machine or HD simply stops responding.
Except for the crashing, everything works fine. Files are created and deleted normally, threads don't die and execute properly, and both tests work alright. Changing memory or the number of threads had no effect on lockup time. I already checked for Sockets or the like not being closed, but I don't even know how to start testing, I had no idea crash a system so catastrophically would be possible with Java.
EDIT: By hangup I mean that, when I run it on an external HD the HD won't get recognized by Linux, and when I run it at the internal HD or a Ram Disk the computer won't respond to any I/O whatsoever, with nothing being written to the disk, cactii logs not being recorded, etc. Can't connect using SSH, for instance.
A example of how the program runs:
List<String> pagesToDownload = getFromDataBase();
for(i=0;i<NumThreads;i++){
launchTestThread();
}
And then, on each thread:
String pageName = getNextPageToDownload();
File downloadedFile = downloadPage(pageName);
TestAResults testAResults = runTestA(downloadedFile);
TestBResults testBResults = runTestB(downloadedFile);
writeToDatabase(downloadedFile, testAResults, testBResults);
downloadedFile.delete();
Individually the functions runTestA
, runTestB
and downloadedPage
work for even larger amounts of files, but when called that way, they don't. And that on the very same hardware.
EDIT2: I think I ruled out the problem being the hardware. A just as hardware intensive software has been running at the very same machine for the last 7 days without any problems. Anyway, as soon as I can get a unused machine I'm gonna test the program in it.
Also, everything on the test is being written to the database up to the point where the crash occurs, and the data is correct. The downloadedFile
ain't passed as a parameter during the writeToDatabase method, just it's name and size.
Finally, I did some extensive checking for memory or file handlers leaks and turned up none, including inside the working tests. Right now, my money is on some strange bug on the file deletion.
EDIT3: I finally managed to get another machine where to test the routine. Another Hardware, but same Ubuntu Version (10.4 LTS). And it crashes there too, so I really doubt it is a hardware problem. That leaves either an OS bug, a JVM bug or a programming bug (there is no JNI or anything like it running). I'm gonna try running the test in some other environment (setting up a test in FreeBSD will be pretty easy, and I can try to find an Windows machine to test that) in order to verify that.
EDIT4: Answering Bob Cross' question about how big the files are, they are typical web pages, with an average of about 20kb. I gotta delete them since the idea is to expand the application, making the disk usage unbearable. But I'm gonna try a Deletion free run as soon as I can. The machine where I was running those tests is being used right now, and I'm having a hard time getting some idle hardware.