I have a proprietary document-oriented search engine software, and I'm trying to get the top performance I can out of it. Specifically I need to get maximum reading speed, writing is not needed at all, this is a read-only system.
First obvious thing I did is putting whole corpus of documents onto a RAM drive. It helped obviously. Also I've fixed it's settings as it was not using CPU properly (always one thread). However the top I got is about 1 second per request, while I need to have at least 0.25 sec/request.
The hardware is a dedicated HP server, 8CPU, 32Gb RAM. RAM can be increased if needed. The corpus of documents weights about 16Gb, all on RAM drive. This is a rare proprietary software, so no known indexing tricks or any proper documentation available.
If go to details I'm worried about FS productivity. I'm using ext2 on the RAM drive, and the system internally uses extreme amount of small files (I've ran out of inodes on RAM drive with default inode size). Can the reading overhead for multimillions of files be a problem by itself? What other filesystem can solve the problem? Or maybe there is something else I didn't think of that can help here apart from FS?
I'm also working on the balancing solution for this software, but again there is nothing ready out of the box, so I need to attach a thirdparty balancer to it, not sure yet if it will work fine this way (in theory it should).