Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 0 down vote favorite

I have a proprietary document-oriented search engine software, and I'm trying to get the top performance I can out of it. Specifically I need to get maximum reading speed, writing is not needed at all, this is a read-only system.

First obvious thing I did is putting whole corpus of documents onto a RAM drive. It helped obviously. Also I've fixed it's settings as it was not using CPU properly (always one thread). However the top I got is about 1 second per request, while I need to have at least 0.25 sec/request.

The hardware is a dedicated HP server, 8CPU, 32Gb RAM. RAM can be increased if needed. The corpus of documents weights about 16Gb, all on RAM drive. This is a rare proprietary software, so no known indexing tricks or any proper documentation available.

If go to details I'm worried about FS productivity. I'm using ext2 on the RAM drive, and the system internally uses extreme amount of small files (I've ran out of inodes on RAM drive with default inode size). Can the reading overhead for multimillions of files be a problem by itself? What other filesystem can solve the problem? Or maybe there is something else I didn't think of that can help here apart from FS?

I'm also working on the balancing solution for this software, but again there is nothing ready out of the box, so I need to attach a thirdparty balancer to it, not sure yet if it will work fine this way (in theory it should).

share|improve this question
1  
Only 1 rps on a RAM drive?!? Good lord, I'm sorry that you have to deal with this software. – EEAA Nov 29 '12 at 15:25
We're looking for a replacement solution, and we have a candidate at the moment, but the reintegration will take time and effort, while I need to launch the site with what I have now, so have to find a solution here. – user1450663 Nov 29 '12 at 17:31

closed as too localized by EEAA, Greg Askew, Michael Hampton, Scott Pack, Adrian Nov 29 '12 at 18:33

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, see the FAQ.