There isn't nearly enough information in your question for me to be fully helpful. However I can give you some pointers for your prefork configuration.
First, are you sure your httpd processes weigh in at 10MB across the board? If so, that's great, but that is much lower than what I have often seen in practice, which is anywhere from 25MB to hundreds (mod_php).
Assuming your httpd processes really are 10MB, your ServerLimit and MaxClients settings will permit Apache to spin up 1000x10MB workers. That's 10GB of RAM. Your machine does not have this, and it will swap. If your machine is swapping then you are essentially dead in the water, and this could be the explanation for the slowness you see.
(Use top, free, and vmstat commands to view swap usage)
Now your other settings StartServers means you are spinning up 3GB worth of apache workers just when you start Apache. On a 4GB machine, this is very likely not what you want. Depending on the footprint of your operating system you could easily be making the box swap just with this.
Knowing absolutely nothing else about your application I would suggest trying the following settings instead:
<IfModule mpm_prefork_module>
StartServers 10
MinSpareServers 10
MaxSpareServers 50
ServerLimit 200
MaxClients 200
MaxRequestsPerChild 9999
</IfModule>
If this is insufficient and you see requests start to get dropped due to apache hitting the ServerLimit you can try increasing this a bit, but I would not go over 300. If this is still insufficient, then you need to scale either horizontally (more web servers) or vertically (more RAM).