VPS goes slow at more than 20 users online at the same time

Question

I have 512 MB VPS (brustable to 1GB)

Somehow, the site goes slow when there are about 10 users, and becomes impossible to load at 20 users online at the same time.

I wonder what could be the problem for this.

The bandwidth connection of the VPS is 1Gbps.

Here is some settings in my VPS:

KeepAlive Off
<IfModule prefork.c>
StartServers      7
MinSpareServers   7
MaxSpareServers   10
ServerLimit       64
MaxClients        64
MaxRequestsPerChild  0
</IfModule>

my.cnf settings -> calculated Max Memory 300MB

Output from UNIXBENCH

                  INDEX VALUES            
TEST                                        BASELINE     RESULT      INDEX

Dhrystone 2 using register variables        376783.7 13429727.4      356.4
Double-Precision Whetstone                      83.1     1137.5      136.9
Execl Throughput                               188.3     1637.4       87.0
File Copy 1024 bufsize 2000 maxblocks         2672.0   148868.0      557.1
File Copy 256 bufsize 500 maxblocks           1077.0    79430.0      737.5
File Read 4096 bufsize 8000 maxblocks        15382.0  1410009.0      916.7
Pipe Throughput                             111814.6  4419722.0      395.3
Pipe-based Context Switching                 15448.6   561505.1      363.5
Process Creation                               569.3    10272.7      180.4
Shell Scripts (8 concurrent)                    44.8      514.3      114.8
System Call Overhead                        114433.5  3537373.8      309.1
                                                                 =========
     FINAL SCORE                                                     295.0

I am afraid that the VPS company limit the number of connection to the VPS... is it possible?

The server is in Japan, but the site has global traffic (some of the traffic are from countries with low speed connection). Could this be the problem?

This is a serious problem :( my site just cant grow if this keeps on happening... please tell me if you have any idea.

Thank You, Bryant