Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am running an python multithreaded application with multiple processes which scrapes data from some websites. While running on my localhost It works great, but on the vps server I am using( Centos 5.8, 2.6 GHZ with 4 cores) performs very slow.

From the nethogs command I get the network usage too low. I get around 8KBps with 15 threads. On other hand, in my PC I get the usage around 100-120KBPS.

I have read about the Python GIL and threading limitations. It seems GIL never releases the lock on the VPS though it should while doing I/0

Is there any configuration in the VPS that I need to change for the threading to work properly?

UPDATE: Actually multithreading is working but it's the cpu that was causing the problem. 15 Threads were too many for it and it became too busy with the thread switching. Though the vps claims that it's 2.6 GHZ cpu, I think It's actually not. Is there a way to measure the real processor speed in the VPS?

share|improve this question
    
Are you using Scrapy? What OS you got on localhost? –  Timothy Makobu Sep 1 '12 at 15:06
    
Usinng a custom script I wrote. Ubuntu on localhost. –  Sabirul Mostofa Sep 1 '12 at 15:25

1 Answer 1

up vote 3 down vote accepted

You are almost certainly sharing the CPU with other VPSs on the same host, so you can't expect to get the same performance as a dedicated CPU. The GIL doesn't behave different on different CPUs, so that's not the cause. Use top to check the CPU utilisation on the VPS; you're mainly interested in the Cpu line like this:

Cpu(s): 30.2%us,  7.8%sy,  0.0%ni, 41.0%id, 20.8%wa,  0.0%hi,  0.2%si,  0.0%st

With a recent hypervisor and OS you should see a non-zero st number -- this is the CPU time "stolen" by other VMs on the same host, from which you can figure out what proportion of the CPU you are getting.

share|improve this answer
    
Thanks for the answer, It always shows 0.0%st. So, I guess I am not given the CPU speed promised. –  Sabirul Mostofa Sep 2 '12 at 2:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.