Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running two servers on Rackspace cloud one for the web app and one for the db and redis instances. The web server has 1Gb of ram and single core. Nginx sits in front of unicorn which is running 2 workers. I also have a sidekiq instance running. This configuration runs great and the servers generally hum along at very low cpu as the app hasn't been launched yet.

However, when I do a unicorn restart, let alone a full app deploy, all hell breaks loose. It looks something like this: This isn't a great graph

Basically my server gets wiped out for 3 minutes. It's somewhat responsive at times but monitoring is triggering downtime alerts all over the place (this is just zero-downtime restart).

If I do a full deploy, the graph spans about 8 minutes even though I'm precompiling assets and uploading, so no on-server compilation.

The interesting part for me is that I have an exact duplicate server setup running on DigitalOcean. I can completely restart that entire server shutdown -r and be up and serving pages in 50 seconds. With this Rackspace server I daren't restart it even to test as it would give me very significant downtime for a production server.

I'm not a Linux server admin, so I'm wondering if people can tell me if this is par for the course with Rackspace cloud servers. I've had a decade of experience running a few dedicated Windows boxes and never had any issues like this.

hdparm against the servers.

Rackspace:

$ sudo hdparm -Tt /dev/xvdc
/dev/xvdc:
Timing cached reads:   5066 MB in  1.99 seconds = 2541.54 MB/sec
Timing buffered disk reads: 238 MB in  3.00 seconds =  79.32 MB/sec 

DigitalOcean

$ sudo hdparm -Tt /dev/vda 
/dev/vda:
Timing cached reads:   15612 MB in  1.99 seconds = 7828.02 MB/sec
Timing buffered disk reads: 1416 MB in  3.00 seconds = 471.89 MB/sec

Obviously the DO server is outstripping the RS server by a significant margin. Interestingly enough, the DO server is actually staging two apps so is doing more work than RS one. Both hdparms run with server load about the same (ie very little). Is this purely slow disk speed or is something else going on here?

top for both servers

Rackspace

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                       
 9832 xxxxxxxx  20   0  525m 214m 4372 S  0.0 21.6   1:31.61 ruby                                                                                           
 9829 xxxxxxxx  20   0  443m 205m 3312 S  0.0 20.6   1:27.67 ruby                                                                                           
15597 xxxxxxxx  20   0  554m 176m 1268 S  0.0 17.8   4:59.36 ruby                                                                                           
 9780 xxxxxxxx  20   0  443m  63m 1088 S  0.0  6.4   0:28.80 ruby                                                                                           
  787 root      20   0  193m  17m 2608 S  2.0  1.7 350:43.06 driveclient                                                                                    
 1556 xxxxxxxx  20   0 77876  11m 1020 S  0.0  1.1  18:54.78 remote_syslog                                                                                  
17415 root      20   0 73096 3364 2608 S  0.0  0.3   0:00.03 sshd

Digital Ocean

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                
20921 xxxxxxxx  20   0  240m 191m 5328 S   0.0 19.1   0:29.62 ruby                                                                   
21009 xxxxxxxx  20   0  204m 178m 5356 S   0.0 17.8   0:20.82 ruby                                                                   
21194 xxxxxxxx  20   0  204m 174m 1724 S   0.0 17.4   0:00.10 ruby                                                                   
21206 xxxxxxxx  20   0  204m 174m 1656 S   0.0 17.4   0:00.10 ruby                                                                   
21181 xxxxxxxx  20   0 98.3m  89m 2184 S   0.3  8.9   0:03.04 ruby                                                                   
 1426 xxxxxxxx  20   0  117m  40m 2272 S   0.0  4.1   1:09.02 ruby                                                                   
 1429 xxxxxxxx  20   0  117m  29m 2180 S   0.0  3.0   1:09.64 ruby                                                                   
 1422 xxxxxxxx  20   0  117m 4652 1172 S   0.0  0.5   0:08.08 ruby                                                                   
22066 xxxxxxxx  20   0  7188 3456 1512 S   0.0  0.3   0:00.09 bash                                                                   
22008 root      20   0 10008 3320 2664 S   0.0  0.3   0:00.03 sshd

Should I be ditching Rackspace?

Edit: Deploy graph (excluding the file upload and decompression of precompiled assets)

This is not the graph of a happy deploy

Edit: vmstat

$ vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0    380     67     13    109    4    4    13    10   10   17  1  1 97  0
 0  0    380     67     13    109    0    0     0     0  650 1011  0  1 99  0
 0  0    380     67     13    109    0    0     0     0  675 1008  0  1 99  0
 0  0    380     67     13    109    0    0     0     0  659 1009  0  0 100  0
 1  0    380     67     13    109    0    0     0    68  661 1027  0  0 99  1
 0  0    380     67     13    109    0    0     0     0  667 1014  0  0 100  0
 1  0    380     67     13    109    0    0     0     0  671 1016  1  0 99  0
 0  0    380     67     13    109    0    0     0     0  668 1008  0  0 99  0
 0  0    380     67     13    109    0    0     0     0  671 1022  0  0 100  0
 0  0    380     67     13    109    0    0     0     0  783 1112  9  3 89  0     
share|improve this question
    
How much iowait are you seeing during deployments? –  EEAA Jul 20 '13 at 13:13
    
Much the same from 80- 170% for over 7 minutes. Have added graph of deploy. –  toxaq Jul 20 '13 at 13:35
1  
Definitely looks like I/O issues. Though keep in mind that DigitalOcean is using SSDs. –  Michael Hampton Jul 20 '13 at 18:44
    
Yes, I chose DigitalOcean as a staging server because of the price but now I'm wondering if I should migrate production for the performance. –  toxaq Jul 21 '13 at 0:11
add comment

3 Answers

From the data you have posted, this is definitely an I/O-bottleneck on the rackspace server. As your graph clearly shows, most of the CPU time is spend with I/O-Wait (i.e., the CPU is waiting for I/O-processes to finish).

This is usually caused by slow disk speed, and, as this is a virtual instance, there are probably other instances using up much of the hostsystem's I/O. I don't see a lot you could do apart from finding a different hoster (or convince them to migrate you to another hostsystem with less I/O load or get another server there on a different hostsystem and try if that one's better).

share|improve this answer
add comment

This certainly looks like an I/O bottleneck, possibly caused by a noisy neighbor.

Have Rackspace move you to another host either by entering their live chat or calling in. They should be able to have the host's usage checked while processing your migration.

share|improve this answer
    
I lodged a ticket, they watched the server and basically said 'nothing to see here'. They did provide a few things to try so will take a look at those. Can't say they were significantly different from what I've tried already... –  toxaq Jul 22 '13 at 11:44
    
:( Are they going to move your box over then? –  Kyle Kelley Jul 22 '13 at 11:51
    
Nope, they said they could see nothing wrong on the box. Implication therefore it is the app or server configuration. –  toxaq Jul 22 '13 at 22:58
add comment

I work at Rackspace and we would like to help you resolve this issue. If you could give us a call at 1-800-961-4454 we can check the health of the host that your server is on and move it to a new one if it does appear to be a noisy neighbor issue. I would also be interested in seeing the output of 'vmstat -S M 1 10', 'sar -b' (after some time has passed) and perhaps 'iostat -x /dev/xvdc 2 6' when this issue is occurring.

Thanks!

-Jimmy

share|improve this answer
    
Hi Jimmy, I've put a ticket in with Rackspace and the server was watched while this happened and apparently there was nothing of note. I will endeavour to get post the results of your queries. I imagine you got the downvote as you should've posted this as a comment, not an answer (it wasn't me). –  toxaq Jul 22 '13 at 0:41
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.