5

My application is crawling a website. I use this crawler :

On my dev environment, the latency is acceptable (3-4 sec) but on my prod server (ec2 T2.micro Linux/Amazon) there is a 30 seconde latency (wait ttfb).

I tested the same code in another server (1&1) and there is a acceptable latency (3-4 sec). So the problem is not inside the code, but about the server config.

I have tried :

  • check the php conf (php.ini)
  • add cache to php (Zend OPCache)
  • add apache 2 vary: accept-encoding header
  • test a t2.medium instance
  • change the region of my instance

I found this similar question : PHP + Apache long wait time but I think that my ec2 volume is an EBS SSD so there is no access time issue.

This unsolved question is also link to mine : https://stackoverflow.com/questions/34299038/deployed-web-application-long-ajax-wait-ttfb-time

An extract of vmstat during the request:

[ec2-user@ip-xxx-xx-xx-xx ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0   1580 534472  29696 240356    0    0     2    31   14   41  0  0 100  0  0   
 0  0   1580 534456  29696 240356    0    0     0     0   19   43  0  0 100  0  0   
 0  0   1580 534456  29696 240356    0    0     0     0   12   37  0  0 100  0  0   
 0  0   1580 534456  29696 240356    0    0     0     0   12   38  0  0 100  0  0   
 0  0   1580 534456  29696 240356    0    0     0     0   15   45  0  0 100  0  0   
 0  0   1580 534456  29696 240356    0    0     0     0   18   49  0  0 100  0  0   
 0  0   1580 534456  29696 240356    0    0     0     0   12   38  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   32   60  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   13   39  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   17   45  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     4   16   49  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   13   40  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   13   38  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   13   38  0  0 100  0  0   
 0  0   1580 533544  29696 240356    0    0     0     0   16   47  0  0 100  0  0   
 0  0   1580 533544  29700 240352    0    0     0     4   17   48  0  0 99  1  0    
 0  0   1580 533544  29700 240356    0    0     0     0   15   39  0  0 100  0  0   
 0  0   1580 533544  29700 240356    0    0     0     0   13   39  0  0 100  0  0   
 0  0   1580 533544  29700 240356    0    0     0     0   13   39  0  0 100  0  0   
 0  0   1580 533544  29700 240356    0    0     0     0   17   46  0  0 100  0  0   
 0  0   1580 533544  29700 240356    0    0     0     4   19   50  0  0 100  0  0   
 0  0   1580 533544  29700 240356    0    0     0     0   15   41  0  0 100  0  0

[UPDATE]
Right now I know that this latence is due to a GuzzleHttp\Client request :

(string) $this->httpClient->get($url)->getBody();

My security group seems ok, all is allowed from outbound and HTTP is allowed from inbound. My network out also seems ok (40,000 - 50,000 bytes when I request).

I'm thinking that this should be a problem with mpm_prefork_module...

[UPDATE 2]
From SSH, If I curl http://google.fr, I have no latency. If I curl my specific url (http://www.leboncoin.fr) the latency is >30 sec. It's specific to my AWS Ec2 with this URL.

  • 2
    What are the iowait and CPU steal percentages at? – EEAA Jan 3 '16 at 22:55
  • My CPU is always under 5%.I use vmstat 1 to see IOWait, it seems to be 0 or 1 during the request. – Julien Malige Jan 4 '16 at 9:17
  • 1
    Is the HTTP request to a specific website? If so, how long does curl http://example.com/ with that website take to return data back? 30 seconds sounds almost like a DNS timeout or fallback. – ceejayoz Jan 4 '16 at 21:42
  • Yes it's specific to the URL ! (no latency with google or stack). But after the 30 seconds my page is loaded. Can you tell me more @ceejayoz ? – Julien Malige Jan 4 '16 at 21:55
  • @JulienMalige Next, I'd try dig example.com and see if that takes a while. If so, you have a DNS issue, probably fixable by updating /etc/resolv.conf. – ceejayoz Jan 4 '16 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.