2

I'm debugging an issue where some http requests randomly take longer times than normally. (Normal being a few ms, abnormally being 1000ms up to a minute or more)

The host has several IP's and the service I'm working with is proxied by nginx on two of those IPs.

If I send requests to the first IP, everything works as expected and all requests are fast. If I send requests to the second IP, some requests randomly take several seconds or more.

Both IPs are on the same nic:

2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq      master bond0 state UP group default qlen 1000
link/ether 00:0c:29:5e:47:7d brd ff:ff:ff:ff:ff:ff
3: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether 00:0c:29:5e:47:7d brd ff:ff:ff:ff:ff:ff
inet 192.168.10.73/24 brd 192.168.10.255 scope global bond0
   valid_lft forever preferred_lft forever
inet 192.168.10.72/32 scope global bond0
   valid_lft forever preferred_lft forever
inet 192.168.10.75/32 scope global bond0
   valid_lft forever preferred_lft forever
inet 192.168.10.76/32 scope global bond0
   valid_lft forever preferred_lft forever
inet 192.168.10.54/32 scope global bond0
   valid_lft forever preferred_lft forever

So, for example:

curl -H "Host: xxx" http://192.168.10.54:6700/test

will randomly be slow, while

curl -H "Host: xxx" http://192.168.10.75:6700/test

is always fast.

Both IPs are served by the same nginx server block:

server {
        listen 192.168.10.75:6700;
        listen 192.168.10.54:6700;

        location / {
                proxy_pass http://...;
        }
}

I've checked open ports, but there are just a few hundred connections on each IP.

This is on Linux (3.16.0-8-amd64 #1 SMP Debian 3.16.64-2 (2019-04-01) x86_64 GNU/Linux) The server is running in vmWare, so NIC is reported as 0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

I have no iptable rules

I'm not sure how to proceed with debugging and would appreciate any hints :-)

Concrete questions:

  • What settings can be different on IP level on the same bond that I could check
  • What metrics can I check to give me hints on the cause
  • Could something else on the network cause such effects (duplicate IPs, vlan configuration, ..?)
  • What could be "overloaded" or saturated on IP but not another?
Improve this question
6
  • What network cards/chipset(s) are in use? Have you discounted a bad network cable or switch port?
    – davidgo
    Commented Sep 15, 2019 at 10:26
  • for me ots unclear because you curl port 80 and on the server bloxk using 6700? whats the rest? what does nginx -T say?
    – djdomi
    Commented Sep 15, 2019 at 15:54
  • Sorry, I missed the port when o wrote the example. Of course I query port 6700..
    – Brimstedt
    Commented Sep 15, 2019 at 18:26
  • Added info about the NIC, it's running on vmWare. I can check real hardware tomorrow. Cables are not replaced yet - but Ive written it off as both IPs run on same NIC and cables..
    – Brimstedt
    Commented Sep 15, 2019 at 18:46
  • Why do you have multiple IPs and bonded ethernet running off a single interface? On a virtual machine?
    – symcbean
    Commented Sep 15, 2019 at 18:49

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .