Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two servers, Both servers hardware Specification are

Processor : Dual Processor
RAM : over 128 G.B
Hard disk : SSD Hard disk
Outging Traffic bandwidth : 3 Gbps
network cards speed : 10 Gbps

Server A : for Encoding videos

Server B : for storage videos andstream videos over web interface like youtube

The inbound bandwidth between two servers is 10Gbps , the outbound bandwidth internet bandwidth is 500Mpbs

Both servers using public ip addresses in public and private network

Both servers transfer and connection on nginx port , and the server B used for streaming media , like youtube stream videos

Both servers in same network , when i do ping from Server A to Server B i got high time latency above 1.0ms , the time range time=52.7 ms to time=215.7 ms -

This is the output of iftop utility

                            353Mb                       707Mb                       1.04Gb                      1.38Gb                1.73Gb
mqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqq
server.example.com                                        => ip.address                                              6.36Mb  4.31Mb  1.66Mb
                                                         <=                                                           158Kb  94.8Kb  35.1Kb
server.example.com                                        => ip.address                                             1.23Mb  4.28Mb  1.12Mb
                                                         <=                                                          17.1Kb  83.5Kb  21.9Kb
server.example.com                                        => ip.address                                            395Kb  3.89Mb  1.07Mb
                                                         <=                                                          6.09Kb   109Kb  28.6Kb
server.example.com                                        => ip.address                                           4.55Mb  3.83Mb  1.04Mb
                                                         <=                                                          55.6Kb  45.4Kb  13.0Kb
server.example.com                                        => ip.address                                             649Kb  3.38Mb  1.47Mb
                                                         <=                                                          9.00Kb  38.7Kb  16.7Kb
server.example.com                                        => ip.address                                             5.00Mb  3.32Mb  1.80Mb
                                                         <=                                                          65.7Kb  55.1Kb  29.4Kb
server.example.com                                        => ip.address                                              387Kb  3.13Mb  1.06Mb
                                                         <=                                                          18.4Kb  39.9Kb  15.0Kb
server.example.com                                        => ip.address                                              3.27Mb  3.11Mb  1.01Mb
                                                         <=                                                          81.2Kb  64.5Kb  20.9Kb
server.example.com                                        => ip.address                                             1.75Mb  3.08Mb  2.72Mb
                                                         <=                                                          16.6Kb  35.6Kb  32.5Kb
server.example.com                                        => ip.address                                               1.75Mb  2.90Mb  2.79Mb
                                                         <=                                                          22.4Kb  32.6Kb  35.6Kb
server.example.com                                        => ip.address                                             3.03Mb  2.78Mb  1.82Mb
                                                         <=                                                          26.6Kb  27.4Kb  20.2Kb
server.example.com                                        => ip.address                                              2.26Mb  2.66Mb  1.36Mb
                                                         <=                                                          51.7Kb  49.1Kb  24.4Kb
server.example.com                                        => ip.address                                              586Kb  2.50Mb  1.03Mb
                                                         <=                                                          4.17Kb  26.1Kb  10.7Kb
server.example.com                                        => ip.address                                               2.42Mb  2.49Mb  2.44Mb
                                                         <=                                                          31.6Kb  29.7Kb  29.9Kb
server.example.com                                        => ip.address                                            2.41Mb  2.46Mb  2.41Mb
                                                         <=                                                          26.4Kb  24.5Kb  23.8Kb
server.example.com                                        => ip.address                                             2.37Mb  2.39Mb  2.40Mb
                                                         <=                                                          28.9Kb  27.0Kb  28.5Kb
server.example.com                                        => ip.address                                             525Kb  2.20Mb  1.05Mb
                                                         <=                                                          7.03Kb  26.0Kb  12.8Kb
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
TX:             cum:    102GB   peak:   1.65Gb                                                              rates:   1.46Gb  1.44Gb  1.48Gb
RX:                    1.31GB           24.3Mb                                                                       19.5Mb  18.9Mb  20.0Mb
TOTAL:                  103GB           1.67Gb                                                                       1.48Gb  1.46Gb  1.50Gb

I check the transfer speed using iperf utility From Server A to Server B

# iperf -c 0.0.0.2 -p 8777
------------------------------------------------------------
Client connecting to 0.0.0.2, TCP port 8777
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  3] local 0.0.0.1 port 38895 connected with 0.0.0.2 port 8777
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.8 sec    528 KBytes    399 Kbits/sec

My Current Connections in Server B

# netstat -an|grep ":8777"|awk '/tcp/ {print $6}'|sort -nr| uniq -c
   2072 TIME_WAIT
     28 SYN_RECV
      1 LISTEN
    189 LAST_ACK
    139 FIN_WAIT2
    373 FIN_WAIT1
   3381 ESTABLISHED
     34 CLOSING

Server A Network Card Information

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Server B Network Card Information

Settings for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

ifconfig server A

eth0      Link encap:Ethernet  HWaddr 00:25:90:ED:9E:AA
          inet addr:0.0.0.1  Bcast:0.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1202795665 errors:0 dropped:64334 overruns:0 frame:0
          TX packets:2313161968 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:893413096188 (832.0 GiB)  TX bytes:3360949570454 (3.0 TiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2207544 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2207544 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:247769175 (236.2 MiB)  TX bytes:247769175 (236.2 MiB)

ifconfig Server B

eth2      Link encap:Ethernet  HWaddr 00:25:90:82:C4:FE
          inet addr:0.0.0.2  Bcast:0.0.0.2  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:39973046980 errors:0 dropped:1828387600 overruns:0 frame:0
          TX packets:69618752480 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3013976063688 (2.7 TiB)  TX bytes:102250230803933 (92.9 TiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1049495 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1049495 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:129012422 (123.0 MiB)  TX bytes:129012422 (123.0 MiB)

Netstat -i on Server B

# netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth2       9000   0 42098629968      0 2131223717      0 73698797854      0      0      0 BMRU
lo        65536   0  1077908      0      0      0  1077908      0      0      0 LRU

I Turn up send/receive buffers on the network card to 2048 and problem still persist

I increase the MTU for server A and problem still persist and i increase the MTU for server B for better connectivity and transfer speed but it couldn't transfer at all

The problem is : as you can see from iperf utility, the transfer speed from server A to server B slow when i restart network service in server B the transfer in server A at full speed, after 2 minutes , it's getting slow

How could i troubleshoot slow speed issue and fix it in server B ?

Notice : if there any other commands i should execute in servers for more information, so it might help resolve the problem , let me know in comments

share|improve this question
    
Please add ifconfig output for both servers' all network cards, and exact network topology information (how are both servers connected to the Internet, and how they are connected between themselves). –  Tero Kilkanen Jun 1 at 9:47
    
@TeroKilkanen i add the output , thanks –  Linux Guy Jun 1 at 14:36
    
@LinuxGuy The iperf documentation I found states that you need to run iperf in server mode and connect to that using iperf in client mode. But your question looks more like you are connecting the iperf client to an nginx server. –  kasperd Jun 1 at 14:49
    
The information on exact network topology is still missing, only guesses can be given. Anyway, I can see that your MTU is set to 1500. That should be 9000 in both ends to make the packet overhead smaller. –  Tero Kilkanen Jun 1 at 15:59
1  
Leave the MTU unless you control the switches. It is theoretically a good idea to use Jumbo Packets but this means every switch in between must accept them and not drop them. If they are in a data center contact their staff before doing changes like that - unless you "own" the network (i.e. control the switches), then check yourself. Also please provide a network diagram with IP addresses / networks. –  TomTom Jun 9 at 13:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.