0

I have a Windows Server VM which connects to my Storage VMs running CentOS in another Data Centre over SFTP and FTP.

The uplink between the servers are all 10Gbps and more and both VMs are ESXi machines connected directory to the uplink and configured with valid static IP addresses (different hosts ofc since they are in different Data Centres).

When I try to upload files (large files, usually above 500MB) from the Windows machine to the CentOS machine over FTP or SFTP the transfer speed is around 6~40MB/s which is fine for the time being but after a few days of work the transfer speed goes down to 300~500Kbps.

The strange thing is that restarting the Network service on the CentOS box resolves the issue for a few days untill it happens again...

I have around 10 cloned pairs of the machines described above and they all have the same issue.

EDIT 1: There are no limitations on the network or the links applied by the Data Centre that we know of. We can make a request for them to look into the issue with enough evidence if enough symptoms point in that direction.

EDIT 2: Both machines have free CPU, Memory and IOPS.

SOLUTION / CAUSE: The issue was caused by network limitations from the Data Centre network on the Destination side.

6
  • Hello and welcome to Server Fault! Are the network / datacenters / links under your control or leased from a third party? You could be hitting a data transfer cap.
    – Mr Shunz
    Feb 15 '19 at 9:43
  • @MrShunz Thank you, The part of the network that is under our control has no restrictions, Uplinks from the upper hand provider too has no limitations that we know of, If the evidence points to such limitations, we have to submit a case to them for further investigation. Do the symptoms described above firmly point to that conclusion? For example, The fact that the issue temporarily being resolved after "service network restart" command on the CentOS box? Feb 15 '19 at 10:03
  • Well, the part about network restart is strange indeed. But usually, if you get (near)wire speed for N hours/days, and then suddenly a sustained and consistend speed drop, I'd suspect a data cap or a rate-limit/qos misconfigured somewhere. That said, if by restarting the network service the problem goes away, it might as well be a problem with the servers, sorry I don't have enough data to give you an answer.
    – Mr Shunz
    Feb 15 '19 at 10:11
  • BTW, you can try some network testing tools, like iperf to check if the problem is the link or your file transfers. I.e. if, while SFTP does 500kbps max, iperf shows a near line rate speed, you'll know the problem is not the network.
    – Mr Shunz
    Feb 15 '19 at 10:12
  • Every time I start and stop the iperf3 test it's either one of the two transfer speeds consistently running for as long as I let it: puu.sh/CMEod/9a2b39eb57.png once it's 300Mb and once it's 1Mb following no logical pattern, I believe it might be a load balancing issue and packet loss in the PortChannels between the uplink switches or the ESXi server host uplinks. I will update when I figure out the root of the issue. Feb 15 '19 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.