0

I have two servers both running Rocky Linux v8.4, one hosts a raid drive that I use to back up a (very) large raid drive mounted on Server A as /home.

Server A (with the large raid of 70TB mounted as /home that needs to be backed up) nfs mounts the backup drive from Server B. The servers are connected via a switch with a 10GB connection. I'm currently doing just a straight:

cp -a /home/* /backup

But the transfer speed seems slow to me. I've been monitoring the transfer with vnstat:


         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     2022-09-07
         18:00     52.79 MiB |    9.23 GiB |    9.28 GiB |   22.15 Mbit/s
         19:00     47.31 MiB |    6.14 GiB |    6.19 GiB |   14.76 Mbit/s
         20:00     47.14 MiB |    6.18 GiB |    6.22 GiB |   14.85 Mbit/s
         21:00     47.43 MiB |    6.24 GiB |    6.28 GiB |   14.99 Mbit/s
         22:00     44.31 MiB |    4.11 GiB |    4.15 GiB |    9.91 Mbit/s
         23:00     44.46 MiB |    4.04 GiB |    4.08 GiB |    9.73 Mbit/s
     2022-09-08
         00:00     44.90 MiB |    4.39 GiB |    4.43 GiB |   10.58 Mbit/s
         01:00     44.97 MiB |    4.63 GiB |    4.67 GiB |   11.15 Mbit/s
         02:00     43.82 MiB |    3.86 GiB |    3.90 GiB |    9.30 Mbit/s
         03:00     54.34 MiB |    9.77 GiB |    9.82 GiB |   23.43 Mbit/s
         04:00     47.28 MiB |    6.41 GiB |    6.45 GiB |   15.40 Mbit/s
         05:00     47.52 MiB |    6.30 GiB |    6.35 GiB |   15.14 Mbit/s
         06:00     47.63 MiB |    6.53 GiB |    6.58 GiB |   15.69 Mbit/s
         07:00     46.99 MiB |    5.98 GiB |    6.03 GiB |   14.38 Mbit/s
         08:00     43.95 MiB |    4.57 GiB |    4.61 GiB |   11.01 Mbit/s
         09:00     44.94 MiB |    4.35 GiB |    4.39 GiB |   10.48 Mbit/s
         10:00     45.17 MiB |    4.69 GiB |    4.74 GiB |   11.30 Mbit/s
         11:00     44.88 MiB |    4.31 GiB |    4.35 GiB |   10.39 Mbit/s
         12:00     44.80 MiB |    4.72 GiB |    4.77 GiB |   11.38 Mbit/s
         13:00     47.73 MiB |    6.24 GiB |    6.29 GiB |   15.01 Mbit/s
         14:00     48.13 MiB |    6.29 GiB |    6.34 GiB |   15.12 Mbit/s
         15:00     45.31 MiB |    4.84 GiB |    4.89 GiB |   11.66 Mbit/s
         16:00     45.89 MiB |    4.90 GiB |    4.95 GiB |   11.81 Mbit/s
         17:00      7.58 MiB |  873.07 MiB |  880.66 MiB |   12.31 Mbit/s
     ------------------------+-------------+-------------+---------------

Server A mounts /home with these options (from /proc/mounts):

/dev/sdb1 /home xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0

and automounts /backup from Server B nfs with these options:

Server_B:/backup /backup nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.33.254,local_lock=none,addr=172.16.33.250 0 0

Server B mounts the backup drive with these options:

/dev/sdb /backup xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0

Are there better options that would speed up the transfer between these things? There are definitely a mix of both very large files (50GB+) and very small files, so it isn't an ideal transfer scenario, but it still seems like it should be faster than this. Both of these are xfs file systems, running in a raid6 hardware config via Megaraid 9750-4i cards. The 70TB raid is composed of 21 4TB WDC WD4000FYYZ-01UL drives, and the backup drive is 21 16TB seagate ironwolf ST16000NE000-2RW103 drives. (I'm planning to migrate the smaller raid to the larger 21 16TB drives after the backup is complete).

Networking: Server A's NIC

Settings for enp132s0f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Server B's NIC

Settings for enp132s0f0np0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  Not reported
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: No
        Link partner advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 255
        Transceiver: internal
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000020f7 (8439)
                               drv probe link ifdown ifup rx_err tx_err hw
        Link detected: yes

iperf3 between servers:

Connecting to host fs01, port 5201
[  5] local 172.16.33.254 port 49004 connected to 172.16.33.250 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec    0    673 KBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    0    708 KBytes
[  5]   2.00-3.00   sec  1.09 GBytes  9.41 Gbits/sec    0    708 KBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.41 Gbits/sec    0    708 KBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0    708 KBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.41 Gbits/sec    0    708 KBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    0    708 KBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0    708 KBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.41 Gbits/sec    0    708 KBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0    708 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  11.0 GBytes  9.37 Gbits/sec                  receiver

iperf Done.
Improve this question

2 Answers 2

Reset to default
0

I would start by installing ethtool (if you haven't already). And then running ethtool against your network adapter. Verifying what link speed each server is connected at. Is it 10/100/1000 and what duplex?

After that I would download iperf (a command line bandwidth testing tool) to each linux box. With iperf you will run it on both linux machines. 1 machine will be the server (listening on a certain port) and the other will be a client, sending data to the server, and the program will test the actual bandwidth

Improve this answer
1
  • Thanks for the response! I've updated the original question to have the information you requested - it does look like the network is connected at 10GB and seems to be operating close to the expected value. Something about the drives themselves maybe?
    – jearl
    Sep 9, 2022 at 15:33
0

Apparently my biggest issue was NOT the mount options like I thought, it was just tons and tons of write operations on an NFS drive. I found this excellent question/answer here about trying to optimize the transfer of a huge amount of data: How to copy a large number of files quickly between two servers

It turns out if I archive with tar first, then the transfer rates suddenly get significantly better. I bet there might be better ways to mount the drives to get more efficient read/write speeds, but I'm happy with this increase. The command I eventually used was this:

for i in *; do echo "Transferring $i..."; tar c $i | tee >(sha512sum >&2) |pv -prab |ssh root@serverB 'tee >(sha512sum >&2) |tar xC /backup';done

Which is cool because it calculates the checksums to confirm everything that was transferred made it without corruption. I did it individually on each directory so if something went wrong I wouldn't have to check the whole drive, I could just rsync the individual directory that failed. My new transfer rates look like this:

vnstat -h

 enp132s0f0  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     2022-09-11
         12:00      6.19 GiB |    1.18 TiB |    1.19 TiB |    2.90 Gbit/s
         13:00      4.68 GiB |  914.89 GiB |  919.56 GiB |    2.19 Gbit/s
         14:00      4.07 GiB |  794.46 GiB |  798.53 GiB |    1.91 Gbit/s
         15:00      3.11 GiB |  599.20 GiB |  602.32 GiB |    1.44 Gbit/s
         16:00     52.35 GiB |  981.27 GiB |    1.01 TiB |    2.47 Gbit/s
         17:00    100.42 GiB |  692.77 GiB |  793.20 GiB |    1.89 Gbit/s
         18:00     15.76 GiB |   42.21 GiB |   57.97 GiB |  138.32 Mbit/s
         19:00      6.87 GiB |    1.52 GiB |    8.38 GiB |   20.00 Mbit/s
         20:00     27.24 GiB |  788.69 GiB |  815.93 GiB |    1.95 Gbit/s
         21:00      3.12 GiB |  232.74 GiB |  235.86 GiB |  562.79 Mbit/s
         22:00      2.12 GiB |  182.84 GiB |  184.95 GiB |  441.31 Mbit/s
         23:00    394.28 MiB |   55.79 MiB |  450.06 MiB |    1.05 Mbit/s
     2022-09-12
         00:00      3.29 GiB |  671.18 GiB |  674.47 GiB |    1.61 Gbit/s
         01:00      3.14 GiB |  690.73 GiB |  693.86 GiB |    1.66 Gbit/s
         02:00      4.93 GiB |    1.04 TiB |    1.05 TiB |    2.56 Gbit/s
         03:00      4.98 GiB |    1.02 TiB |    1.03 TiB |    2.51 Gbit/s
         04:00      4.91 GiB |    1.04 TiB |    1.05 TiB |    2.56 Gbit/s
         05:00      5.29 GiB |    1.09 TiB |    1.10 TiB |    2.68 Gbit/s
         06:00      4.75 GiB |    1.02 TiB |    1.03 TiB |    2.50 Gbit/s
         07:00      2.90 GiB |  667.92 GiB |  670.82 GiB |    1.60 Gbit/s
         08:00    962.17 MiB |  207.28 GiB |  208.22 GiB |  496.82 Mbit/s
         09:00    408.99 MiB |   75.80 GiB |   76.20 GiB |  181.81 Mbit/s
         10:00    763.05 MiB |  139.60 GiB |  140.34 GiB |  334.87 Mbit/s
         11:00    140.79 KiB |   39.67 KiB |  180.46 KiB |    4.93 kbit/s
     ------------------------+-------------+-------------+---------------
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .