2

I have a 16 core virtualization FreeNAS 11.3-U4.1 instance running on Hyper-V with 32GB of RAM with a 18TB iscsi target on DELL T630 server grade hardware.

HDD are on IT mode from the RAID card, so presented to the FreeNAS box as individual drives.

I do not have any dedupe running, default compression only.

The windows share is running ReFS as a iscsi initiator on Windows Server 2019 and is not full only 1TB or so, used for transferring test movie files.

From multiple machines tested (one at a time), it is initially fast on any transfer - read or write @ 130MB/sec - but then after a minute or so slows down to 40Mbps and at times even keeps stalling.

Network tests are fast both internally via machines on same box and also external machines:

root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.69.139, port 49435
[  5] local 192.168.69.4 port 5201 connected to 192.168.69.139 port 49437
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   105 MBytes   878 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   939 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  5]   5.00-6.00   sec   111 MBytes   935 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   939 Mbits/sec
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
[  5]  10.00-10.00  sec   339 KBytes   955 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.09 GBytes   933 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.69.5, port 50601
[  5] local 192.168.69.4 port 5201 connected to 192.168.69.5 port 50602
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   1.00-2.00   sec   407 MBytes  3.41 Gbits/sec
[  5]   2.00-3.00   sec   426 MBytes  3.57 Gbits/sec
[  5]   3.00-4.00   sec   430 MBytes  3.60 Gbits/sec
[  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec
[  5]   5.00-6.00   sec   429 MBytes  3.60 Gbits/sec
[  5]   6.00-7.00   sec   372 MBytes  3.12 Gbits/sec
[  5]   7.00-8.00   sec   429 MBytes  3.60 Gbits/sec
[  5]   8.00-9.00   sec   424 MBytes  3.56 Gbits/sec
[  5]   9.00-10.00  sec   425 MBytes  3.56 Gbits/sec
[  5]  10.00-10.00  sec   125 KBytes  2.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate

CPU load via top is minimal:

last pid: 15377;  load averages:  1.15,  0.90,  1.06    up 0+12:42:50  03:48:22
44 processes:  1 running, 43 sleeping
CPU:  0.1% user,  0.0% nice, 15.7% system,  0.0% interrupt, 84.2% idle
Mem: 14M Active, 488M Inact, 1060M Laundry, 28G Wired, 1584M Free
ARC: 25G Total, 852M MFU, 19G MRU, 4773M Anon, 580M Header, 75M Other
     20G Compressed, 23G Uncompressed, 1.12:1 Ratio
Swap: 6144M Total, 46M Used, 6097M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
  104 root         41  20    0   329M   258M kqread  5   3:15   0.77% python3.7
15347 root          1  20    0  7892K  3452K CPU5    5   0:00   0.16% top
 1917 www           1  20    0 30920K 10268K kqread  6   0:01   0.03% nginx
   93 root          1  20    0  9164K  5548K select  7   0:04   0.02% devd
 5597 root          8  20    0 40284K 17892K select  6   1:25   0.00% rrdcached
 1323 root          1  20    0 12484K 12580K select  2   0:02   0.00% ntpd
 1522 root          1  20    0  9012K  5024K select  5   0:01   0.00% zfsd
 1433 root          1  20    0   127M   105M kqread  2   0:03   0.00% uwsgi-3.7
 5622 root         11  20    0   354M   261M nanslp  7   6:57   0.00% collectd

zpools are below:

root@freenas[~]# zpool list -v
NAME                                     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot                             110G  1.25G   109G        -         -   0%     1%  1.00x  ONLINE  -
  da0p2                                  110G  1.25G   109G        -         -   0%     1%
pool1                                   32.5T  2.60T  29.9T        -         -   1%     7%  1.00x  ONLINE  /mnt
  raidz2                                32.5T  2.60T  29.9T        -         -   1%     7%
    gptid/e7aba92d-dc27-11ea-b1e2-00155d450302      -      -      -        -     -      -      -
    gptid/ea0339c1-dc27-11ea-b1e2-00155d450302      -      -      -        -     -      -      -
    gptid/eadd3d73-dc27-11ea-b1e2-00155d450302      -      -      -        -     -      -      -
    gptid/eb18ed64-dc27-11ea-b1e2-00155d450302      -      -      -        -     -      -      -
    gptid/eb80f3cc-dc27-11ea-b1e2-00155d450302      -      -      -        -     -      -      -
    gptid/eb99fe09-dc27-11ea-b1e2-00155d450302      -      -      -        -     -      -      -

using root@freenas[~]# ./solnet-array-test-v2.sh i can see (the write is showing will take 467 minutes, i am sure write speeds are OK as initially they are fast):

Disk    Disk Size  MB/sec %ofAvg
------- ---------- ------ ------
da0       130048MB    658    268 ++FAST++
da1      5723166MB    204     83 --SLOW--
da2      5723166MB    176     72 --SLOW--
da3      5723166MB    164     67 --SLOW--
da4      5723166MB    161     66 --SLOW--
da5      5723166MB    186     76 --SLOW--
da6      5723166MB    167     68 --SLOW--

Performing initial parallel array read
Wed Aug 12 04:06:35 PDT 2020
The disk da0 appears to be 130048 MB.
Disk is reading at about 1075 MB/sec
This suggests that this pass may take around 2 minutes

                   Serial Parall % of
Disk    Disk Size  MB/sec MB/sec Serial
------- ---------- ------ ------ ------
da0       130048MB    658      0      0 --SLOW--
da1      5723166MB    204    204    100
da2      5723166MB    176    164     93
da3      5723166MB    164    166    101
da4      5723166MB    161    159     99

To add, Windows Storage Spaces with the same hardware is @ 150MB/s constant (pretty much what I am seeing above with the test), but without the slowdown after 1 minute to 30-50MB/sec that i am seeing in actual use of Freenas.

Any other diagnostics i would need to run or ideas on what this could be?

1
  • 1
    What is your network configuration? I'd had an issues with network throughput being slow on Hyper-V VMs, after I did PCIe SR-IOV to the VM my setup gone faster. I was testing CentOS VM, though. – Stuka Aug 14 '20 at 6:38
2

FreeNAS and the whole ZFS on BSD kernel story is dying, get Ubuntu and ZoL, there’s much wider community coverage and better hardware support either way.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.