0

my freebsd 10 server running on zfs raidz. I have a problem with reading files, test

dd of=/dev/null if=/tmp/test/1000mb.bin bs=1024k count=1000
953+1 records in
953+1 records out
1000000000 bytes transferred in 31.144059 secs (32108853 bytes/sec)

As you can see 1Gb of data reads in 31s. :(

When i'm trying to count md5 of this file, it starts and didn't respond for some minutes. I'm working with large video files on this server ~2Gb, as well with small thumbs ~300Kb, it's a storage server, we do not access it much. Here is server configuration:

Intel® Core™ i7-2600 Quad-Core with Hyper-Threading-Technologie, 16 GB DDR3 RAM, 4 x 4 TB SATA HDD 3 Gb/s

ZFS is up to date. We have turned primary and secondary cache off. We have tuned zfs already with:

/boot/loader.conf

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"
geom_mirror_load="YES"

vfs.zfs.zio.use_uma="0"
vfs.zfs.arc_min="2048M"
vfs.zfs.arc_max="8186M"
vm.kmem_size_max="32G"
vm.kmem_size="24G"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
kern.maxvnodes=250000
vfs.zfs.write_limit_override=1073741824

Some info: zpool status

pool: tank
state: ONLINE
scan: scrub repaired 0 in 24h47m with 0 errors on Fri Sep  5 14:22:04 2014
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada2p3  ONLINE       0     0     0
            ada3p3  ONLINE       0     0     0

errors: No known data errors

zfs get all tank

NAME  PROPERTY              VALUE                  SOURCE
tank  type                  filesystem             -
tank  creation              Thu Jul 31 14:45 2014  -
tank  used                  2.65T                  -
tank  available             4.20T                  -
tank  referenced            209K                   -
tank  compressratio         1.00x                  -
tank  mounted               no                     -
tank  quota                 none                   default
tank  reservation           none                   default
tank  recordsize            128K                   default
tank  mountpoint            none                   local
tank  sharenfs              off                    default
tank  checksum              on                     default
tank  compression           off                    default
tank  atime                 on                     default
tank  devices               on                     default
tank  exec                  on                     default
tank  setuid                on                     default
tank  readonly              off                    default
tank  jailed                off                    default
tank  snapdir               hidden                 default
tank  aclmode               discard                default
tank  aclinherit            restricted             default
tank  canmount              on                     default
tank  xattr                 on                     default
tank  copies                1                      default
tank  version               5                      -
tank  utf8only              off                    -
tank  normalization         none                   -
tank  casesensitivity       sensitive              -
tank  vscan                 off                    default
tank  nbmand                off                    default
tank  sharesmb              off                    default
tank  refquota              none                   default
tank  refreservation        none                   default
tank  primarycache          none                   local
tank  secondarycache        none                   local
tank  usedbysnapshots       0                      -
tank  usedbydataset         209K                   -
tank  usedbychildren        2.65T                  -
tank  usedbyrefreservation  0                      -
tank  logbias               latency                default
tank  dedup                 off                    default
tank  mlslabel                                     -
tank  sync                  standard               default
tank  refcompressratio      1.00x                  -
tank  written               209K                   -
tank  logicalused           2.51T                  -
tank  logicalreferenced     43.5K                  -

Thanks for help.

UPDATE:

gstat while running dd

dT: 1.018s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    4    275    185   4780    5.2     89   2514    2.3   42.9| ada0
    5    352    245   5503    9.6    106   2585    2.4   77.9| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p2
    4    275    185   4780    5.2     89   2514    2.3   43.0| ada0p3
    0      0      0      0    0.0      0      0    0.0    0.0| ada1p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada1p2
    5    352    245   5503    9.6    106   2585    2.6   78.2| ada1p3
    0    359    261   5350   11.2     96   2518    4.8   88.4| ada2
    2    238    153   4678   18.4     83   2565    1.9   83.8| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/86db209a-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/87034dda-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/88396782-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/8861da11-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| ada2p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2p2
    0    359    261   5350   11.2     96   2518    4.9   88.4| ada2p3
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p2
    2    238    153   4678   18.4     83   2565    2.0   83.8| ada3p3
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/8999db59-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/89c1f2eb-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/8af4d439-18b0-11e4-977c-5404a6f17240
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/8b1c89ed-18b0-11e4-977c-5404a6f17240

UPDATE: Hardware info

ada0 at ahcich2 bus 0 scbus0 target 0 lun 0
ada0: <HGST HMS5C4040BLE640 MPAOA5D0> ATA-8 SATA 3.x device
ada0: Serial Number PL1331LAGS5T0H
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich3 bus 0 scbus1 target 0 lun 0
ada1: <HGST HMS5C4040BLE640 MPAOA5D0> ATA-8 SATA 3.x device
ada1: Serial Number PL1331LAGS40KH
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich4 bus 0 scbus2 target 0 lun 0
ada2: <HGST HMS5C4040BLE640 MPAOA5D0> ATA-8 SATA 3.x device
ada2: Serial Number PL1331LAGS5LHH
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad8
ada3 at ahcich5 bus 0 scbus3 target 0 lun 0
ada3: <HGST HMS5C4040BLE640 MPAOA5D0> ATA-8 SATA 3.x device
ada3: Serial Number PL1331LAGS15VH
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad10
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xfe501000-0xfe5017ff irq 20 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
ada0 at ahcich2 bus 0 scbus0 target 0 lun 0
ada1 at ahcich3 bus 0 scbus1 target 0 lun 0
ada2 at ahcich4 bus 0 scbus2 target 0 lun 0
ada3 at ahcich5 bus 0 scbus3 target 0 lun 0
ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)

UPDATE: zdb -m tank

Metaslabs:
    vdev          0
    metaslabs   115   offset                spacemap          free      
    ---------------   -------------------   ---------------   -------------
    metaslab      0   offset            0   spacemap     36   free    5.12G
    metaslab      1   offset   2000000000   spacemap     57   free    7.84G
    metaslab      2   offset   4000000000   spacemap     58   free    18.7G
    metaslab      3   offset   6000000000   spacemap     60   free    5.00G
    metaslab      4   offset   8000000000   spacemap     61   free    4.96G
    metaslab      5   offset   a000000000   spacemap     62   free    5.06G
    metaslab      6   offset   c000000000   spacemap     63   free    5.05G
    metaslab      7   offset   e000000000   spacemap     64   free    5.10G
    metaslab      8   offset  10000000000   spacemap     67   free    5.08G
    metaslab      9   offset  12000000000   spacemap     70   free    6.30G
    metaslab     10   offset  14000000000   spacemap     72   free    5.09G
    metaslab     11   offset  16000000000   spacemap     73   free    5.07G
    metaslab     12   offset  18000000000   spacemap     76   free    4.93G
    metaslab     13   offset  1a000000000   spacemap     77   free    4.98G
    metaslab     14   offset  1c000000000   spacemap     78   free    4.93G
    metaslab     15   offset  1e000000000   spacemap     79   free    5.09G
    metaslab     16   offset  20000000000   spacemap     80   free    5.08G
    metaslab     17   offset  22000000000   spacemap     81   free    4.96G
    metaslab     18   offset  24000000000   spacemap     84   free    5.22G
    metaslab     19   offset  26000000000   spacemap     85   free    5.95G
    metaslab     20   offset  28000000000   spacemap     87   free    5.03G
    metaslab     21   offset  2a000000000   spacemap     88   free    5.05G
    metaslab     22   offset  2c000000000   spacemap     35   free    6.40G
    metaslab     23   offset  2e000000000   spacemap     59   free    5.08G
    metaslab     24   offset  30000000000   spacemap     89   free    5.19G
    metaslab     25   offset  32000000000   spacemap     90   free    5.05G
    metaslab     26   offset  34000000000   spacemap     91   free    5.08G
    metaslab     27   offset  36000000000   spacemap     92   free    5.11G
    metaslab     28   offset  38000000000   spacemap     93   free    6.27G
    metaslab     29   offset  3a000000000   spacemap     94   free    6.42G
    metaslab     30   offset  3c000000000   spacemap     95   free    5.11G
    metaslab     31   offset  3e000000000   spacemap     69   free    20.1G
    metaslab     32   offset  40000000000   spacemap     71   free    13.0G
    metaslab     33   offset  42000000000   spacemap     99   free    13.7G
    metaslab     34   offset  44000000000   spacemap    100   free    20.0G
    metaslab     35   offset  46000000000   spacemap    102   free    74.6G
    metaslab     36   offset  48000000000   spacemap    103   free     117G
    metaslab     37   offset  4a000000000   spacemap    105   free     117G
    metaslab     38   offset  4c000000000   spacemap    106   free    72.8G
    metaslab     39   offset  4e000000000   spacemap    107   free    25.4G
    metaslab     40   offset  50000000000   spacemap     83   free    71.8G
    metaslab     41   offset  52000000000   spacemap    108   free    24.5G
    metaslab     42   offset  54000000000   spacemap     86   free    23.1G
    metaslab     43   offset  56000000000   spacemap    110   free    22.0G
    metaslab     44   offset  58000000000   spacemap     34   free    5.14G
    metaslab     45   offset  5a000000000   spacemap     66   free     126G
    metaslab     46   offset  5c000000000   spacemap    112   free     121G
    metaslab     47   offset  5e000000000   spacemap    113   free     125G
    metaslab     48   offset  60000000000   spacemap    114   free     125G
    metaslab     49   offset  62000000000   spacemap    115   free     128G
    metaslab     50   offset  64000000000   spacemap    116   free     128G
    metaslab     51   offset  66000000000   spacemap    117   free     128G
    metaslab     52   offset  68000000000   spacemap    120   free    78.0G
    metaslab     53   offset  6a000000000   spacemap     68   free    5.04G
    metaslab     54   offset  6c000000000   spacemap     75   free    4.99G
    metaslab     55   offset  6e000000000   spacemap     97   free     128G
    metaslab     56   offset  70000000000   spacemap     98   free    71.0G
    metaslab     57   offset  72000000000   spacemap    101   free     118G
    metaslab     58   offset  74000000000   spacemap    121   free     107G
    metaslab     59   offset  76000000000   spacemap    123   free     101G
    metaslab     60   offset  78000000000   spacemap      0   free     128G
    metaslab     61   offset  7a000000000   spacemap      0   free     128G
    metaslab     62   offset  7c000000000   spacemap     82   free    5.00G
    metaslab     63   offset  7e000000000   spacemap    124   free     125G
    metaslab     64   offset  80000000000   spacemap    111   free     122G
    metaslab     65   offset  82000000000   spacemap    109   free     123G
    metaslab     66   offset  84000000000   spacemap     65   free     127G
    metaslab     67   offset  86000000000   spacemap      0   free     128G
    metaslab     68   offset  88000000000   spacemap      0   free     128G
    metaslab     69   offset  8a000000000   spacemap      0   free     128G
    metaslab     70   offset  8c000000000   spacemap      0   free     128G
    metaslab     71   offset  8e000000000   spacemap      0   free     128G
    metaslab     72   offset  90000000000   spacemap      0   free     128G
    metaslab     73   offset  92000000000   spacemap      0   free     128G
    metaslab     74   offset  94000000000   spacemap    119   free    88.7G
    metaslab     75   offset  96000000000   spacemap     74   free     128G
    metaslab     76   offset  98000000000   spacemap      0   free     128G
    metaslab     77   offset  9a000000000   spacemap    104   free     128G
    metaslab     78   offset  9c000000000   spacemap      0   free     128G
    metaslab     79   offset  9e000000000   spacemap      0   free     128G
    metaslab     80   offset  a0000000000   spacemap      0   free     128G
    metaslab     81   offset  a2000000000   spacemap    122   free     123G
    metaslab     82   offset  a4000000000   spacemap      0   free     128G
    metaslab     83   offset  a6000000000   spacemap      0   free     128G
    metaslab     84   offset  a8000000000   spacemap     96   free    89.2G
    metaslab     85   offset  aa000000000   spacemap    129   free     128G
    metaslab     86   offset  ac000000000   spacemap      0   free     128G
    metaslab     87   offset  ae000000000   spacemap      0   free     128G
    metaslab     88   offset  b0000000000   spacemap      0   free     128G
    metaslab     89   offset  b2000000000   spacemap      0   free     128G
    metaslab     90   offset  b4000000000   spacemap      0   free     128G
    metaslab     91   offset  b6000000000   spacemap      0   free     128G
    metaslab     92   offset  b8000000000   spacemap      0   free     128G
    metaslab     93   offset  ba000000000   spacemap      0   free     128G
    metaslab     94   offset  bc000000000   spacemap      0   free     128G
    metaslab     95   offset  be000000000   spacemap      0   free     128G
    metaslab     96   offset  c0000000000   spacemap    118   free     128G
    metaslab     97   offset  c2000000000   spacemap      0   free     128G
    metaslab     98   offset  c4000000000   spacemap      0   free     128G
    metaslab     99   offset  c6000000000   spacemap      0   free     128G
    metaslab    100   offset  c8000000000   spacemap      0   free     128G
    metaslab    101   offset  ca000000000   spacemap      0   free     128G
    metaslab    102   offset  cc000000000   spacemap      0   free     128G
    metaslab    103   offset  ce000000000   spacemap      0   free     128G
    metaslab    104   offset  d0000000000   spacemap      0   free     128G
    metaslab    105   offset  d2000000000   spacemap      0   free     128G
    metaslab    106   offset  d4000000000   spacemap      0   free     128G
    metaslab    107   offset  d6000000000   spacemap      0   free     128G
    metaslab    108   offset  d8000000000   spacemap      0   free     128G
    metaslab    109   offset  da000000000   spacemap      0   free     128G
    metaslab    110   offset  dc000000000   spacemap      0   free     128G
    metaslab    111   offset  de000000000   spacemap      0   free     128G
    metaslab    112   offset  e0000000000   spacemap      0   free     128G
    metaslab    113   offset  e2000000000   spacemap      0   free     128G
    metaslab    114   offset  e4000000000   spacemap      0   free     128G

UPDATE: Enabling primarycache=all, increases read speed a bit, thanks to Eugene.

UPDATE: zdb tank | head -n 200 | grep ashift i need to terminate the command by CTRL + C after about 5 minute of execution, here is the output:

        ashift: 12
        ashift: 12
        ^C
22
  • do you have active NFS clients on this host ? Disabled prefetch is also slowing you down. As I see, you got 33 MB/sec. This isn't fast, but also isn't a catastrophic.
    – drookie
    Nov 13, 2014 at 12:35
  • also a gstat screenshot when issuing dd would be good.
    – drookie
    Nov 13, 2014 at 12:36
  • Updated thread with gstats data. I have a smbfs network partition on this server, but unmounting this partition and running tests again didn't changed results. Thanks.
    – Tony
    Nov 13, 2014 at 12:52
  • and NFS mounts from this server - do you have any ?
    – drookie
    Nov 13, 2014 at 12:55
  • no, nfs on remote server
    – Tony
    Nov 13, 2014 at 12:59

0

You must log in to answer this question.

Browse other questions tagged .