2

yesterday I added new harddrives(four as a raidz1 and one as hot-spare) to a opensolaris server, after extending the zpool the server hangs when writing large files but not when reading large files(large files = > 1GiB).

The zpool configuration before the upgrade looked like this:

state: ONLINE

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
  c9t0d0 ONLINE 0 0 0
  c9t1d0 ONLINE 0 0 0
  c9t2d0 ONLINE 0 0 0
  c9t3d0 ONLINE 0 0 0

After the upgrade the zpool looks like this:

state: ONLINE

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
  c9t0d0 ONLINE 0 0 0
  c9t1d0 ONLINE 0 0 0
  c9t2d0 ONLINE 0 0 0
  c9t3d0 ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
  c9t4d0 ONLINE 0 0 0
  c9t5d0 ONLINE 0 0 0
  c9t6d0 ONLINE 0 0 0
  c9t7d0 ONLINE 0 0 0
 spares
  c9t8d0 AVAIL

As you can see all drives are Online an even the 3Ware 9690SA-4I Controller tells me that everything is okey:

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
----------------------------------------------------------------------------- -
u0 SINGLE OK - - - 1862.63 RiW ON
u1 SINGLE OK - - - 1862.63 RiW ON
u2 SINGLE OK - - - 1862.63 RiW ON
u3 SINGLE OK - - - 1862.63 RiW ON
u4 SINGLE OK - - - 1862.63 RiW ON
u5 SINGLE OK - - - 1862.63 RiW ON
u6 SINGLE OK - - - 1862.63 RiW ON
u7 SINGLE OK - - - 1862.63 RiW ON
u8 SINGLE OK - - - 1862.63 RiW ON

VPort Status Unit Size Type Phy Encl-Slot Model
----------------------------------------------------------------------------- -
p8 OK u0 1.82 TB SATA - /c9/e0/slt1 SAMSUNG HD203WI
p9 OK u1 1.82 TB SATA - /c9/e0/slt3 SAMSUNG HD203WI
p10 OK u2 1.82 TB SATA - /c9/e0/slt5 SAMSUNG HD203WI
p11 OK u4 1.82 TB SATA - /c9/e0/slt6 SAMSUNG HD203WI
p12 OK u5 1.82 TB SATA - /c9/e0/slt8 SAMSUNG HD203WI
p13 OK u3 1.82 TB SATA - /c9/e0/slt10 SAMSUNG HD203WI
p14 OK u6 1.82 TB SATA - /c9/e0/slt13 SAMSUNG HD203WI
p15 OK u7 1.82 TB SATA - /c9/e0/slt15 SAMSUNG HD203WI
p16 OK u8 1.82 TB SATA - /c9/e0/slt17 SAMSUNG HD203WI

But when I start writing Files to this zfs the server hangs sometime during the write process and sometimes just after writing the whole file but for sure the server hangs... . Reading large files(7-8GiB) on the otherside is no problem!

Thanks for your answers!

cu

Guido

edit:

fyi: The server runs at svn_111b

edit 2:

scrub: scrub completed after 6h20m with 0 errors on Thu Jul 22 00:33:29 2010

As you can see there are no file system errors... .

Improve this question
9
  • Do you see any particular in /var/adm/messages? When the write hangs, if you do a "zfs list -t all" what do you get?
    – PiL
    Jul 22, 2010 at 8:12
  • storage 3,85T 6,83T 29,9K /storage storage/share 3,85T 6,83T 3,85T /storage/share But it seems I found the reason the 3Ware 9690SA-4I controller produced timeouts because write caching was enabeld(by default) and no BBU is available wich let the controller gone mad after adding the new hdds...
    – user48937
    Jul 22, 2010 at 9:41
  • It's always better disable hardware cache on a ZFS system. Especially when no BBU is present.
    – PiL
    Jul 22, 2010 at 10:21
  • But the server still crashes... /var/adm/messages : Jul 22 18:19:00 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 3 satapkt 0xffffff019bdfd510 timed out Jul 22 18:23:20 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 2 satapkt 0xffffff019846d0b8 timed out Jul 22 18:39:35 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 2 satapkt 0xffffff01aa17c6c8 timed out Jul 22 18:51:31 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 2 satapkt 0xffffff019e86ba50 timed out
    – user48937
    Jul 23, 2010 at 6:36
  • You said you have a 3ware...once i had problems with it, especially if it's associated with a particular board. Which server do you have? You run a pretty old version of opensolaris...have you tried to upgrade?
    – PiL
    Jul 23, 2010 at 7:58

1 Answer 1

Reset to default
-2

It's a 3+ years bug with ZFS ARC that still persists!

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6522017

It will also go out-of-bounds from the VM limits of a hypervisor!

Improve this answer
29
  • 1
    No, it isn't. Please stop assuming that this bug is relevant to every ZFS issue and/or question people post.
    – EEAA
    Jul 26, 2010 at 2:20
  • 1
    No I haven't, but it's pretty easy to determine from the text in the bug report that it has nothing to do with crashing or poor performance. Perhaps you linked to the wrong bug report?
    – EEAA
    Jul 26, 2010 at 2:58
  • 1
    What makes you think this person is running ZFS in a VM environment?
    – EEAA
    Jul 26, 2010 at 3:09
  • 1
    No, the only vested interest I have is in ensuring the accuracy of answers on SF. Up until this point, I've seen zero evidence that the bug you posted has anything to do with any of the questions you answered.
    – EEAA
    Jul 26, 2010 at 3:36
  • 1
    @user48838 at the moment it seems that resizing the ARC had solved my problem(I resized it to 1,5GiB from 2,4GiB[default]), this morning I could copy 10 DVD images to the server without a crash. The performance has declined but its still better than a crashing server :D ... Do you thing adding more memory would let me extend the ARC size again?
    – user48937
    Jul 26, 2010 at 7:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .