1,225 reputation
39
bio website nex7.com
location Mercer Island, WA
age
visits member for 3 years, 8 months
seen Jul 8 at 18:28

( http://www.nexenta.com )

I am a Solutions Architect at Nexenta Systems, Inc, formerly a Nexenta Support Engineer.

My involvement in this website is strictly voluntary, and on my own initiative as time allows. Please do not message me privately about active NexentaStor support cases. If you have a Nexenta or ZFS-related question on ServerFault, I don't mind if you ping me with a link and request I take a look, but I cannot guarantee a response, much less a timely one.

I do not represent Nexenta in any official capacity on this site, and my comments/posts are my own, and do not represent official Nexenta responses or policy.


Jul
8
comment zpool import: volume FAULTED with corrupted data, is it possible to save some data?
I suggest zpool import -fFX simply because you've /already/ done -fFn. It might be "safer" to start playing with zdb to determine what, if anything, is wrong with the pool that would kill a normal import. Sometimes this is something silly like missing disk labels (zdb -l <disk> on each disk should help you see if all 4 labels are intact on each disk and are identical).
Jul
8
comment zpool import: volume FAULTED with corrupted data, is it possible to save some data?
Oh and to answer your question, yes. It is extremely likely ALL or nearly all of your data is intact, it is merely a matter of getting around the various potential gotchya's that are preventing the normal method of import from working.
Jul
8
comment zpool import: volume FAULTED with corrupted data, is it possible to save some data?
I'd do it on latest code, but if you have v28 you may have some version of this -- try zpool import -fFX vol4disks8tb
Mar
30
comment ZFS import unable to find any pools
If you tried with -d <disk dev path including partition> and it didn't show up, try everything again but on an illumos OS. If that still can't see it, I'm out of ideas. You may need to engage a data recovery expert if the data has monetary value, or start looking at the code (src.illumos.org) while on the illumos derivative and dtrace'ing the zpool import command to see what path it takes and try to figure out why it can't see your pool.
Mar
28
comment ZFS import unable to find any pools
I don't know that this is the proper forum for this, because the 'answer' to the question involves a lot of trial & error. For now, try 'zpool import -d </dev/path/to/disk>'. -D lists destroyed pools, -d takes an argument of the location of a disk to look at, and can be specified multiple times on the command line (but in your case, only once will be needed as you have but the one disk). See what that does.
Mar
26
comment ZFS checksum errors: what files are affected?
Do you have any snapshots? They often are a culprit for errors showing up even after the offending file is gone.
Mar
11
comment ZFS checksum errors: what files are affected?
Oh, and stop making non-redundant pools. That thing has CKSUM errors because there's no ZFS-level redundancy in place, so it can't correct them automatically.
Mar
11
comment ZFS checksum errors: what files are affected?
Run a scrub, it might list off a file once it gets to it on the disk. That is only possible if the damage is actually in the file and not in metadata AND 'scratch' isn't a zvol (if it's a zvol, ZFS has no clue what file structure is inside).
Feb
16
comment ZFS, NFS and Nexenta
So if I understand correctly, the problem appears when you override the defaults so that your r/wsizes are set to 4k? And that this only affects CentOS 6.5? If that's accurate, let me know, so I can try to replicate in lab.
Feb
14
comment ZFS, NFS and Nexenta
Also, for the record, we have 1000's of RHEL/CentOS clients talking to NexentaStor every day and aren't reporting similar problems. Can you tell us more about your environment? Does anything differ between the Debian and CentOS machines, other than the OS? Hardware? Where they're plugged in/network environment? VM vs hardware?
Feb
14
comment ZFS, NFS and Nexenta
It is likely very telling that it is only happening on RHEL/CentOS clients, but I'm afraid that's not a smoking gun for me - nothing jumps to mind. Have you tried asking on a RHEL/CentOS support forum/area? That the problem is not repeatable on Debian would imply the problem is not with Nexenta, but with RHEL/CentOS specifically.
Feb
14
comment ZFS, NFS and Nexenta
What sort of performance do you get on the Debian system?
Feb
8
comment zfs regular read pauses
Status of C-states? And to pre-empt an answer of 'on', turn off C-states. They are a huge killer of performance (for instance, you'll never get full 10Gbe if C-states is on, fun huh?). You can't really (AFAIK) disable them fully in OS. Go into your BIOS, find anything about C-States and disable it. Then find where it talks about power, and set it to 'Performance' or the equivalent. If your CPU is capable of dropping below C1 (or maybe C1E), you're going to feel performance penalties, and at times you wouldn't even expect it.
Feb
6
comment ZFS, NFS and Nexenta
Way more information necessary. What does 'hddisco' say (as this is a Nexenta box), what does 'zpool status' say, what is in 'dmesg', what is the hardware in question, what is the network environment, what are the clients you're testing from, are you sure you're using NFSv3 and not NFSv4 (some clients default to 4), how are you attempting to copy files.
Feb
1
comment Is “zfs receive” an atomic operation?
That statement by Matt should be read carefully. The 'reclaim' is similar to a zfs rollback, thus a zfs destroy. If the sent dataset is 10 TB and completed most of it before failing, the resulting reclaim can be incredibly IOPS intensive. Always keep that in the back of your mind when digging into performance oddities, as well as when performing maintenance tasks that involve pool export/import.
Jan
23
comment ZFS: Memory issues with dedup even though zdb -DD looks fine
I thought I posted this but I guess I didn't. Don't assume your data is actually dedupable, btw. Also, in my experience, dedupe ratios usually end up falling over time, not increasing. So if you only start at 3.5x, I wouldn't place bets on it increasing. ZFS dedupe is block level not file level, and not 'intelligent' about bit shift. If you have two otherwise identical blocks of data, but there's a single bit offset on the data, it will not dedupe at all. Most the 'dedupe is what we do' style products tend to have algorithms to look for such things, so they tend to return better ratios.
Jan
22
comment ZFS: Memory issues with dedup even though zdb -DD looks fine
Considering the still-not-insignificant issues regarding 'memory' in the ZFS On Linux project, I would really not recommend you try dedupe at any scale on ZoL. Stick to an illumos-derivative preferably, or a FreeBSD box if that's not an option.
Jan
21
comment XenServer Linux guest booting off iSCSI SAN (ZFS storage): Will a non-checksumming guest file system still be at risk?
Also make sure your full environment is using ECC RAM. The FreeNAS box, and the clients. Non-ECC RAM can silently corrupt data and you'll never know until something crashes/breaks.
Dec
27
comment ZFS: bringing a disk online in an unavailable pool
Very glad you got it to work out, @indotoonster! This is an area that ZFS really needs both more documentation about and an easier/more robust set of options/tools to deal with weird situations. Hopefully developers will donate their time and effort at streamlining this sort of stuff, and it will become better understood as time goes on. :)
Dec
8
comment How often should I defrag ZFS pools?
I would say not /nearly/ so often. A scrub simply reads every block on the system. That's all it does. By virtue of how ZFS handles reads, ZFS will automatically repair any bad blocks when they're read. This means a scrub is only necessary to read data that isn't commonly being read anyway. Thus, the asker's comment of very busy pools once per week and not so busy pools once per month is actually backwards. A very busy pool, especially one that reads the majority of the live data regularly, need not scrub that often. A quiet pool that is rarely accessed should scrub more often.