Hot answers tagged

35

I've built a number of these "all-in-one" ZFS storage setups. Initially inspired by the excellent posts at Ubiquitous Talk, my solution takes a slightly different approach to the hardware design, but yields the result of encapsulated virtualized ZFS storage. To answer your questions: Determining whether this is a wise approach really depends on your goals....


15

The conversion of a SCSI chassis DL380 G4 to the SAS chassis is possible, but not at all practical. It requires a bezel change, backplane modification, a new drive cage, a Smart Array P600 RAID controller (or SAS/SATA HBA) and will be severely crippled in performance. In addition, it can only accommodate 2.5" small-form-factor 1st generation SAS and SATA ...


14

This has been solved. They key is that deduplicated volumes need to have the dedup flag turned off before deletion. This should be done at the pool level as well as the zvol or filesystem level. Otherwise, the deletion is essentially being deduplicated. The process takes time because the ZFS deduplication table is being referenced. In this case, RAM helps. I ...


10

ZFS does not do disk I/O, device drivers below ZFS do disk I/O. If the device does not respond in a timely manner, or as in this case, disrupts all other devices on the expander, then it is not visible as a failure to ZFS. All ZFS sees is a slow I/O. There is a bug in Intel X-25M firmware that affects their behaviour during heavy loads and can cause reset ...


10

My current recommendation is NexentaStor, available in a free community-supported edition and as a commercial offering. Also see: Anybody have experience with using Nexenta? NexentaStor CE or Openfiler? Which do you recommend?


10

FreeBSD 8.2, running ZFS. ZFS includes the following out of the box: Supports NFS & iSCSI out of the box. ZFS includes Snapshots, data checksums, multiple copies, filesystem compression RAID-Z - Similar to RAID-5, but without the RAID-5 write hole. All disk writes are atomic copy-on-write transactions, so the on-disk state is never inconsistent (No ...


10

Certainly ZFS is plenty stable enough to do this kind of thing, there are many very large high-profile and reliable production platforms out there based entirely on ZFS and Nexenta. That said always like to have on-site disk-based backups such as the one you're suggesting AND removable-disk or tape based backups that go off-site daily to protect against ...


10

(assuming you're referring to using dedupe within ZFS versus your backup software) I would not recommend using ZFS native deduplication for your backup system unless you design your storage system specifically for it. Using dedupe in ZFS is extremely RAM intensive. Since the deduplication occurs in real-time as data is streamed/written to the storage pool,...


8

A quick note about Openfiler (and I hear NexentaStor is the same) when used as an iSCSI target - you are almost guaranteed to see timeout errors and targets dropping offline, requiring a reboot of the server to correct. This usually happens under heavy load (though I've seen it happen under light loads, too). We went through hell with Openfiler using iSCSI ...


8

If you're not running snapshots then restoring from backup is your only option. I would advise you to look into snapshotting, as it's extremely useful on fileservers. Users are dumb, and they overwrite/delete files way more often than you can run a backup. Edit: As mentioned by ErikA - providing snapshots on a file server also gives users a easy way of ...


7

On the Nexenta side, there's a volume-check script that's setup to run hourly by default. It will: Check volume health and capacity, clear correctable device errors, validate mountpoints. It also sends a weekly summary report via email. However, there are some things you should consider when planning a Nexenta storage solution for the purposes you've ...


7

ZFS scrub operations operate on some fairly brain-dead principles. Most notably, it only spends time scrubbing when there's nothing else going on. If you poke a pool with just a bit of data access on a fairly constant basis, scrub will effectively starve itself and do nearly nothing. Tunables to explore, with my quick notes on what it does (I last looked ...


6

If you have some monitoring system like Nagios in place, you easily could write a check evaluating the output of zpool list and checking it against thresholds within your comfort zone. If you don't have a monitoring system, you should use this opportunity to install one - a SAN is a critical piece of infrastructure equipment which needs constant monitoring ...


5

As a long time user of Sun/Oracle ZFS 7000-series appliances, I can tell you without question dedupe isn't polished. Never confuse sales with delivery! The salesguys will tell you "Oh, it's been fixed". In real life - my real life - I can tell you 24GB isn't enough to handle the "DDT tables". That is, the back end index which stores the dedupe table. That ...


5

You can make the expert_mode option persistent here. Run option expert_mode = 1 -s. The -s makes the setting permanent. From now on, you'd just need to run !bash to obtain a shell from the nmc console. You can also ssh as the admin user and use su when needed. However, NexentaStor is really meant to be an appliance managed either by the web interface or ...


5

As mentioned, you cannot "upgrade" the OS portion of the system. You can however, export your existing data zpool (tank), install Solaris on the OS disks and re-import them without any trouble. Things that may not survive are iSCSI configurations, but if you're just using it as an NFS NAS, that configuration information is contained in the zfs filesystems. ...


5

There seems to be more momentum behind NexentaStor. You haven't provided much detail on the hardware arrangement other than it being old. What are the CPU/RAM numbers? However, one reason I'd go the NexentaStor route is the presence of inline compression of its storage volumes. Your setup probably isn't suitable for the deduplication features, but the ...


5

Please see the notes about configuring an all-in-one ZFS setup in my post at: Hosting a ZFS server as a virtual guest. If you're talking about creating an SSD pool or adding drives as raw device-mappings (RDM's), disregard the rest of this answer. The preference there would be to run through the HBA versus RDM. Use SAS expanders if needed. The main reasons ...


5

Is it strongly recommended to avoid using RDM's for this purpose. Pass your SATA controller through to the virtualized NexentaStor instance using "PCI Passthrough". This requires a reasonably-current processor to do so. See: Hosting a ZFS server as a virtual guest That said, there's no major issue running ZFS on VMDK's if you have an underlying hardware ...


4

9 times out of 10, this is not NexentaStor at all, except that the default port, 2000, is one claimed by Cisco. Login to the appliance using SSH as root, and type "setup appliance init" and then hit No to the first question then it asks for HTTP/HTTPS and the port to use - you can use HTTP or HTTPS, but change the port to something else, I use 7878. Then try ...


4

In my experience, for ZFS LSI non-raid cards are the best. I use a Dell SAS6i (LSI 1068E rebadge) for internal drives and LSI SAS3801E (dual 3gbps 4x miniSAS ports) for external drives (MD1000) under Solaris 10 (s10u9) for ZFS at work, while I use the LSI SAS 9200-16e (quad 6gbps 4x miniSAS) at home with Nexenta (NCP3). Nothing but good things to say about ...


4

It's not a good idea to share an SSD between pools for reasons of data integrity and performance. First, ZFS needs to be able to trigger the device's onboard cache to flush when a synchronous write is requested, to ensure that the write is really on stable storage before returning to the application. It can only do this if it controls the whole device. ...


4

Let's start by saying that since this is for a home lab it really doesn't matter one way or another unless you have some business need to go with one solution over another. By business need I mean to gain experience with a particular application in order to put it on your resume. I personally do not like NexentaStor because it doesn't provide an UI based ...


4

ZFS handles 4k sectors well as long as the drive advertises them correctly. However, some drives have 4k sectors internally but present a logical 512 sector size to the operating system for backwards compatibility. If ZFS believes the drive, and writes in 512 byte chunks to 4k sectors, you'll suffer a heavy read-modify-write penalty. Have a look at the ...


4

We were recently in the same situation, and no, unfortunately FreeNAS is not ready for the enterprise. With 8.0.1, we had tons of issues with Active Directory integration working intermittently, as well as periodic lockups. All you need to look at are the change logs for recent beta releases to see that there are still major bugs being addressed in the 8.0 ...


4

8Gb Fibre Channel MPIO Enabling compression on your ZFS zvols/LUNs. Who is consuming this data? Are you actually seeing bottlenecks? And remember, it's not always about throughput. Have you taken time to understand your I/O patterns? The mix of read to write? The nature of the transfers? Can you give more information about the setup?


4

I'd suggest dedicated drives for OS and dedicated disks for data. NexentaStor is software RAID and the loss of an OS disk can be painful. You can combine L2ARC and ZIL, but the class of SSDs you're probably using won't be helpful as ZIL devices. I understand the desire to try to use all of these ZFS features, but you really have to do it the right way. I ...


3

Dell's SAS 6/ir 'budget raid' adapters do JBOD unless the drive is in a RAID array. The cards work relatively well from my experience although I cannot attest to whether they work with Nexenta or Solaris. See also this answer


3

We use the LSI 1068 chipset HBA's such as LSI SAS 3081E-R with OpenSolaris. You can also find that chipset OEM'd by Intel and SuperMicro. We use these internally on our ASUS boxes w/o a backplane. It should work on a backplane, but we just have that feature.


3

I agree with those others who have suggested that it's likely the nexenta is ignoring requests from "offsite" addresses as some kind of half-arsed security feature. If the approaches others have suggested don't help, and nexenta can't tell you how to reconfigure the box to disable this feature, you might consider trying to "bounce" the connection off a local ...



Only top voted, non community-wiki answers of a minimum length are eligible