Hot answers tagged

15

The character device /dev/nvme0 is the NVME device controller, and block devices like /dev/nvme0n1 are the NVME storage namespaces: the devices you use for actual storage, which will behave essentially as disks. In enterprise-grade hardware, there might be support for several namespaces, thin provisioning within namespaces and other features. For now, you ...


9

You can RAID NVMe, but you just can't RAID them with a traditional RAID controller. For example, if you're on an Intel CPU and running compatible Intel drives then you can use their Rapid Storage Technology enterprise software to create a RAID between the two disks. I know some Dell servers have a special PCIe controller that does the RAIDing instead of the ...


8

I have found a solution to this problem. It turns out that it is not possible to boot from NVMe in BIOS mode. Only UEFI is supported. However, my install disk was booted in BIOS mode, so the installer tried to configure things as such. The solution was to go into the BIOS setup, force the boot order to only consider UEFI devices and then re-install the ...


8

It's a bad idea to use consumer SSDs in your SDS deployments. VMware VSAN and Microsoft S2D both assume writes will be "atomic", so one ACK-ed by host is actually on persistent memory; consumer SSDs don't have any power outage protection so they MIGHT lose your data. Write endurance is also very different. https://blogs.technet.microsoft.com/filecab/2016/...


8

The 200GB SSD disk that you see is called Instance storage (or ephemeral storage) and is destroyed everytime you stop the instance and created new every time you start the instance. That means two things: Don't store any precious data you want to retain over stop/start - it will be all gone when you stop it. These instance storage disks are great for caches,...


5

By considering random VPS vs dedicated server with NVMe storage, you are comparing a ride on a bus with buying a sportcar. And no, Virtuozzo, implementing container virtualization here aren't the bottleneck. Instead containers usually introduce less overhead over full virtualization technologies. But as you are renting the VPS your don't really have ...


5

I realize that this question wasn't seen very much, but just in case, I'm hoping my results can help out someone in the future (maybe even me the next time I attempt this). I would like to thank Steve E. from Amazon support for helping me get my instance migrated <3 Anyways, there were 2 issues when migrating my Ubuntu 16.04 M3 (PV) instance to an Ubuntu ...


5

(Please don't ask multiple questions in one post - it makes answering really difficult...) queue depth which is the number of outstanding I/O requests [...] which handles the I/O requests and sends the commands to disk (r/w) and it (not strictly?) drops the requests Excessive requests generally aren't dropped - there's just nowhere to queue them in the ...


5

There are different reasons why your real writes were so much inflated. Lets mark some base point: first, let set a baseline: from your zpool iostat output, we can infer a continuous ~1.5 MB/s write stream to each of the mirror leg. So, in 245 days, it add up to 1.5*86400*245 = 32 TB written; the number above already take into account both ZFS recordsize ...


5

In addition to already given advice to reduce recordsize — there's no reason not to use LZ4 compression (zfs set compression=lz4 …) as well by default, thus reducing size even more (and sometimes very significantly).


4

NVMe has log pages and the only thing close to what you want is found in log page 02h, it has the following pieces of info: Data Units Read Data Units Written Host Read Commands Host Write Commands Controller Busy Time So you can take the number of host read and write commands and divide them with the controller busy time, if you do this in relatively ...


4

NVMe is PCIe based, and using different drivers designed for that. You can essentially take an M2 formfactor NVM, pop it into the appropriate adaptor, and run it on any linux, windows or BSD system with appropriate drivers. Essentially all NVMe does is standardises PCIe based SSDs to a single set of drivers, designed to take full advantage of them. ...


4

PowerEdge R730s do not support m.2 drives. They do support NVMe drives - but NVMe is just a protocol, not a physical interface. In the case of Dell servers, the "native" NVMe is delivered via special 2.5" bays in the front of the server such as an R730xd. The disks that go in those bays look like normal 2.5" SSDs with a SATA connector, but they are not. You ...


4

Try adding this to /etc/fstab: /dev/nvme0n1 /database xfs defaults 0 0 That should do the trick.


4

First, ZFS checksumming is not redundant: it is an end-to-end (RAM to physical media) cheksum, while HDD/SSD checksum in as "inside media" error control. To have something similar with classical filesystem, you had to use T10/DIF-compatible disks and controllers, which SATA devices lack (you are forced to use SAS SSD, which are much more expensive). That ...


4

I'm writing a new answer because after further analysis, I don't think the previous answer is correct. If we look at the write_dirty_buffers function, it issues a write request with the REQ_SYNC flag, but it doesn't cause a cache flush, or barrier, to be issued. That is accomplished by the blkdev_issue_flush call, which is appropriately gated by a ...


4

A few items... If this is a leased server, isn't the provider responsible for the health of the equipment? Your ZFS filesystem ashift values, pool txg_timeout and a few other parameters may make sense to review.


4

i'm quite convinced that 100% utilization shown for the underlying NVMe drives that are used to create md-raid is a kernel bug - see the related reports. temporarily upgrading to kernel 5.2 made the 100% utilization go away, we've also benchmarked that md-raid1 on older/newer kernels - including those reporting constant full utilisation - and did not notice ...


4

The simplest solution to this problem is probably just to switch from using IRQ/interrupt mode to polling mode for the NVMe driver. Add this to /etc/modprobe.d/nvme.conf: options nvme poll_queues=4 then run update-initramfs -u, reboot, and you should see a vast reduction in IRQs for NVMe devices. You can also play around with the poll queue count in sysfs ...


3

Think of namespaces as partitions that don't go away when you secure erase the SSD. The controller and namespace(s) are separate. Namespaces can be shared, and can allow remote controllers to access them. Companies like Excelero or Lightbits leverage this technology with their product offerings I believe. In large devices, say 15TB NVME, you may want to ...


3

Using NVMe or traditional HDD drives do not override the purposes of RAID, which is providing better IO rates than a single drive can, replication for increased data durability and larger storage capacity per logical drive. Of course, better performant hardware like NVMe drives fixes some use cases that were traditionally solved by RAID (mainly IO and ...


3

It can be a side-effect of the internal write-intent bitmap used. Use mdadm <dev> --grow --bitmap=none to remove it, and re-try with fio. Anyway, I strongly suggest you against going into production phase without a bitmap-enabled array, as a crash/power outage will force the array to do a full byte-per-byte scan/compare. A write intent bitmap will ...


3

I needed to test this for myself... I purchased four Intel 750 PCIe NVMe SSDs to install in HP ProLiant DL380p Gen8 servers. The servers are not the current generation Intel 2600v3 series CPUs, but rather the 2600v2 CPUs. The takeaway: NVMe is an interface specification. Under Linux, the devices are enumerated as /dev/nvmeXnY, e.g. /dev/nvme0n1 and /dev/...


3

Another way to put your question is this: when doing a checkpoint, i.e. when writing the data in the journal to the actual filesystem, does ext4 flush out the cache (of the rotating disks, in your case) before marking the transaction as completed and updating the journal accordingly? If we look at the source code of jbd2 (which is responsible to handle the ...


3

From https://tobert.github.io/post/2014-04-17-fio-output-explained.html submit and complete represent the number of submitted IOs at a time by fio and the number completed at a time. In the case of the thrashing test used to generate this output, the iodepth is at the default value of 1, so 100% of IOs were submitted 1 at a time placing the results in the ...


3

This answer was inspired by comment by @shodanshok, who commented (so I can't upvote his contribution -- posting an answer instead) Edit 2021/06/09 - iperf3 developers identified a possible problem; newer releases of the package may have a different behaviour, YMMV. See: https://github.com/esnet/iperf/issues/1159 Originally, I was using iperf3 -F .... to ...


2

this article in PCWorld may be of interest.


2

My two cents... NVMe got the various SSD mfg to focus on and adopt a base standard... Basically you can get Nand Flash performance from an SSD connected to NVMe servers for NET less. Also their is more NVMe over fabric features (that I am not that familiar with yet) See https://www.brighttalk.com/webcast/663/132761 Content "The Performance Impact of NVMe ...


2

I wrote an article at Thinkmate that tries to give a nice overview of NVMe and works as a nice little guide on choosing the right drive and system, something that I've found to be missing online. We mostly sell Supermicro servers, and I agree - Things can get a bit confusing... That is why I wrote the article! As for adoption, I can't speak for the industry ...


2

Red Hat kernels are quite different from vanilla ones. Even tunables (eg: sysctl) have significant different default values. I strongly suggest you to stay with RH kernels unless absolutely necessary. Please at least consider using ELRepo kernels, if you really need a more update kernels. Anyway, this question should be issued in the linux kernel mailing ...


Only top voted, non community-wiki answers of a minimum length are eligible