30 votes
Accepted

Why is there both character device and block device for nvme?

The character device /dev/nvme0 is the NVME device controller, and block devices like /dev/nvme0n1 are the NVME storage namespaces: the devices you use for actual storage, which will behave ...
telcoM's user avatar
  • 4,618
11 votes
Accepted

Supermicro Server fails to boot from NVMe SSD

I have found a solution to this problem. It turns out that it is not possible to boot from NVMe in BIOS mode. Only UEFI is supported. However, my install disk was booted in BIOS mode, so the installer ...
Keno Fischer's user avatar
11 votes
Accepted

Automatically mount SSD instance storage on AWS EC2 in Ubuntu 16.04

The 200GB SSD disk that you see is called Instance storage (or ephemeral storage) and is destroyed everytime you stop the instance and created new every time you start the instance. That means two ...
MLu's user avatar
  • 25.5k
10 votes
Accepted

MongoDB on NVMe in production

You can RAID NVMe, but you just can't RAID them with a traditional RAID controller. For example, if you're on an Intel CPU and running compatible Intel drives then you can use their Rapid Storage ...
Mark Henderson's user avatar
8 votes
Accepted

The setup of S2D that delivers up to 2M IOPS to SQL FCI

It's a bad idea to use consumer SSDs in your SDS deployments. VMware VSAN and Microsoft S2D both assume writes will be "atomic", so one ACK-ed by host is actually on persistent memory; consumer SSDs ...
BaronSamedi1958's user avatar
7 votes

Unexpected poor performance with NVMe drives on an X9DR3-F

Samsung MZVLB256HBHQ-000L7 are small SSDs (256 GB), so you are going to hit the internal NAND bandwidth bottleneck for any writes spanning multiple GB. You can trim them (losing all data currently ...
shodanshok's user avatar
  • 50.6k
6 votes

Why is there both character device and block device for nvme?

Think of namespaces as partitions that don't go away when you secure erase the SSD. The controller and namespace(s) are separate. Namespaces can be shared, and can allow remote controllers to access ...
Phil Dominguez's user avatar
6 votes
Accepted

What does iodepth in fio tests really mean? Is it the queue depth?

(Please don't ask multiple questions in one post - it makes answering really difficult...) queue depth which is the number of outstanding I/O requests [...] which handles the I/O requests and sends ...
Anon's user avatar
  • 1,255
6 votes
Accepted

How to limit ZFS writes on NVME SSD in RAID1 to avoid rapid disk wear?

There are different reasons why your real writes were so much inflated. Lets mark some base point: first, let set a baseline: from your zpool iostat output, we can infer a continuous ~1.5 MB/s write ...
shodanshok's user avatar
  • 50.6k
6 votes

How does hardware RAID handle firmware updates for the underlying drives?

The NVMe standard specifies a firmware update process. Any NVMe device supporting that process can be updated by a standard routine located in the RAID controller's firmware. The update can be ...
Zac67's user avatar
  • 11.6k
6 votes

How is Hardware RAID for NVME drives configured on SuperMicro?

Manual (available on the "Resources" page under your link) states there is a SATA RAID. I strongly suspect that is so called "fake" RAID, e.g. not a true hardware RAID. There is no ...
Nikita Kipriyanov's user avatar
5 votes
Accepted

Cannot connect to EC2 instance after converting Ubuntu 16 PV to Ubuntu 16 HVM

I realize that this question wasn't seen very much, but just in case, I'm hoping my results can help out someone in the future (maybe even me the next time I attempt this). I would like to thank Steve ...
Alex's user avatar
  • 221
5 votes
Accepted

NVMe-Dedicated Server without Virtualisation: Big speed benefit?

By considering random VPS vs dedicated server with NVMe storage, you are comparing a ride on a bus with buying a sportcar. And no, Virtuozzo, implementing container virtualization here aren't the ...
NStorm's user avatar
  • 1,342
5 votes

How to limit ZFS writes on NVME SSD in RAID1 to avoid rapid disk wear?

In addition to already given advice to reduce recordsize — there's no reason not to use LZ4 compression (zfs set compression=lz4 …) as well by default, thus reducing size even more (and sometimes very ...
poige's user avatar
  • 9,543
5 votes

How is Hardware RAID for NVME drives configured on SuperMicro?

Try following the steps outlined here. In short: press DEL during boot to enter BIOS go to Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel VMD Technology set ...
shodanshok's user avatar
  • 50.6k
5 votes
Accepted

Understanding ZFS NVMe benchmarks with FIO

You are probably running fio on a recordsize=128k dataset. When reading/writing 4K blocks you are going to see up to 32x I/O amplification. To get maximum IOPs you can create a new dataset with ...
shodanshok's user avatar
  • 50.6k
4 votes
Accepted

Mounted disk disappears after reboot

Try adding this to /etc/fstab: /dev/nvme0n1 /database xfs defaults 0 0 That should do the trick.
Soruk's user avatar
  • 301
4 votes

ZFS: Configuration advice 1x NVMe as ARC and ZIL and 4x SSDs for zvols for virtualization

First, ZFS checksumming is not redundant: it is an end-to-end (RAM to physical media) cheksum, while HDD/SSD checksum in as "inside media" error control. To have something similar with classical ...
shodanshok's user avatar
  • 50.6k
4 votes

Installing 960 Evo NVMe to PowerEdge R730

PowerEdge R730s do not support m.2 drives. They do support NVMe drives - but NVMe is just a protocol, not a physical interface. In the case of Dell servers, the "native" NVMe is delivered via special ...
Mark Henderson's user avatar
4 votes

NVMe latency monitoring

NVMe has log pages and the only thing close to what you want is found in log page 02h, it has the following pieces of info: Data Units Read Data Units Written Host Read Commands Host Write Commands ...
Baruch Even's user avatar
  • 1,091
4 votes
Accepted

Disabling ext4 write barriers when using an external journal

I'm writing a new answer because after further analysis, I don't think the previous answer is correct. If we look at the write_dirty_buffers function, it issues a write request with the REQ_SYNC flag,...
jcharaoui's user avatar
  • 332
4 votes

What does iodepth in fio tests really mean? Is it the queue depth?

From https://tobert.github.io/post/2014-04-17-fio-output-explained.html submit and complete represent the number of submitted IOs at a time by fio and the number completed at a time. In the case of ...
Baruch Even's user avatar
  • 1,091
4 votes

How to limit ZFS writes on NVME SSD in RAID1 to avoid rapid disk wear?

A few items... If this is a leased server, isn't the provider responsible for the health of the equipment? Your ZFS filesystem ashift values, pool txg_timeout and a few other parameters may make ...
ewwhite's user avatar
  • 198k
4 votes
Accepted

Nearly 100% disk utilization after upgrading to debian 10 without any reason

i'm quite convinced that 100% utilization shown for the underlying NVMe drives that are used to create md-raid is a kernel bug - see the related reports. temporarily upgrading to kernel 5.2 made the ...
pQd's user avatar
  • 30.2k
4 votes
Accepted

How to explain these fio bandwidth results?

The first fio (the one with --numjobs=1) sequentially executes any read operation, having no benefit from your stripe config apart for quick read-ahead/prefetch: iodepth only applies to async reads ...
shodanshok's user avatar
  • 50.6k
4 votes

How can I override IRQ affinity for NVME devices

The simplest solution to this problem is probably just to switch from using IRQ/interrupt mode to polling mode for the NVMe driver. Add this to /etc/modprobe.d/nvme.conf: options nvme poll_queues=4 ...
Andrew H's user avatar
  • 126
4 votes
Accepted

Health Tests on NVMe

SMART self-test were conceived for mechanical disks. SATA SSDs almost completely mirrors earlier HDD interface-level behavior supporting such self-test but not doing very much when you run it, ...
shodanshok's user avatar
  • 50.6k
4 votes

Install an NVMe SSD on a Dell Precision 5820 tower

The 5820 dates from 2017, and Dell were kind of late to the NVMe party (as were most mainstream enterprise servers/appliances). Even on their high-density storage servers NVMe was kind of hit and miss ...
Mark Henderson's user avatar
3 votes
Accepted

Does any PCIe slot support NVMe SSD's automatically or do I need a specific M.2 slot?

NVMe is a standard for storage over PCI express (and later revised to support other transports, but that isn't relavent here). PCIe is an electrical interface that can be delivered on multiple ...
Peter Green's user avatar
  • 4,296
3 votes

MongoDB with NVMe SSD

Using NVMe or traditional HDD drives do not override the purposes of RAID, which is providing better IO rates than a single drive can, replication for increased data durability and larger storage ...
ma.tome's user avatar
  • 1,179

Only top scored, non community-wiki answers of a minimum length are eligible