30
votes
Accepted
Why is there both character device and block device for nvme?
The character device /dev/nvme0 is the NVME device controller, and block devices like /dev/nvme0n1 are the NVME storage namespaces: the devices you use for actual storage, which will behave ...
- 4,618
11
votes
Accepted
Supermicro Server fails to boot from NVMe SSD
I have found a solution to this problem. It turns out that it is not possible to boot from NVMe in BIOS mode. Only UEFI is supported. However, my install disk was booted in BIOS mode, so the installer ...
- 271
11
votes
Accepted
Automatically mount SSD instance storage on AWS EC2 in Ubuntu 16.04
The 200GB SSD disk that you see is called Instance storage (or ephemeral storage) and is destroyed everytime you stop the instance and created new every time you start the instance.
That means two ...
- 25.5k
10
votes
Accepted
MongoDB on NVMe in production
You can RAID NVMe, but you just can't RAID them with a traditional RAID controller. For example, if you're on an Intel CPU and running compatible Intel drives then you can use their Rapid Storage ...
- 69.1k
8
votes
Accepted
The setup of S2D that delivers up to 2M IOPS to SQL FCI
It's a bad idea to use consumer SSDs in your SDS deployments. VMware VSAN and Microsoft S2D both assume writes will be "atomic", so one ACK-ed by host is actually on persistent memory; consumer SSDs ...
- 14.1k
7
votes
Unexpected poor performance with NVMe drives on an X9DR3-F
Samsung MZVLB256HBHQ-000L7 are small SSDs (256 GB), so you are going to hit the internal NAND bandwidth bottleneck for any writes spanning multiple GB. You can trim them (losing all data currently ...
- 50.7k
6
votes
Why is there both character device and block device for nvme?
Think of namespaces as partitions that don't go away when you secure erase the SSD.
The controller and namespace(s) are separate. Namespaces can be shared, and can allow remote controllers to access ...
6
votes
Accepted
What does iodepth in fio tests really mean? Is it the queue depth?
(Please don't ask multiple questions in one post - it makes answering really difficult...)
queue depth which is the number of outstanding I/O requests [...] which handles the I/O requests and sends ...
- 1,255
6
votes
Accepted
How to limit ZFS writes on NVME SSD in RAID1 to avoid rapid disk wear?
There are different reasons why your real writes were so much inflated. Lets mark some base point:
first, let set a baseline: from your zpool iostat output, we can infer a continuous ~1.5 MB/s write ...
- 50.7k
6
votes
How does hardware RAID handle firmware updates for the underlying drives?
The NVMe standard specifies a firmware update process. Any NVMe device supporting that process can be updated by a standard routine located in the RAID controller's firmware. The update can be ...
- 11.7k
6
votes
How is Hardware RAID for NVME drives configured on SuperMicro?
Manual (available on the "Resources" page under your link) states there is a SATA RAID. I strongly suspect that is so called "fake" RAID, e.g. not a true hardware RAID. There is no ...
- 12.6k
5
votes
Accepted
Cannot connect to EC2 instance after converting Ubuntu 16 PV to Ubuntu 16 HVM
I realize that this question wasn't seen very much, but just in case, I'm hoping my results can help out someone in the future (maybe even me the next time I attempt this). I would like to thank Steve ...
- 221
5
votes
Accepted
NVMe-Dedicated Server without Virtualisation: Big speed benefit?
By considering random VPS vs dedicated server with NVMe storage, you are comparing a ride on a bus with buying a sportcar.
And no, Virtuozzo, implementing container virtualization here aren't the ...
- 1,342
5
votes
How to limit ZFS writes on NVME SSD in RAID1 to avoid rapid disk wear?
In addition to already given advice to reduce recordsize — there's no reason not to use LZ4 compression (zfs set compression=lz4 …) as well by default, thus reducing size even more (and sometimes very ...
- 9,558
5
votes
How is Hardware RAID for NVME drives configured on SuperMicro?
Try following the steps outlined here. In short:
press DEL during boot to enter BIOS
go to Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel VMD Technology
set ...
- 50.7k
5
votes
Accepted
Understanding ZFS NVMe benchmarks with FIO
You are probably running fio on a recordsize=128k dataset. When reading/writing 4K blocks you are going to see up to 32x I/O amplification.
To get maximum IOPs you can create a new dataset with ...
- 50.7k
4
votes
Accepted
Mounted disk disappears after reboot
Try adding this to /etc/fstab:
/dev/nvme0n1 /database xfs defaults 0 0
That should do the trick.
- 301
4
votes
ZFS: Configuration advice 1x NVMe as ARC and ZIL and 4x SSDs for zvols for virtualization
First, ZFS checksumming is not redundant: it is an end-to-end (RAM to physical media) cheksum, while HDD/SSD checksum in as "inside media" error control. To have something similar with classical ...
- 50.7k
4
votes
Installing 960 Evo NVMe to PowerEdge R730
PowerEdge R730s do not support m.2 drives. They do support NVMe drives - but NVMe is just a protocol, not a physical interface. In the case of Dell servers, the "native" NVMe is delivered via special ...
- 69.1k
4
votes
NVMe latency monitoring
NVMe has log pages and the only thing close to what you want is found in log page 02h, it has the following pieces of info:
Data Units Read
Data Units Written
Host Read Commands
Host Write Commands
...
- 1,091
4
votes
Accepted
Disabling ext4 write barriers when using an external journal
I'm writing a new answer because after further analysis, I don't think the previous answer is correct.
If we look at the write_dirty_buffers function, it issues a write request with the REQ_SYNC flag,...
- 332
4
votes
What does iodepth in fio tests really mean? Is it the queue depth?
From https://tobert.github.io/post/2014-04-17-fio-output-explained.html
submit and complete represent the number of submitted IOs at a time by fio and the number completed at a time. In the case of ...
- 1,091
4
votes
How to limit ZFS writes on NVME SSD in RAID1 to avoid rapid disk wear?
A few items...
If this is a leased server, isn't the provider responsible for the health of the equipment?
Your ZFS filesystem ashift values, pool txg_timeout and a few other parameters may make ...
- 198k
4
votes
Accepted
Nearly 100% disk utilization after upgrading to debian 10 without any reason
i'm quite convinced that 100% utilization shown for the underlying NVMe drives that are used to create md-raid is a kernel bug - see the related reports. temporarily upgrading to kernel 5.2 made the ...
- 30.2k
4
votes
Accepted
How to explain these fio bandwidth results?
The first fio (the one with --numjobs=1) sequentially executes any read operation, having no benefit from your stripe config apart for quick read-ahead/prefetch: iodepth only applies to async reads ...
- 50.7k
4
votes
How can I override IRQ affinity for NVME devices
The simplest solution to this problem is probably just to switch from using IRQ/interrupt mode to polling mode for the NVMe driver.
Add this to /etc/modprobe.d/nvme.conf:
options nvme poll_queues=4
...
- 126
4
votes
Accepted
Health Tests on NVMe
SMART self-test were conceived for mechanical disks. SATA SSDs almost completely mirrors earlier HDD interface-level behavior supporting such self-test but not doing very much when you run it, ...
- 50.7k
4
votes
Install an NVMe SSD on a Dell Precision 5820 tower
The 5820 dates from 2017, and Dell were kind of late to the NVMe party (as were most mainstream enterprise servers/appliances). Even on their high-density storage servers NVMe was kind of hit and miss ...
- 69.1k
3
votes
Accepted
Does any PCIe slot support NVMe SSD's automatically or do I need a specific M.2 slot?
NVMe is a standard for storage over PCI express (and later revised to support other transports, but that isn't relavent here).
PCIe is an electrical interface that can be delivered on multiple ...
- 4,296
3
votes
MongoDB with NVMe SSD
Using NVMe or traditional HDD drives do not override the purposes of RAID, which is providing better IO rates than a single drive can, replication for increased data durability and larger storage ...
- 1,179
Only top scored, non community-wiki answers of a minimum length are eligible