Questions tagged [storage]
Primarily questions about disk storage, sometimes questions about application storage or cloud storage. Always questions about storing data.
1,963
questions
190
votes
2
answers
48k
views
What are the different widely used RAID levels and when should I consider them?
This is a Canonical Question about RAID levels.
What are:
the RAID levels typically used (including the RAID-Z family)?
deployments are they commonly found in?
benefits and pitfalls of each?
- 101k
162
votes
5
answers
33k
views
Are SSD drives as reliable as mechanical drives (2013)?
SSD drives have been around for several years now. But the issue of reliability still comes up.
I guess this is a follow up from this question posted 4 years ago, and last updated in 2011. It's now ...
- 14.6k
114
votes
6
answers
37k
views
Why is Enterprise Storage so expensive?
This is a Canonical Question about the Cost of Enterprise Storage.
See also the following question:
What's the best way to explain storage issues to developers and other users
...
82
votes
24
answers
70k
views
Storing a million images in the filesystem
I have a project that will generate a huge number of images. Around 1,000,000 for start. They are not large images so I will store them all on one machine at start.
How do you recommended on ...
- 1,509
78
votes
5
answers
158k
views
What is the difference between SAN, NAS and DAS?
What is the difference between SAN, NAS and DAS?
sevugarajan
76
votes
13
answers
72k
views
ZFS vs XFS
We're considering building a ~16TB storage server. At the moment, we're considering both ZFS and XFS as filesystem. What are the advantages, disadvantages? What do we have to look for? Is there a ...
- 863
74
votes
13
answers
51k
views
SSD or HDD for server
Issue
I have read many discussions about storage, and whether SSDs or classic HDDs are better. I am quite confused. HDDs are still quite preferred, but why?
Which is better for active storage? For ...
- 891
73
votes
6
answers
248k
views
How to check disk I/O utilization per process?
I'm having a problem with a Linux system and I have found sysstat and sar to report huge peaks of disk I/O, average service time as well as average wait time.
How could I determine which process is ...
- 1,344
62
votes
8
answers
7k
views
What are the advantages of tape drives?
IBM still develop and sell tape drives today. The capacity of them seems to be on a par with today's hard drives, but the search time and transfer rate are both significantly lower than that of hard ...
- 2,877
62
votes
13
answers
118k
views
Windows 2008: WinSXS directory growing uncontrollably, blocking server
I run a (remotely hosted) virtual Server with Windows 2008 Server for a client.
Initially, it had 10 GB of space.
During the course of a few weeks - during which nothing was done on the machine ...
- 2,178
53
votes
6
answers
67k
views
How to move files between two S3 buckets with minimum cost?
I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone.
How could I ...
- 823
52
votes
1
answer
2k
views
Fibre channel long distance woes
I need a fresh pair of eyes.
We're using a 15km fibre optic line across which fibrechannel and 10GbE is multiplexed (passive optical CWDM). For FC we have long distance lasers suitable up to 40km (...
- 2,854
48
votes
7
answers
15k
views
Probability of Blade Chassis Failure
In my organisation we are thinking about buying blade servers - instead of rack servers. Of course technology vendors also make them sound very nice. A concern, that I read very often in different ...
- 737
48
votes
10
answers
17k
views
Linux on VMware - why use partitioning?
When installing Linux VMs in a virtualized environment (ESXi in my case), are there any compelling reasons to partition the disks (when using ext4) rather than just adding separate disks for each ...
- 553
47
votes
3
answers
53k
views
What are the merits of SAS vs SATA drives? [closed]
What are the merits of buying SAS drives over SATA drives, or vice versa?
- 4,785
45
votes
9
answers
12k
views
Is it safe to use consumer MLC SSDs in a server?
We (and by we I mean Jeff) are looking into the possibility of using Consumer MLC SSD disks in our backup data center.
We want to try to keep costs down and usable space up - so the Intel X25-E's ...
- 37.6k
45
votes
11
answers
33k
views
Does an unplugged hard drive used for data archival deteriorate?
If I were to archive data on a hard drive, unplug it, and set it on a (not dusty, temperature-controlled) shelf somewhere, would that drive deteriorate much?
How does the data retention of an ...
- 1,862
40
votes
8
answers
3k
views
What's the best way to explain storage issues to developers and other users
When server storage gets low developers all start to moan, "I can get a 1 TB drive at Walmart for 100 bucks, what's the problem".
How can the complexities of storage be explained to developers so ...
- 735
40
votes
4
answers
25k
views
What is the use case of a SD slot on motherboard?
My HP Proliant ML110 G7 has a full-size SD slot on the motherboard. What is its use case?
The PDF manual mentions it on page 10: item 17 just to show its placement, but nothing more. In a later ...
- 485
37
votes
2
answers
8k
views
What is a Storage Area Network, and which benefits does it have over different storage solutions?
I'm proposing this to be a canonical question about enterprise-level Storage Area Networks.
What is a Storage Area Network (SAN), and how does it work?
How is it different from a Network Attached ...
- 71.3k
35
votes
7
answers
13k
views
Best way to test new HDD's for a cheap storage server
I want to build a storage server and bought 10 x 2TB WD RED's. The HDD's just arrived.
Is there any tool you guys use to check for bad drives or to best defend against infant mortality before copying ...
- 1,155
34
votes
8
answers
14k
views
Why do IOPS matter?
I understand what IOPS and throughput are. Throughput measures data flow as MB/s and IOPS says how many I/O operations are happening per second.
What I don't understand is why many storage services ...
- 513
31
votes
7
answers
165k
views
How to monitor the hard disk status behind Dell PERC H710 Raid Controller with CentOS 6?
I have a Dell server running CentOS 6 using PERC H710 Raid Controller card with Raid 5 setup and I want to monitor the hard disk failure/working status behind the Raid Controller.
Then I should be ...
- 685
31
votes
8
answers
10k
views
Write speed requirement : 1.1GB/s possibilities?
We will have a machine at work, that on peak performance, should be able to push 50 ("write heads") x 75GB of data per hour. That's peak performance of ~1100MB/s write speed. To get that from the ...
- 749
31
votes
11
answers
21k
views
Distributed Storage Filesystem - Which one/Is there a ready to use product?
With Hadoop and CouchDB all over in Blogs and related news what's a distributed-fault-tolerant storage (engine) that actually works.
CouchDB doesn't actually have any distribution features built-in, ...
30
votes
2
answers
16k
views
Why are enterprise SAS disk enclosures seemingly so expensive?
I will begin by stating that I do not believe this is a duplicate of Why is Enterprise Storage so expensive?.
My question is specifically about SAS drive enclosures, and justifying their expense.
...
- 365
29
votes
5
answers
181k
views
What is the difference between a HBA card and a RAID card?
I thought I knew the difference between HBA and RAID.
In my mind, HBA is offloading from the main motherboard/CPU and is simply JBOD... usually has an external SAS ports, whilst a RAID card does the ...
- 3,556
29
votes
3
answers
19k
views
Linux - real-world hardware RAID controller tuning (scsi and cciss)
Most of the Linux systems I manage feature hardware RAID controllers (mostly HP Smart Array). They're all running RHEL or CentOS.
I'm looking for real-world tunables to help optimize performance for ...
- 198k
29
votes
3
answers
11k
views
Tuning iSCSI storage
This is a Canonical Question about iSCSI we can use as a reference.
iSCSI is a protocol that puts SCSI commands as payload into TCP network packets. As such, it is subject to a different set of ...
28
votes
5
answers
10k
views
Do I need to RAID Fusion-io cards?
Can I run reliably with a single Fusion-io card installed in a server, or do I need to deploy two cards in a software RAID setup?
Fusion-io isn't very clear (almost misleading) on the topic when ...
- 198k
27
votes
1
answer
15k
views
How should I set up my Fibre Channel (FC) network?
How do FC switches work, and how should I configure mine?
- 8,881
27
votes
1
answer
43k
views
KVM and Libvirt - How do I hotplug a new virtio disk?
I am trying to hot-add a file-based disk to a running KVM virtual server. I've created a new disk from scratch using the command
dd of=/home/cloud/vps_59/test.img bs=1 seek=5G count=0
and I was ...
- 271
27
votes
9
answers
17k
views
What kind of storage do people actually use for VMware ESX servers?
VMware and many network evangelists try to tell you that sophisticated (=expensive) fiber SANs are the "only" storage option for VMware ESX and ESXi servers. Well, yes, of course. Using a SAN is fast, ...
- 914
26
votes
4
answers
8k
views
Is it reasonable to use NFS on a production web server?
Can NFS be reasonably used on production servers as a means of connecting a compute server to a storage server, assuming the connection is over a LAN 1Gbe or 10Gbe connection?
There's obviously some ...
- 371
26
votes
5
answers
10k
views
BBWC: in theory a good idea but has one ever saved your data?
I'm familiar with what a BBWC (Battery-backed write cache) is intended to do - and previously used them in my servers even with good UPS. There are obvously failures it does not provide protection for....
- 22.4k
26
votes
2
answers
65k
views
How exactly does a SAS SFF-8087 breakout cable work? + RAID/connection questions
Please let me know if my question does not makes any sense as I am not sure if I am interpreting it correctly from my thoughts due to my lack of technical knowledge on this.
If I am using a ...
- 1,055
26
votes
12
answers
8k
views
Storing and backing up 10 million files on Linux
I run a website where about 10 million files (book covers) are stored in 3 levels of subdirectories, ranging [0-f]:
0/0/0/
0/0/1/
...
f/f/f/
This leads to around 2400 files per directory, which is ...
- 4,607
25
votes
2
answers
8k
views
ZFS Data Loss Scenarios
I'm looking toward building a largish ZFS Pool (150TB+), and I'd like to hear people experiences about data loss scenarios due to failed hardware, in particular, distinguishing between instances where ...
- 557
24
votes
8
answers
633k
views
How to format as FAT32 from Windows 7/Vista
What is the best way to format a USB drive with FAT32 (for Mac compatibility) from within Windows 7/Vista?
I ask because the Disk Management only lets you pick exFAT (because the disk is over 32&...
- 848
24
votes
2
answers
14k
views
How to decide between a docker volume container and a docker volume?
After reading the docs I found myself somewhat confused as to how best to manage productive application/service data.
There seem to be 3 options:
Simply map volume to host directory (i.e. -v ...
- 709
23
votes
1
answer
22k
views
Hosting a ZFS server as a virtual guest
I'm still new to ZFS. I've been using Nexenta but I'm thinking of switching to OpenIndiana or Solaris 11 Express. Right now, I'm at a point of considering virtualizing the ZFS server as a guest within ...
- 3,885
23
votes
2
answers
13k
views
Newly created XFS filesystem shows 78 GB used
We have a 12 TB RAID 6 array which is supposed to be set up as a single partition with an XFS file system. On creating the new file system, it says it has 78 GB in use, but there are no files on ...
- 1,243
23
votes
4
answers
25k
views
what is exactly an URE?
I have been looking into RAID5 Vs RAID6 lately and I keep seeing that RAID5 is not secure enough anymore because of the URE ratings and increasing size of the drives. Basically, most of the content I ...
- 408
23
votes
4
answers
18k
views
How effective is LSI CacheCade SSD storage tiering?
LSI offers their CacheCade storage tiering technology, which allows SSD devices to be used as read and write caches to augment traditional RAID arrays.
Other vendors have adopted similar technologies;...
- 198k
22
votes
8
answers
76k
views
RAID-5: Two disks failed simultaneously?
We have a Dell PowerEdge T410 server running CentOS, with a RAID-5 array containing 5 Seagate Barracuda 3 TB SATA disks. Yesterday the system crashed (I don't know how exactly and I don't have any ...
- 444
22
votes
3
answers
4k
views
Can "enterprise" drives be safely replaced by near/midline in some situations?
When specifying servers, like (I would assume) many engineers who aren't experts in storage, I'll generally play it safe (and perhaps be a slave to marketing) by standardising on a minimum of 10k SAS ...
- 1,872
22
votes
3
answers
15k
views
Are RDX removable disks a good replacement for LTO tape?
Over a period of three weeks, I experienced six complete failures of LTO-1 and LTO-2 tape drives at client sites. Some had failed mechanisms. Others lost the ability to write reliably. These were HP ...
- 198k
21
votes
8
answers
5k
views
Do snapshots + RAID count as a good on-site backup solution?
The two main reasons I can think of for taking backups seems to be taken care of when I use both snapshots and RAID together with btrfs. (By RAID here, I mean RAID1 or 10)
Accidental deletion of data:...
- 347
21
votes
3
answers
23k
views
Economical way to get many harddrives into rack mount?
Please bear with me as being a bit of a newcomer at 19" rack-mounted equipment.
I've thought a fair bit lately about the best way of getting 4x or 6x of 2.5" hard drives into my rack and are ...
- 1,599
21
votes
1
answer
2k
views
Birth time of files are missing if file is created in a logical volume with size less than 512 MB
I have a volume group in my system, created with single physical volume.
I'm creating two logical volumes - one with size 100M and one with size 512M.
Any file created on LV with 100M size does not ...
- 363