How do the filepaths work for websites with multiple storage drivers/servers?

Question

So I am developing this site on my localhost and currently when I 'upload' a file, I have set the file to be stored in an uploads folder, with sub-folders being generated consisting of the first 2 characters of the assigned file ID. For example, something like this:

/uploads/ab/abcde.mp4

Now that's nice and easy when I have the entire site on one machine/drive, but wanting to plan the site with scalability in mind, how would I plan for if I ever need to add extra storage to the site?

The sites would be hosted using Apache (probably) and Ubuntu. I have a few questions though:

1. For a single, let's say, 16GB RAM VPS, how much additional storage would it be likely you could add?

2. If additional drives are added, how would you point the server to that file? For example, let's say I have the following HTML:

<img src='uploads/aa/aaaaa.jpg'/> <-- this file is on one drive
<img src='uploads/bb/bbbbb.jpg'/> <-- this file is on another drive

On Windows, if you plug in any external device or whatever, it obviously appears as drive E:\ or something - is that also true if you add additional storage into a webserver? So that file A might be in uploads/..., but file B would have to be saved in E:/uploads/...?

I thought this could have been how it would work and then you'd use Apache/Nginx to perhaps map different urls to different drives - for example, the full urls could be http://d1.mysite.com/uploads/aa/aaaaa.jpg and http://d2.mysite.com/uploads/bb/bbbbb.jpg (note that the difference is "d1" and "d2") where, with Apache, you could have a rule in .htaccess to map the different incoming requests to different drives, based on the URL. I have tried searching for things like "Apache/.htacess map url/request to different drive" etc, but haven't been able to find anything, which leads me to think this isn't how you do it.

If someone could point me in the right direction, I'd really appreciate it

Answer 1

On server systems and with modern operating systems you have more options than accessing and using each disk as a separate unique device.

Fairly typical are logical volume management where, in software, one or more disks/partitions can be grouped together and addressed as a single volume. Adding a physical new disk to the system allows the administrator to add it to an existing logical volume, making it bigger and then (after growing the filesystem) you have additional free space that you and your applications can use without any reconfiguration there...

With RAID and storage controllers you can achieve more or less the same in hardware too.

For virtual servers the "disk" is usually a virtual disk image that can be increased at will to almost any size (limited mostly only by the virtualisation platform used). You need more storage: simply increase the VPS (disk) size, grow the filesystem and you're done. If you read the limits there: often one or more additional volumes can also be added.

Answer 2

Yes, you can have 50 TB in one volume. Be sure to also budget performance, try and estimate the number of IOPS it needs based on the number of requests it will serve.

Some cloud providers offer managed file shares, like Azure Files or Amazon Elastic File System. These you create a volume big as you want and mount it via NFS on your instances.

When doing it yourself, create a storage array out of disks. For example, XFS on Linux LVM on top of a Linux md RAID 6. Or, a ZFS pool. Either of these can serve NFS share to clients. And storage oriented server form factors exist that allow dozens of drives in one chassis.

Either of these giant file share solutions require the application to pick a file name. And file system performance tends to degrade badly above a hundred thousand files in a directory. Which may require a more even scheme of distributing files, like hashing the file name.

Or, consider replacing file shares with object storage. A content addressable API storage so you don't deal with file names. For example, S3 compatible storage. Build MinIO storage nodes, buy an EMC ECS array, or use actual AWS, all allow the client to use S3 buckets. These are very abstract from the underlying storage volumes or any one server, and allow scale out growth to very large numbers of objects.