0

Our wordpress web site is several years old and has many posts indexed and ranking well on google. With any serious traffic my wordpress server tanks - and this happens even after several rounds of wordpress optimization. We have had enough of wordpress issues and have decided to migrate.

We are migrating from wordpress to static site for better performance so that pages are not rendered for every request and static html,css,js and image files can be served directly by nginx web server instead of hitting another server at the back end.

The issue is that we have over 400,000 posts and every post will have a static page and hence a static folder in which we will be storing the relevant files like the html and image file for that post. So our main web folder will have over 400,000 subfolders. Will that be an issue on linux? Or will that be an issue for my web server performance? Is there anything on the hosting side that I should care about in this situation?

Has anyone here tried using ext4 with nginx with large number of subfolders in a folder? Does it really affect the performance? There are conflicting reports about performance of ext4 handling large number of folders... We do not want to migrate with added complexity unless it is really necessary. The migration is a big exercise already for us :) and we would like to keep it as simple as possible unless there is a real risk of performance degradation. Has anyone used nginx webserver with large number of subfolders or files in a single folder?

Thank you in advance.

8
  • Please take a look at this stackoverflow.com/questions/466521/… for information regarding number of files per folder sizing limits and perfeormance penalties. Jul 20 '20 at 8:45
  • Why not store your content in a database? Jul 20 '20 at 8:50
  • Maybe you can customise your folder structure for the static site to be something like example.com/year/month/postname or similar, to reduce the impact. Gerard, I think using a database somewhat defeats the purpose of having a static site. Wordpress is very resource hungry and not the easiest to optimise.
    – Tim
    Jul 20 '20 at 9:47
  • Instead of moving into static solution, I would first implement caching in nginx. Jul 20 '20 at 10:19
  • Our urls are already well distributed and ranking high on google - so we dont want to change the url structure. By default nginx serves the page from the same folder name as that in the url - for example an incoming request to nginx on example.com/folder1/index.html will look for folder1 in the root folder specified. Can this behavior be changed so that the request goes to /root_folder/2020/07/01/folder1/index.html or any such scheme - so that we dont have to change the incoming urls - just change the location of the file on the server... Jul 20 '20 at 12:26
2

Here is a method of reducing the number of directories as given in Artem S. Tashkinov's answer and configuring nginx to obey the original URL structure.

Create a directory structure for each URL with the first two characters of each URL being a directory under the document root. Place the static content beginning with those two characters under that directory.

The nginx location that makes this possible is pretty simple:

    location ~ /(..) {
            root /srv/www/example.com/$1;
    }

This simply takes the first two characters of the URL after the initial / and appends that to the document root.

Note that this requires everything to be moved into two character subdirectories. That includes the top level /index.html, which must be placed at $root/in/index.html. As another example, a top level URL path /images must be moved to $root/im/images. The original document root will contain nothing but these two-character directory names.

Your document URLs will remain unchanged. For example, a blog post accessible at /15-things-to-do-when-visiting-dubai will be on your filesystem at $root/15/15-things-to-do-when-visiting-dubai/index.html, but still accessible at the original URL. (Note that if your original URLs did not have a trailing slash, one will be added, and 301 redirects are generated for SEO preservation.)

In the end the document root directory will have only a few thousand directories at most, and each of them will probably have at most a few hundred directories or files. This is very easily handled by any Linux filesystem.

7
  • Thank you so much.. This looks workable since there can only be about couple of thousand 2 letter combinations at the most considering hyphens and numbers. My only concern is that we may still end up with many subfolders under some commonly occurring 2 letter characters - but even with such random lumping up we would be much better off than having half a million folders in the root. Is there any way to even out this distribution besides the two lettered approach? Would first byte from a sha1 of the url lead to a more uniform distribution? Would there be any way to implement this in nginx? Jul 20 '20 at 23:06
  • @MuhammadEbrahym Anything you want to do will make nginx do some work to process the request. Do you really want it to try to calculate an SHA hash of every file and then look them up in a database? Some backend app would be better at that, but you said you wanted to avoid such approaches. Jul 20 '20 at 23:34
  • true - database lookups are to be avoided - i was hoping for some sort of a pattern matching approach as you suggested but with higher chance of uniformity of distribution. Would nginx be able to hash incoming urls and get the first byte and append it to the url request in the location directive - like the first 2 letters that you suggested? Jul 21 '20 at 7:12
  • Has anyone here tried using ext4 with nginx with large number of subfolders in a folder? Does it really affect the performance? There are conflicting reports about performance of ext4 handling large number of folders... We do not want to migrate with added complexity unless it is really necessary. The migration is a big exercise already for us :) and we would like to keep it as simple as possible unless there is a real risk of performance degradation. Jul 21 '20 at 8:08
  • @MuhammadEbrahym Someone probably has, but I switched to XFS years ago. I already know it is much better at handling large directories. It has been the default for Red Hat Enterprise Linux and CentOS since 2013. If you are using something else like Ubuntu you have to select XFS explicitly at installation time. Jul 21 '20 at 13:44
1

In a best case scenario, you should avoid having more than a few thousand files per directory in most file systems because otherwise traversing it will take too much time and resources.

You could create a directory structure such as:

  • 00
  • 01
  • 02

...

  • FE
  • FF

That will give you 256 directories, and you can nest them infinitely.

Or you could try organizing posts by /YYYY/MM/DD/$UID-post-title

4
  • if we change the folder structure for file storage then would the urls need to be changed? These urls are ranking high on google and we would not want to change the url for the posts. Is there a way to tell nginx to look up a certain folder for an incoming url request? I m4ean can we keep the url structure same and ask nginx to lookup a json file and serve a static file from a folder location that is different from the url? Jul 20 '20 at 12:20
  • You can always set up redirects. Jul 20 '20 at 12:29
  • Thanks for the reply... is there a way to automate such redirects on nginx based on some json file or some logic like this: If the hash of the folder name in the url starts with 00 then change url to /00/folder name OR some other logic? What logic would you suggest to minimize the number of sub-folders and distribute them evenly? Also will such redirection affect google ranks of our urls - now that the url is being permanently redirected to another url? Jul 20 '20 at 12:42
  • 1
    Please read this post: tendenci.com/help-files/nginx-redirect-maps As to how you organize your directories - it's really up to. Jul 20 '20 at 12:48
0

Since the site is static how about hosting it on AWS S3 and making it AWS's problem?

S3 can host websites directly and each bucket can store a virtually unlimited number of files (which it calls objects) in a bucket. You used to have to be very careful about file naming but that has largely been solved and isn't a big issue now. Read the performance guidelines though, and test well.

S3 isn't always cheap for storage or bandwidth, you should use the AWS calculator to work out your costs (the new calculator doesn't seem to do S3 pricing). You can mitigate traffic costs somewhat by adding caching headers to every object when you upload it to S3, then putting your S3 bucket behind CloudFlare CDN (see this question). CloudFlare have free and paid plans, but with this much traffic and content I expect you'd want a paid plan.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.