9

I am serving a static folder with NGINX.

Inside that folder I have one particular file that is 60GB, but it is only meant to be read in small chunks, using an HTTP request like the following:

enter image description here

So if a user tries to download the entire file, NGINX should refuse it.

There are someways I see that this could be done, for eg.:

  1. Check if the HTTP request header Range exists and the byte range is less than X.

  2. Check if the requested response payload is less than, for eg., 100MB.

But I don't know how any of these can be implemented within NGINX.

8
  • 1
    What sort of file is it? A zip file or something else? A 60GB download doesn't sound like a great solution and it's possibly better off being rearchitected. Is this just downloaded using a web browser or some custom application? You could also look at storing that file on S3/Cloudfront and then you won't have to worry about people using up all of your bandwidth or choking your server May 26, 2020 at 23:12
  • 3
    It is a BAM file. I have no choice, the user needs to access it (with byte range), that I cannot change, that's simply how you navigate in genomic files. I just want to avoid the user to be able to download the whole file, while still allowing the user to access parts of the file.
    – PedroD
    May 27, 2020 at 9:31
  • 1
    CGI is a bit old-school but it seems like a simple CGI script in your language of choice would work well here. If you must do it in the nginx config I thing you are going to need to add an additional module since by default you can't do math in the nginx config.
    – 9072997
    May 30, 2020 at 15:38
  • @PedroD, What are the headers if the user attempts to grab the entire 60GB file? Might have a solution for you.
    – James F
    Jun 1, 2020 at 15:44
  • 1
    "I just want to avoid the user to be able to download the whole file" For which reasons? Performance? Or confidentiality? In the last case nothing will prohibit user to do multiple queries with different range and reconstruct the file locally... Jun 1, 2020 at 22:29

2 Answers 2

3
+50

I think, that You can configure nginx to check specific location (60gb file location) for specific (Range) header existence. Then You could block requests that don't contain Range header.

You could check these links:

https://stackoverflow.com/questions/18970620/nginx-reject-request-if-header-is-not-present-or-wrong

https://stackoverflow.com/questions/35342049/nginx-reject-request-if-header-not-present/35366796

Edit

It seems a bit problematic coding complex conditions in Nginx configuration. Maybe You should consider serving large file using some server-side scripting, ie. PHP. There are plenty of examples of streaming scripts.

https://gist.github.com/ranacseruet/9826293

After configuring Nginx to using such a script, You need to add logic to script (Range conditions) and also prevent direct access to large file.

1

What you're looking for is covered in the Slice module here - https://www.nginx.com/resources/wiki/modules/slice/

1
  • My problem is not achieving this slicing feature, HTTP already provides that with the "Range" header. My issue here is that I want to force the client to always use that header when requesting a specific file. Or maybe, alternatively, get NGINX to kill any connection/session that takes more than X seconds to terminate (in order to avoid the download of big files, which would take much more than X seconds).
    – PedroD
    May 31, 2020 at 12:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .