Hot answers tagged static-content
11
The app engine is a cloud computing platform and is not designed to be a CDN. While your data may be stored on multiple nodes those nodes are not edge-cache nodes so they will not offer the same benefits that a CDN would. You can compare GAE vs various CDNs using the CloudHarmony.com speed test. Here were the results when I tested today:
Order Service ...
11
Here is a useful article which also addresses how to use a cookieless sub-domain: http://www.ravelrumba.com/blog/static-cookieless-domain/.
In order to set up a cookieless subdomain you have to make sure that your server or application only sets cookies for www.example.com and not the top-level example.com. (Cookies set at the top-level domain apply to ...
8
The cookies you mention are set by Google Analytics, they are usually set on a domain-wide basis to cover all subdomains.
Nginx cannot make a browser not send any cookies, there is not part of the HTTP specification that allows a web server to say it's not interested in cookies so a browser will always send them. Many of the biggest sites where this ...
6
it really depends on the file system you're using to store the files.
some filesystems (like ext2 and to a lesser extent ext3) are hideously slow when you have thousands of files in one directory, so using subdirectories is a very good idea.
other filesystems, like XFS or reiserfs(*), don't slow down with thousands of files in one directory, so it doesn't ...
6
Did you turn on the static content feature?
http://weblogs.asp.net/anasghanem/archive/2008/05/23/don-t-forget-to-check-quot-static-content-service-quot-in-iis7-installation.aspx
5
I would strongly recommend the use of a proxy such as varnish or squid rather than downloading the files and keeping them yourself, so that the proxy takes care of all of the cache expiry and other entertainments that make caching so much fun.
If you're trying to cache content that is dynamically generated and doesn't have proper expiry information in the ...
5
lighttpd and ngingx
Correct, these are HTTP daemons that do the same job as Apache, but are generally geared to different kinds of content serving. I don't know much about them (I'm an Apache/IIS man myself), so I don't know why you would choose one over the other. I'm sure there's lots of questions on serverfault about why you would though.
For more ...
5
I suggest you start by setting up a local test server (virtual machines are great for this) and use that to learn what you need to know before putting it on the Internet.
From your question I get the impression that you've never done anything like this before and you therefore will make mistakes that will either render your system inoperable, inaccessible ...
5
Well, one possibility is Amazon's static website hosting over S3 together with a CNAME mapping.
The S3 latency isn't super low either. If your site is popular (so that caches would be likely to not evict your content) then you could experiment with adding Amazon Cloudfront CDN (i.e. edge proxy caching).
5
You need to use the ProxyPass ! argument with a path, not in a <Location> block, for example:
ProxyPass /static !
ProxyPass / http://localhost:8081/
ProxyPassReverse / http://localhost:8081/
I believe these rules are processed in the order they appear in the config, so be sure to specify exclude rules first.
4
DDOS attacks rely in some large part on sheer volume of bandwidth. If your underlying infrastructure is being swamped by massive numbers of false connections then it doesn't really matter what sorts of limits are being placed on connections to your servers. Limiting by IP address doesn't address the problem of many hundreds of thousands of sources (i.e. ...
4
Add fingerprints/hashes your CSS and JS URLs and set far-future Expires/Cache-Control headers.
This is also the technique that Rails Asset Pipeline uses.
4
A couple different things jump to mind first and foremost:
First, you already have a static mirror of your site that is designed for just this use-case: Cloudflare. As well as providing your DNS, I assume you have them set up as a CDN to dull the brunt of traffic that is coming towards you. Cloudflare has a feature called Always Online which is intended ...
4
Millions of concurrent connections? Unless you are hosting video streams or other large files, http requests usually finish in such short time that even with busier sites you don't get that many concurrent connections. If you do, you seriously don't have only one server. Also with that kind of traffic your network traffic would be on such high level that I ...
4
If you're using nginx, then you're talking just a few KB of overhead per active connection. If you're using something like Apache, you'll have one thread per connection, which means hundreds of KB or even megabytes per connection.
However, nginx does not support asynchronous disk IO on Linux (because async disk IO on Linux is basically horribly broken by ...
4
Benchmark, benchmark, benchmark! You'll probably find no significant difference between the two options, meaning that your time is better spent on other problems. If you do benchmark and find no real difference, go with whichever scheme is easier -- what's easy to code if only programs have to access the files, or what's easy for humans to work with if ...
4
As others have said, directory hashing is very probably going to be most optimal.
What I would suggest you do though is make your URIs independent of whatever directory scheme you use, using nginx's rewrite module, e.g. map example.com/123456.jpg to /path/12/34/123456.jpg
Then if your directory structure needs to change for performance reasons you can ...
3
You could put a squid cache in front on your nginx server. Squid can either keep the popular images in memory, or use it's own file layout for fast look ups.
For Squid, the default is 16 level one directories and 256 level two. These are reasonable defaults for my file systems.
If you don't use a product like Squid, and create your own file structure, ...
3
Doing some basic directory hashing is generally a good idea. Even if your file system deals well with 70k files; having say millions of files in a directory would become unmanageable. Also - how does your backup software like many files in one directory, etc etc.
That being said: To get replication (redundancy) and easier scalability consider storing the ...
3
wget usually works.
GNU page: https://www.gnu.org/software/wget
Wiki: http://wget.addictivecode.org
Example:
wget.exe -x -r http://www.acmecorp.com
Would create a directory called www.acmecorp.com under your current directory, and recursively download all of the content.
There are a myriad of options.
wget --help
GNU Wget 1.11.1, a ...
2
More information is required to be sure (lighty response headers), but I would assume your backend is not setting an Expires nor Cache-control header.
This is required so Varnish knows how long to store the data in cache for, or you can also specify a default ttl in the DAEMON_OPTS via -t default_ttl_in_seconds.
Or indeed within the VCL itself, as answered ...
2
You have to "disable" the proxy on static files (i.e. enable it only on other files):
$HTTP["host"] =~ "www.my-domain.com" {
server.document-root = "/var/www/my-domain"
$HTTP["url"] !~ "^/static" {
proxy.server = ("" => ( "tomcat" => (
"host" => "127.0.0.1",
"port" => 8080,
"fix-redirects" ...
2
Instead of buying the NodeBalancer fits-all ready-to-use feature from Linode, you might as well just buy another regular Linode, and implement load balancing and caching by yourself.
You can use nginx, and have it act as a proxy and balancer to your real web-site.
Depending on whether you require your website to change every couple of hours/days or not, ...
2
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, ...
2
This link states that G-WAN can scale better than nginx and it can benefit more from multi-core CPUs and uses less resources. You can have a look at the architecture and performance sections.
In my opinion, I recommend you to install anyone of them and do your own benchmarks on your estimated/expected loads.
2
This comparative test (G-WAN vs Nginx) was done by an Academic expert of distributde architectures.
This is the very same test used by Phoronix, the official multi-OS benchmark.
2
I can think of one solution to your question without the need to hack nginx.
Shortly, it can be handled on the file-system level.
In more details, you can configure an incron script to be executed on the IN_ACCESS event. This means that your script will be executed whenever a specific file is accessed (read). In this script, you can do whatever you want ...
2
i'm not sure if i understand the problem but you can:
if you want to serve static files not from /var/www/static use:
Alias /static/ /somewhere/somewhere/dir/
if your static directory is already in /var/www/static and you get forbidden error - make sure that those files are readable for user under which apache runs:
chown www-data:www-data -R ...
2
I'm not exactly sure what you're asking, but you could always put entries in your hosts file if it's a question of resolving static.domain.tld, domain.tld, etc.
%systemroot%\system32\drivers\etc\hosts:
static.domain.tld 127.0.0.1
someotherstatic.domain.tld 127.0.0.1
2
You didn't really say what is failing or provide error messages - but I am going to go out on a limb and take a guess.
My guess is that when you are doing the <clear /> within modules you are getting a "Lock Violation" error. This is because at the higher level of the IIS7 settings hierarchy, the modules section (and individual modules themselves) ...
Only top voted, non community-wiki answers of a minimum length are eligible
