My fairly ignorant opinion is this is some kind of DNS problem although I've made no DNS changes for weeks. Maybe S3 is having a problem but no one seems to be reporting and S3 issues.
Sometime last night files in one of our S3 buckets became only occasionally reachable through the URL we typically use. The file is reliably reachable if I use a native amazon URL. Here's an example:
Not reliably reachable: http://guiders.iguiders.com/Guider-1309959830.js
Reachable https://s3.amazonaws.com/guiders.iguiders.com/Guider-1309959830.js http://guiders.iguiders.com.s3.amazonaws.com/Guider-1309959830.js
guiders.iguiders.com is CNAME'd to guiders.iguiders.com.s3.amazonaws.com however tracerout to the 2 names returns different IP addresses - I don't know if that's significant or not.
guiders.iguiders.com is reachable in a browser but I get the "It Works" ubuntu default www page which suggests it's actually pointing to one of our EC2 instances even though the DNS is set to point to amazon S3.
I don't know how it could be an S3 problem but it seems even less likely that DNS is just suddenly not working for this one URL.
Also, I know it's not a local problem (like it's not that I had the DNS changer virus) becuase I use pingdom to check the ability to reach a file in that bucket and every few minutes it tells me the server is down, then it's up, then it's down.
Any ideas for something I should try to discover what the issue is?