AFAIK there is nothing special to it. In your public DNS just create "your-name.your-domain.com" as a CNAME to "your-bucket-name.s3.amazonaws.com" and have the folder name match your domain name.
CNAME record for my Amazon S3 domain, so have nicer URLs
I assume you have your website hosted elsewhere. Inside the HTML there are links to images, videos etc hosted on S3. You'd like these links to look nicer.
Is this a reasonable thing to do?
Well, for some. I don't think too many end users will look into your HTML or notice what their browser is loading -- and if they do, I don't think they care.
Are there any drawbacks?
One more single point of failure (your DNS config), but it's probably insignificant as you already have this same issue for your main website. A (quite) small slowdown for your users, as they have to make one additional DNS lookup now. First one for your CNAME (fx static.yourdomain.com) and then one for S3. That's maybe ~100 ms on average, not much.
A quick note: It seems you're already using S3. If it works well, then that's fine. If your audience is somewhat global, and you have fair traffic, then you could use a Content Delivery Network instead. Amazon's own Cloudfront of course comes to mind, but it's not the fastest CDN around. To get a feel for the speed of various CDNs you can use Cloud Harmony's test tools from a couple of locations. For small files (CSS, JS, images) pay special attention to latency. For video throughput is more important, but that's rarely a problem for a CDN.