This is S3's SSL certificate:
The Common Name on the certificate is
*.s3.amazonaws.com, which is not going to match
app.mysite.com. Therefore you cannot use your own hostname (via a CNAME record) with S3 when using HTTPS (at least if you want it to validate), since S3's SSL certificate doesn't (and can't) include your hostname. You must use the
<bucket>.s3.amazonaws.com hostname instead.
Now, technically a wildcard in SSL certificate hostname only matches a single part of the hostname, so the above certificate will match
foo.s3.amazonaws.com, but not
foo.bar.s3.amazonaws.com. This is why the documentation says that the bucket name cannot contain periods, since
app.mysite.com.s3.amazonaws.com will not match the certificate, but
I assume that the custom certificate validation they are referring to is if you are writing your own client which accesses S3, which is not the case here.
Edit: As @ceejayoz points out, you can still use path-style access for buckets with periods in the name, however, you should use the correct region specific hostname since AWS can't select the correct endpoint when the DNS request is made.