This is S3's SSL certificate:
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=*.s3.amazonaws.com
The Common Name on the certificate is *.s3.amazonaws.com, which is not going to match app.mysite.com. Therefore you cannot use your own hostname (via a CNAME record) with S3 when using HTTPS (at least if you want it to validate), since S3's SSL certificate doesn't (and can't) include your hostname. You must use the <bucket>.s3.amazonaws.com hostname instead.
Now, technically a wildcard in SSL certificate hostname only matches a single part of the hostname, so the above certificate will match foo.s3.amazonaws.com, but not foo.bar.s3.amazonaws.com. This is why the documentation says that the bucket name cannot contain periods, since app.mysite.com.s3.amazonaws.com will not match the certificate, but app-mysite.s3.amazonaws.com will.
I assume that the custom certificate validation they are referring to is if you are writing your own client which accesses S3, which is not the case here.
Edit: As @ceejayoz points out, you can still use path-style access for buckets with periods in the name, however, you should use the correct region specific hostname since AWS can't select the correct endpoint when the DNS request is made.
https://s3-us-west-1.amazonaws.com/app.mysite.com/