top shows an average CPU usage during peak times of about 20% while CloudWatch monitoring shows an average CPU usage of 40%. What causes this discrepancy?
A very good observation and we have run into this as well. Here's what I found -- http://www.metamul.com/blog/measuring-cpu-usage-from-within-ec2-it-doesnt-mean-what-you.html
When you run top within the EC2 instance, it is measuring the CPU usage of the physical core machine that is running your instance and others. This usage is incorrect if you want to be measuring cpu usage of your instance alone (the EC2 compute unit assigned to your instance).
Which is why cloudwatch metrics is real since it is measured external to the instance for the EC2 compute unit(s) assigned to your instance alone.