This really all depends. What kind of traffic are you serving up? High graphic intensity? Plain text?
If you want to look into your issues in more detail, check out YSlow from Yahoo! - it's a FireFox Plugin (actually its a plugin to FireBug, which is a FF plugin), and its very good at telling you where you're going wrong. Ignore their scoring about a CDN though, it's irrelevant.
We run several websites that are very graphically intense and they all load, from the initial request to the final rendered page, < 200ms if they've been here before. First ever request = ~ 900ms (to download all the external content). Caching is your friend!
Edit: -- I am SO sorry, I read your question as "Google Analytics", not "Webmaster Tools". In this case, these numbers are perfectly fine. Our average is 1500ms for a huge site and 500ms for a really basic site. And our sites are heavily optimised!