Yes, provided you have set a sensible expiry time then browsers don't have to come back to your server each time to ftch the content. There are a lot (and with the growing scarcity o IPV4 addresses) an increasing number of caching proxies out there - the world wide web is a CDN (but obviously not for HTTPS).
That your server does not have to deliver content each time reduces the amount of work it has to do, hence making more resources available for serving up the non-cached content.
It will have a massive impact at the client. You should be able to see this yourself using tools such as firebug.
Note that it can actually have an adverse impact on performance if the cache goes stale and you start getting conditional requests / 304 responses (use mod_headers to trip if-none-match and if-modified-since for content items which are not huge).