Does your server slow for all users or only the users connecting from the internet? Are these users mobile users from across the internet by chance?
Yes, it is entirely possible that your internet users can be slowing down your entire host. Let's say your link speed to your server from the connected network is 30ms, but your connection from your external users is 300ms (on average). It will take ten times the length of time for handshakes to the server to take place, to send data, to receive data, etc... This interrupt service for the stack carries a price tag with it as it is hghest priority for the system, causing applications to wait while this high priority network activity is addressed. Keep the stack busy longer and you will keep the CPU busy longer servicing the stack and as a result have less CPU for all of your other users. The degrading effect can be ewspecially pronounced if you have a lot of local users accessing your site.
There are some things you can do:
Introduce compression to your servers. Smaller files take less time to transfer
Embrace effective client cache management. Take an active role in reducing the number of files which need to be transferred after the initial handshake. Even if you make your cache age a week you should experience a significant drop off in request levels for static content. Reduced requests=reduced overhead to your server.
Optimize your graphics. Do you need a 15 million color pallete in a graphic which only has 256 colors. Optimized palletes result in smaller file sizes
Consider a different graphic format, one which by default has smaller graphic sizes even before compression, such as PNG
Content Distribution? You have a couple of models you can go to, including one which would use a front end caching appliance to serve the majority of load. Take a look at Squid on the open source front (which you could deploy at a hosting provider as a front end) or any of the AKAMAI like services. Heck, even Google is getting into the content distribution game
In short, big_site+small_pipe is a recipe for poor performance for all when you have to spend so much time servicing network interrupts for the slow link users.