A system is too loaded when it becomes unresponsive.
System load average can be deceiving because you can have a system with a high load that is still quite responsive to requests...
If your system is slamming virtual memory (thrashing the hard disk), is low on actual memory, or times out...pauses when you're trying to enter information at the keyboard, network connection time out, etc...then you need to look at finding what is bottlenecking and work on fixing that. If the response time is acceptable and the system behavior is acceptable without errors then you should be just fine.
I'd keep an eye on your memory use and disk use and possibly network throughput and otherwise listen to your users for feedback.