When a default install LAMP server has a problem with memory consumption, it's usually Apache.
Since mod_php is not thread safe, Apache is generally running with MPM prefork. That means that for each request server is spawning separate process. Since apache process with mod_php is usually consuming around ~35-50MB, that means that as the number of users grows so does the memory consumption.
Another problem is that apache doesn't do garbage cleanup so, process size tends to grow over time, so it's not unusual to see apache processes of size greater than 100MB.
First thing you need to do is try to tune Apache/PHP to lower your memory footprint.
Ideally, you would want to turn off all the unnecessary modules, both PHP and apache. This is some minimum set of apache modules that you will probably need:
# apachectl -t -D DUMP_MODULES
If your users keep the connection open after the data is fetch, that means higher RAM usage for you. To circumvent that, change your Keepalive settings. You can either turn it off:
or set really low timeout:
Next thing is to tune prefork. Take special care in determining your MaxClient setting - this depends solely on amount of RAM you have dedicated to your Apache. Also, lower MaxRequestsPerChild to force Apache to recycle processes more often, thus avoid growth of memory usage per process:
You can find scripts online that calculate memory usage of Apache processes and can calculate what is the setting you can safely use for MaxClients/ServerLimit.
Moving to MySQL - it's usage isn't so depended on number of users, but still you want to tune it. The simplest way is to use mysqltuner script, but avoid listening to all advices, like increasing Query Cache. Query Cache of around 32 to 64MB is more then enough.
Hope this helps.