I have inherited system for which I need to tune performance as much as possible from hardware perspective. First of all I am a web developer, not a SysAdmin, I don't want to make any premature performance tuning, so in the past few days I have done a little bit of research also setted up Server and Application monitoring and wrote small plan how things could be improved, but I need somebody who is experiences SysAdmin to review this.
The reason why we are doing this is because site seams to be slow and traffic is growing steadily. The main plan is to rewrite system, because of many bad architectural decisions that were maid, database itself, is not normalized properly. So meanwhile while rewriting is happening we want to do some "stupid"/simple scaling (if possible), just to keep customers happy.
Our current system
- OS: CentOS 5 64 bit (Linux 2.6.18-238.12.1.el5 )
- Web server: HP DL 320, Intel Xeon 4-core L5520 @ 2.26GHz CPU, 8GB RAM, 2 x 500 GB SATA Drives, RAID Controller, Redundant PSU
- MySQL 5.0
- PHP 5.2.10
- Apache 2.2.3 with php_mod
Server monitoring results
We are running single server, which hosts web application and database.
- Average requests per second ~1.4, at peak ~3 RPS
- Disk I/O utilization, at peak 25% (this is only reached when we are running some background tasks), other than that ~2-5%
- Physical memory, at the peak around 70% are used.
- We are currently are using mixture of InnoDB and MyISAM tables
- Query cache is turned off
- innodb_buffer_pool_size = 8M
- innodb_flush_method = (empty)
- innodb_log_file_size = 5M
- innodb_thread_concurrency = 8
I am not concerned about MyISAM parameters, since I am planning to convert all tables to InnoDB, so that it would be easier to tune for me.
My current plan
1. System monitoring
Keep system monitoring for a weak, so we could have a better picture later if performance tuning had succeeded.
2. Analyze Application Monitoring
Gather information which pages took longest to load
3. Use Varnish
As much as I understand, Varnish, roughly speaking, caches entire page, and acts as a proxy which can later serve those pages without Apache involvement. In our case we have a lot of content that does not change frequently, although consumes quite a lot of resources to generate.
4. Benchmark Using Custom Scripts
Write custom script that mimics user's behavior, and tries to use "greedy" features. Run those scripts simultaneously, when site is not visited, and gather system's performance information. Do this before and after site optimization. Although, this bit makes sense to me, I haven't really heard about doing something like that before, just saw similar idea in Rails benchmarking, I would like to avoid premature benchmarking, so I would be happy to hear some opinions about this.
5. Convert all Tables to InnoDB
This will make all tuning process a lot simpler.
6. Database Layer Tuning
Turn on Query caching and set it to 256MB, than constantly measure cache performance using following equation:
Qcache_hits / Qcache_inserts x 100 = Your Query Cache's effectiveness.
- Create a cron job that runs twice an hour and executes:
FLUSH QUERY CACHE;
- We are using Sphinx, so for its SELECT queries I will also add
query_cache_limitshould prevent Sphinx for trashing cache anyways.
7. Physical Memory Tuning
This is a bit tricky, since we only have around ~3GB extra memory that could be allocated to MySQL. Total size of Innodb TableSpaces, in our case, is around 3GB. Following article: Choosing innodb_buffer_pool_size suggests that
innodb_buffer_pool_size should be set to 10% more than total size of InnoDB TableSpaces, which in our case would be 3.1GB, so we don't have enough memory. However it is not a problem to add more physical memory to our server.
In addition to
innodb_buffer_pool_size tunning, following MySQL parameter changes should be made:
innodb_flush_method = O_DIRECT(avoid double buffering )
innodb_log_file_size = 256M
8. Page Generation Time Tuning
Most of the blogs I have encountered suggest to use Nginx and APC (PHP cache). I have seen benchmarking results, and they all seem to be impressive, when compared to Apache with mod_php enabled. But fact worth noting in our case is that we serve 3RPS at peak time. Changed Apache with Nginx is not an easy process, and frankly I would rather have somebody who is a experienced SysAdmin to do this, if needed. Maybe somebody could give any advice, if it is worth migrating servers in our case?
Can anybody, taking into account my situation, give me any feedback on this plan?