Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have two Ubuntu servers with very similar, if not identical, configuration: both have 16 cores (4 E5540 Xeons), 16GB RAM, HP Smart Array P212 controllers with 2 HDDs in RAID1. I/O scheduler used is deadline. Both run the same versions of MySQL (5.1.26, later upgraded to 5.1.47/Percona Server 10.2). The databases are similar, but not exactly the same. MySQL configuration is also very similar. But the performance is incomparable:

Server1:

mysql> alter table documents add column test int(10) unsigned not null default '0';
Query OK, 2496892 rows affected (7 min 18.41 sec)
Records: 2496892  Duplicates: 0  Warnings: 0
Server2:
mysql> alter table documents add column test int(10) unsigned not null default '0';
Query OK, 2497414 rows affected (1 min 59.15 sec)
Records: 2497414  Duplicates: 0  Warnings: 0

Server2 is 3.7 times as fast.

MySQL benchmark (sql-bench) only confirms that Server2 is much faster:

Server1:

alter-table: Total time:  7 wallclock secs 
ATIS: Total time:  3 wallclock secs 
big-tables: Total time:  0 wallclock secs
connect: Total time:  1 wallclock secs 
create: Total time:  7 wallclock secs 
insert: Total time:  3 wallclock secs
select: Total time:  2 wallclock secs 
wisconsin: Total time:  4 wallclock secs 
Server2:
alter-table: Total time:  1 wallclock secs 
ATIS: Total time:  1 wallclock secs 
big-tables: Total time:  0 wallclock secs 
connect: Total time:  0 wallclock secs 
create: Total time:  0 wallclock secs 
insert: Total time:  2 wallclock secs 
select: Total time:  2 wallclock secs
wisconsin: Total time:  2 wallclock secs 

The only difference between the servers is the OS version. Server1 is Ubuntu 9.10 with 2.6.31-20-server #58-Ubuntu SMP kernel, while Server2 is Ubuntu 10.05 with 2.6.32-22-server #33-Ubuntu SMP.

I had serious suspicions about I/O on Server1. So, iostat -dkx gives significantly higher %util values on Server1, about 25-30% vs 1% on Server2.

I ran I/O benchmarks (iozone3), but to my surprise, Server1 produced slightly better results. RAM test with mbw was also in favor of Server1.

What should I test next? What could be the problem?

Thank you.

share|improve this question

1 Answer 1

I'm not an Ubuntu guy, but a 10 second search shows 9 uses ext3 and 10 uses ext4 by default. Post the output of 'tune2fs -l' on your data partitions to contrast/compare.

share|improve this answer
1  
Very good suggestion, was about to post it myself. EXT3 is -horrible-, switch to XFS ASAP. –  pauska Jun 22 '10 at 10:02
    
Good idea, indeed, but it is ext4 on both machines, sorry for not mentioning. –  minaev Jun 22 '10 at 10:15
    
tune2fs output is also very similar. Interesting differences are in the features list: Server1: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Server2: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file I'm afraid, that doesn't say much to me... –  minaev Jun 22 '10 at 10:20
    
UPDATE: Sorry, I was wrong, but it's getting even curiouser... Server1 uses ext4, while Server2 has ext3. –  minaev Jun 22 '10 at 10:25
    
there is a release note for 10 LTS that states performance for ext4 will be worse in some cases. Even though server1 is only 1 kernel revision behind that could be a big deal for ext4 performance, not to mention any special patches Ubuntu adds in packaging (if they're anything like Red Hat). I suspect your performance problem is related to the ext4 on Server1. –  troyengel Jun 22 '10 at 10:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.