Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have thwo servers. The old one is 24GB RAM, i7-950 @ 3,0Ghz, 2x 1,5TB SATA HDD (ST31500341AS). The new is 32GB RAM, i7-3770 @ 3,4GHz, 2x 3,0TB SATA HDD (ST3000DM001-9YN166).

Now I noiticed, creating a new virtual machine on the new host takes about twice as long as it does on the old host, even though the new CPU should be 66% faster than the old one and it has more memory. Also the harddrives were faster during initial raid sync, but I have no numbers on that.

So I suspected the harddrives to be the limiting factor and I ran bonnie++ on both systems. Both systems run ubuntu 12.04 and the tests took place on a lvm volume formatted with xfs, lazy-counters enabled.

Results from the old server: (Which at the same time was busy managing 10 productive xen VMs.)

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   8     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
              1432M   499  97 57940  11 36990  10   657  98 64289   9 195.1   6
Latency             31917us   35567us     340ms   40027us     162ms     445ms
Version  1.96       ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256  6232  26 12543  19  5888  23  4247  17   317   0   368   1
Latency              1199ms   95254us    1804ms    2313ms    1183ms   12470ms

Results from the new server: (Which at the same time was not busy managing 1 idle backup VM.)

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   8     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                11G   447  96 93517  14 59222  18   644  97 144128  21 310.6  11
Latency             58753us     299ms     575ms   45753us     292ms     238ms
Version  1.96       ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256  4838  16 321243 100  4927  12  4601  16 309249  99   405   1
Latency              3598ms     466us    8146ms    2342ms      58us   13765ms

I think this clearly shows, that the read/write performance on the new server is indeed much better than on the old server. (Might be because of faster hard drives or by the load on the old server). However file creation (and deletion) is much slower.

What can I do to improve or analyze this?

This is the lspci output of the old server:

00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV620 LE [Radeon HD 3450]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV620 HDMI Audio [Radeon HD 3400 Series]
07:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
ff:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
ff:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
ff:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
ff:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
ff:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
ff:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
ff:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
ff:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
ff:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
ff:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
ff:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05   )
ff:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
ff:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
ff:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
ff:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05   )
ff:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
ff:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
ff:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
ff:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05   )

lspci -vvv sata controller section on the old server:

00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller (prog-if 01 [AHCI 1.0])
        Subsystem: Micro-Star International Co., Ltd. Device 7522
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin B routed to IRQ 324
        Region 0: I/O ports at a000 [size=8]
        Region 1: I/O ports at 9c00 [size=4]
        Region 2: I/O ports at 9880 [size=8]
        Region 3: I/O ports at 9800 [size=4]
        Region 4: I/O ports at 9480 [size=32]
        Region 5: Memory at fbafa000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/16 Maskable- 64bit-
                Address: fee4000c  Data: 4121
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
        Capabilities: [b0] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP-
        Kernel driver in use: ahci

And this is the lspci output of the new server:

00:00.0 Host bridge: Intel Corporation Ivy Bridge DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Ivy Bridge PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Ivy Bridge Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation Panther Point USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation Panther Point MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation Panther Point USB Enhanced Host Controller #2 (rev 04)
00:1c.0 PCI bridge: Intel Corporation Panther Point PCI Express Root Port 1 (rev c4)
00:1c.4 PCI bridge: Intel Corporation Panther Point PCI Express Root Port 5 (rev c4)
00:1d.0 USB controller: Intel Corporation Panther Point USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Panther Point LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation Panther Point 6 port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation Panther Point SMBus Controller (rev 04)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 09)

lspci -vvv sata controller section on the new server:

00:1f.2 SATA controller: Intel Corporation Panther Point 6 port SATA Controller [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
        Subsystem: ASUSTeK Computer Inc. Device 84ca
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin B routed to IRQ 323
        Region 0: I/O ports at f0b0 [size=8]
        Region 1: I/O ports at f0a0 [size=4]
        Region 2: I/O ports at f090 [size=8]
        Region 3: I/O ports at f080 [size=4]
        Region 4: I/O ports at f060 [size=32]
        Region 5: Memory at f7c12000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00338  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
        Capabilities: [b0] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP-
        Kernel driver in use: ahci

The xfs_info result is binary identical for both test partitions:

# xfs_info /mnt
meta-data=/dev/xvda3             isize=256    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Now I repeated the test within identical VMs on both machines. Results of the VM on the old server:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm.sh-solutions. 2G   454  97 63287   9 17729   4   644  98 64923   8 120.5   2
Latency             99698us   55973us     602ms   50737us     242ms    1851ms
Version  1.96       ------Sequential Create------ --------Random Create--------
vm.sh-solutions.de  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256  7996  36 248222  98 16188  51  9244  37 263652  99   588   2
Latency               666ms     200us     491ms     636ms      77us    9010ms

Results of the VM on the new server:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm.sh-solutions. 2G   760  96 152604  19 71291  16  1084  97 214864  23 453.0   7
Latency             23365us   42943us     442ms   55716us   42838us     124ms
Version  1.96       ------Sequential Create------ --------Random Create--------
vm.sh-solutions.de  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256  4215  11 403505  97  7103  13  3637   9 405289  99   481   1
Latency              2746ms     115us    1819ms    4146ms      22us   17072ms

Once again, the new server has much higher throughput, but is much slower for create (and delete).

Old server parted /dev/sda print:

Model: ATA ST31500341AS (scsi)
Disk /dev/sda: 1500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 4      16.5MB  38.9GB  38.9GB  primary               lvm
 1      38.9GB  49.2GB  10.2GB  primary  xfs          raid
 2      49.2GB  59.4GB  10.2GB  primary  xfs          boot, raid
 3      59.4GB  1500GB  1441GB  primary               raid

New server parted /dev/sda print:

Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name   Flags
 1      1049kB  3146kB  2097kB               grub   bios_grub
 2      3146kB  68.7GB  68.7GB               swap   raid
 3      68.7GB  79.5GB  10.7GB  xfs          root1  raid
 4      79.5GB  90.2GB  10.7GB  xfs          root2  raid
 5      90.2GB  3001GB  2910GB               raid   raid

Results on new server in the same VM as above, after creating the fs with ´mkfs.xfs -d su=4k,sw=1 -l su=4k`. (no improvement.)

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm.sh-solutions. 2G   764  95 160175  21 71847  17   669  97 215201  23 444.1   8
Latency             20880us   46904us     327ms   31263us   50802us     137ms
Version  1.96       ------Sequential Create------ --------Random Create--------
vm.sh-solutions.de  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256  4180  13 257543 100  7547  17  4372  12 377933  99   410   1
Latency              1739ms     160us    1774ms    2073ms      79us    9058ms

UPDATE: By now I moved all virtual machines to the new server, allowing me to reinstall the old server with the very same version of everything as the new server. I repeated the in vm test on that machine:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm.sh-solutions. 2G   701  97 110157  14 27335   6  1034  98 163694  17 341.1   5
Latency             22008us     118ms    1481ms   30413us   26635us     742ms
Version  1.96       ------Sequential Create------ --------Random Create--------
vm.sh-solutions.de  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256  9775  34 389576 100 19943  50 10919  37 374190  99   599   2
Latency               900ms     164us     356ms     533ms      26us    8015ms

Note: During this thest on the old host, no other VMs were active on that host. It reaeched 10000 create operations per second. The same (earlier) test on the new host (also with no other active VMs) resulted in only 4215 creates on the new server. Even when it old server way "busy" it reached up to 6200 creates... This makes this worse than expected...

UPDATE: From now on I have no access to the old server any more. I am trying to get my hands on a similar machine though.

cat /proc/mounts on the new server had (beside others):

/dev/xvda3 /mnt2 xfs rw,relatime,attr2,delaylog,sunit=8,swidth=8,noquota 0 0
share|improve this question
    
Can you please share mount and xfs_info on both filesystems? –  Fox Apr 20 '13 at 11:51
    
Are the hard drives for the vm's the type same on both servers? Fixed size vs expanding. –  Simon Bennett Apr 20 '13 at 13:52
    
@SimonBennett: The tests were run in dom0 so xen was not involved. All hard drives I use are LVM logical volumes with fixed size, no matter if dom0 (host) or domU (VM/guest). –  Steffen Heil Apr 20 '13 at 17:34
    
@Fox: I added the results of xfs_info. However, there is no difference in that between the two servers. –  Steffen Heil Apr 20 '13 at 17:46
    
This really seems that latency gets added somewhere on the way. Are the kernels/xens the same version? –  Fox Apr 21 '13 at 18:33

4 Answers 4

I noticed a difference between the two configurations which could in principle explain the reversed performance rankings for latency and throuput:

Sector size (logical/physical): 512B/512B
Sector size (logical/physical): 512B/4096B

I don't know whether parted /dev/sda print would tell you about misaligned partitions, though. So you should have a look at

  1. the sector addresses
  2. the mdraid configuration (with respect to this)
  3. the LVM (PV) configuration

pvcreate has --dataalignment and --dataalignmentoffset for handling this.

share|improve this answer
    
Is there a way to test if the pv was aligned correctly? I cannot completely reinstall the server just to test that but I could reinstall it if I am sure, that is the reason. –  Steffen Heil May 1 '13 at 19:22
    
@SteffenHeil This shows you the offset of the first PE: pvs -o pe_start,pv_name --units s In addition you can have a look at the offsets of the LVs: dmsetup table name with name what you see in /dev/mapper. The number at the end is the offset. I think the real question is: Are the partitions aligned. Before reinstalling you can make a test: Create a (smaller) DM device (dmsetup create) within an empty LV, with an offset of 1-7 sectors (so that it should be correctly physically aligned). Run mkfs on it and after that your benchmark. That should show whether this is the problem. –  Hauke Laging May 1 '13 at 22:19
    
I am quite sure, parted aligned the partitions correctly. (Yet I have no proof and don't know how to check.) For what I know, I should be able to create and PV insinde a LV, creating another VG. While this is not a good idea for a productive system, it might help testing performance as during it's operation there is very little overhead by this and accessing data should only access the blocks that are really neccesary, right? –  Steffen Heil May 2 '13 at 7:10
    
pvs -o pe_start,pv_name --units s resulted in 384S /dev/md0. As 384 is a multiple of 8, that seems okay... –  Steffen Heil May 2 '13 at 7:14
    
@SteffenHeil You can check the partitions' start sectors by fdisk -l -u S /dev/sda. Have you made the experiment with the offset DM devices? –  Hauke Laging May 2 '13 at 12:29

Just throwing this out there. The data input/output tests show the per-second data rate to be superior on the new system in all tests. It is only the file management side of things that seems slow. Could it be an issue with XFS?

It seems there is a contention issue with some versions of XFS, which could match the issue, since the reads do seem marginally superior, it's just the writes/deletes that are suffering:

http://oss.sgi.com/archives/xfs/2012-02/msg00219.html

I hope that helps

share|improve this answer
    
I could now reinstall the old server to the exact version running on the new server and it is still way faster. (Even more than before, I will post bonnie++ results shortly.) This excludes XFS versions as a cause. –  Steffen Heil May 1 '13 at 18:55

could you post how much slower it is ? I mean the comparison ? and also the iostat during the progress?

share|improve this answer
    
I already included the results of bonnie++ for both. –  Steffen Heil May 1 '13 at 18:54

What are the SATA chipsets of those boards? The chipset can make all the difference here. That, or you could try looking for newer versions of drivers for the chipset of the newer server - either from the vendor or by compiling a newer kernel. Look for some doco on the chipset in relation to Linux - there could be some sysctl or hdparm parameters you could tweak.

Hope this helps

share|improve this answer
    
I don't know much about hardware and specialized sata drivers. I added the output of lspci. Does it show anything that can help? –  Steffen Heil Apr 20 '13 at 17:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.