Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am having trouble setting up flashcache on a system with LVM and mdadm, I suspect I am either just missing an obvious step or getting some mapping wrong and hoped someone could point me in the right direction?

system info:

CentOS 6.4 64 bit

mdadm config

md0 : active raid1 sdd3[2] sde3[3] sdf3[4] sdg3[5] sdh3[1] sda3[0]
      204736 blocks super 1.0 [6/6] [UUUUUU]


md2 : active raid6 sdd5[2] sde5[3] sdf5[4] sdg5[5] sdh5[1] sda5[0]
      3794905088 blocks super 1.1 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]


md3 : active raid0 sdc1[1] sdb1[0]
      250065920 blocks super 1.1 512k chunks


md1 : active raid10 sdh1[1] sda1[0] sdd1[2] sdf1[4] sdg1[5] sde1[3]
      76749312 blocks super 1.1 512K chunks 2 near-copies [6/6] [UUUUUU]

pcsvan

PV /dev/mapper/ssdcache   VG Xenvol   lvm2 [3.53 TiB / 3.53 TiB free]
Total: 1 [3.53 TiB] / in use: 1 [3.53 TiB] / in no VG: 0 [0   ]

flashcache create command used:

flashcache_create -p back ssdcache /dev/md3 /dev/md2

pvdisplay

--- Physical volume ---
PV Name               /dev/mapper/ssdcache
VG Name               Xenvol
PV Size               3.53 TiB / not usable 106.00 MiB
Allocatable           yes
PE Size               128.00 MiB
Total PE              28952
Free PE               28912
Allocated PE          40
PV UUID               w0ENVR-EjvO-gAZ8-TQA1-5wYu-ISOk-pJv7LV

vgdisplay

--- Volume group ---
VG Name               Xenvol
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  2
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                1
Act PV                1
VG Size               3.53 TiB
PE Size               128.00 MiB
Total PE              28952
Alloc PE / Size       40 / 5.00 GiB
Free  PE / Size       28912 / 3.53 TiB
VG UUID               7vfKWh-ENPb-P8dV-jVlb-kP0o-1dDd-N8zzYj

So that is where I am at, I thought that was the job done however when creating a logical volume called test and mounting it is /mnt/test the sequential write is pathetic, 60 ish MB/s

/dev/md3 has 2 x SSD's in Raid0 which alone is performing at around 800 MB/s sequential write and I am trying to cache /dev/md2 which is 6 x 1TB drives in raid6

I have read a number of pages through the day and some of them here, it is obvious from the results that the cache is not functioning but I am unsure why.

I have added the filter line in the lvm.conf

filter = [ "r|/dev/sdb|", "r|/dev/sdc|", "r|/dev/md3|" ]

It is probably something silly but the cache is clearly performing no writes so I suspect I am not mapping it or have not mounted the cache correctly.

dmsetup status

ssdcache: 0 7589810176 flashcache stats:
    reads(142), writes(0)
    read hits(133), read hit percent(93)
    write hits(0) write hit percent(0)
    dirty write hits(0) dirty write hit percent(0)
    replacement(0), write replacement(0)
    write invalidates(0), read invalidates(0)
    pending enqueues(0), pending inval(0)
    metadata dirties(0), metadata cleans(0)
    metadata batch(0) metadata ssd writes(0)
    cleanings(0) fallow cleanings(0)
    no room(0) front merge(0) back merge(0)
    force_clean_block(0)
    disk reads(9), disk writes(0) ssd reads(133) ssd writes(9)
    uncached reads(0), uncached writes(0), uncached IO requeue(0)
    disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
    uncached sequential reads(0), uncached sequential writes(0)
    pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
    lru hot blocks(31136000), lru warm blocks(31136000)
    lru promotions(0), lru demotions(0)
Xenvol-test: 0 10485760 linear

I have included as much info as I can think of, look forward to any replies.

share|improve this question
add comment

1 Answer

I can see that /dev/md2 is not disabled by lvm.conf but It should.

I think in such complex setup you'd better to explicitly add LVM devices and disable all others:

filter = [ "...", "a|/dev/md0|", "a|/dev/md1|", "a|/dev/mapper/ssdcache|", "r|.*|" ]

Additionally iostat can be used to monitor actual devices activity.

PS:

I am very pessimistic about your crazy storage layout where drives are split to many partitions which participate in such many different RAIDs.

System {RAID1 (/dev/ssd1p1+/dev/ssd2p1)}
AND
Data {RAID10 (6 whole drives) + flashcache on RAID1 (/dev/ssd1p2+/dev/ssd2p2)}

-- much more attractive :).

UPD:
Or even better:

RAID1 on whole SSDs: System and partition for flashcache
AND
RAID10/6 on whole HDDs + flashcache

share|improve this answer
    
Why would md2 be disabled when it is the primary storage volume i.e. mapped to /dev/Xenvol ? The mdadm layout is as it needs to be, raid 10 on the primary storage is not an option due to lack of space however the mdadm layout is really inconsequential to this. –  Backtogeek Nov 11 '13 at 13:13
    
After you create flashcache device ALL data access should be done using this device. No one underlying device should be touched! Otherwise you can easily loss your data. As I can see /dev/md2 is participating in the flashcache setup. Your volume group should be based on /dev/mapper/ssdcache, not /dev/md2. –  Veniamin Nov 11 '13 at 13:58
    
The main problem with your storage configuration - after any disk failure you will get a great headache with preparing and inserting new one. A little mistake - and your data may be lost! In lack of space, RAID6 may be used. It is not so smart on write operations but that can be accelerated with the write-back ssd cache. The only notice: RAID0 is not a good choice for write-back cache :). –  Veniamin Nov 11 '13 at 14:18
    
Ok I can change the write back cache to raid 1 that is fine, but none of this actually answers the question which is specific to the flashcache issue. –  Backtogeek Nov 11 '13 at 15:20
    
Raid1 for cache - just notice. I see that your main pool md2 is not allocated yet. If so, you may destroy LVM and try to write directly with dd to /dev/mapper/ssdcache. Thus, check flashcache counters. –  Veniamin Nov 11 '13 at 20:14
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.