This is a bit wordy, so please bear with me. :)

We have a RAC cluster of 3 Oracle 11gR2 servers and purchased FusionIO ioDrive2 PCIx cards for these servers to use as OracleDB's Flash Cache. These servers run both our production db instance and a smaller development db. We would like to configure both instances to use these devices as their Flash Cache, dev db having a much smaller size.

It is my understanding that only 1 device can be configured as a destination for Flash Cache.

At the same time this ioDrive2 device supports what's called "Virtual Controllers". It basically splits this device into 2 to double the IOPs (total combined bandwidth remains the same).

So to put this whole picture together: I'll need 2 partitions, one for each db instance. If I utilize "Virtual Controllers" I'll have 2 equally-sized drives.

What I'm thinking of doing is to combine 2 virtual controllers into 1 using LVM and then partition that one combined volume for each db.

Storage layout

How much performance and management overhead will I get with LVM? Are there any caveats I should be aware of with this setup? Is there a better approach to this situation?

  • I'm a touch confused here. You say that LVM only allows a single flashcache destination, yet it sounds like you're trying to set up two. What am I missing? – sysadmin1138 Feb 17 '13 at 3:33
  • @sysadmin1138 sorry if I wasn't clear. Oracle itself allows only 1 device as Flash Cache destination. That's why I'm thinking of using LVM to combine 2 devices into 1. – Mxx Feb 17 '13 at 3:50

I'm doing something similar with another product, but what you're looking to do is actually pretty low-overhead. The devices I'm using present four 'drives' per PCIe card, and using MDRAID to convert that into a RAID0 device upon which I layer LVM has worked extremely well for me. You lose some blocks to metadata, but the I/O code-paths in the kernel are really well optimized for this.

The caveat here is that "RAID0" thing. I have had one of those virtual 'drives' fail on me, and that did indeed lose me the whole card (that's OK, this service is designed to tolerate whole-node failures). I don't know Oracle's flash-cache, so I can't tell you if it's tolerant of such failures.

  • Afaik failure of this card won't be fatal. That's why I don't mind RAID0. 'Source of truth' remains our storage backend. These cards are used as a caching tier..If the card fails (I hope) Oracle will simply take it out of the operations loop and continue to function the old way. If it does take down the whole server, we have 2 other servers still running to take over the load until we I fix the down one. – Mxx Feb 17 '13 at 17:48

I don't think anyone here could tell you how much performance overhead you'll experience. It really is a test and observe situation. I'll assume that all-else-equal, there would be a performance gain.

Do you have any specifications on the host system? When I was evaluating FusionIO and the need to (software) RAID 1+0 across multiple FusionIO Duo cards, I paid attention to the CPU, RAM and system board capabilities of the host server(s). There's going to be an impact CPU-wise if you're pushing the I/O that you seem to planning for.

Another question... What problem will this solve? Are you really just trying to add another tier of storage to the DB? For the RAC cluster, what is the working set of data; e.g. what drive capacities are you working with?

Do you already have everything in place and purchased?

  • Yes, we do have everything purchased and working (other than config for ioDrive2). Servers are identical Dell R710 with 96GB ram and one X5570. Storage backend is 3PAR F400 with 32 FC hdd. Right now I don't remember our working data set size, but AWR report said that during 24hour period at various sizes we could offload I think as much as 80% of IOPs to Flash Cache. We want to offload as much as possible reads from our 3PAR and lower reads latency. – Mxx Feb 17 '13 at 17:42
  • Decent spec. Are you running single quad-core X5570's for licensing purposes? – ewwhite Feb 17 '13 at 17:47
  • Unfortunately yes :). But since we recently managed to get more licenses at a significant discount I am scoping a 4th server(and eventually replacement for these 3 since their warranty is running out) with a 'proper' CPU, something like dual E5-2690 – Mxx Feb 17 '13 at 17:54
  • I don't think you'll have an issue with that config. An X5670, X5675, X5680 or X5690 would work in the same server and give you more cores and a little bump in speed, but your setup is similar to what I needed to do, except using X5680's and RAID 1+0 on the FusionIO. – ewwhite Feb 17 '13 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.