Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an iSCSI fail over implementation setup so if one of my storage units fails the other takes over immediately (it also runs the NFS shares). When fail over occurs, volumes are exported, the IP is switched to the other machine and the targets are reconfigured. The fail over of the storage system itself works just fine. I use NexentaStor for my filer.

When I do a test (manual) fail over of my storage the following occurs:

Note: I run the admin VM's on NFS and customer based VM's on iSCSI

  1. All NFS based VM's remain up and working perfectly through the failover and after
  2. All VM 's running on iSCSI eventually report the following:
    • An error about not being able to write to a particular block
    • An error about journaling not working
    • Then the file system goes RO

To get the VM's working again I have to do the following:

  1. Force shutdown of the "broken" VM's.
  2. Detach the iSCSI SR
  3. Re-attach the iSCSI SR
  4. Boot the VM on a different server (5 in my pool) If I don't boot on a different server I get this error "Internal error: Failure("The VDI <uuid&gt; is already attached in RW mode; it can't be attached in RO mode!")" The only way I have found to fix that error is to reboot the entire server it was running on previously which is obviously a huge pain.

Currently multipathing is NOT enabled (but can be and the same thing still occurs). I have edited much of the /etc/iscsid.conf file to work with the timeout settings but to no avail.

In short, my storage fails over properly but XenServer does not keep the connection alive. As a thought, the error that shows up in #4 above might be the ultimate cause and fixing that would fix everything?

Any help would be appreciated more than you know.

share|improve this question

2 Answers 2

I had a very similar problem with iSCSI failover. It's addressed in this question. You can see my accepted solution that I discovered on my own for info on how I solved it.

Basically it involved setting

node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0
node.session.timeo.replacement_timeout = 86400

so that the iSCSI session has enough time to recover before it reports errors up the chain to the kernel.

share|improve this answer
    
yeah, i set that today, same exact values even. I did only set that on one of the servers in my cluster (the only one running domU's atm) so I'll push the change out to all tomorrow and keep trying. I've got a post up on the citrix forums as well to see if someone from Citrix will see it. –  jemmille Feb 19 '10 at 0:48

xe-toolstack-restart fixed it for me.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.