Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We are working on a project which involves different hardware all hosted in a single rack. The machines are mainly IBM servers: 2 x206 (scsi), 1 x226(scsi), 2 x3400(sata) and another assembled machine with sata controllers. We are using several raid controller. Some machines have only one Serveraid controller, others have one or more controllers not always Adaptec ones. All the firmwares and bios are updated. All the servers and connected devices are under ups.

Over the last 4 months we experienced several strange behaviours in our hardware. Suddenly and randomly we loose 2 or 3 drives and the raid volumes stop to work. It can happen once a week but never at the same time of the day or week.

Most of the times a rebuild process fixes the problem, sometimes we loose the data. Very often we just need to unplug the raid controllers, restart the server and the problem is fixed.

At the beginning we thought it was due to firmware bugs but we performed an accurate update for every machine and raid controller and there is nothing else we can do on the hardware. We have really no hint on what's causing all these troubles.

We are starting to think that it's an environmental problem but we don't know if there could be something interfering with our hardware. Have you ever heard of something like that? Do you have any idea on how to investigate the problem?

share|improve this question
    
How's the temperature in the server room? What kind of cooling and airflow do you have? –  joeqwerty Feb 11 '10 at 12:08
    
Which servers have the issue, and what model are the hard drives with the problems? –  Greg Askew Feb 11 '10 at 15:50
    
@joeqwerty it's an open rack and the room temperature is 18 deg. The room is a laboratory and not a common "server room". I'm absolutely sure that the temperatures are good. @Greg all the servers :-( all the drives from any brand and technology (scsi,sata). I would list them but I guess it's useless, don't you think? Thank you very very much for the support. –  damko Feb 11 '10 at 18:55

2 Answers 2

This can easily be due to firmware bugs, not on the controller, but on the drives. Seen that much too often to count.

share|improve this answer
    
we updated the hard drives firmware too. What's strange is that different hard drives have the same behaviour. I guess it's statistically hard that we purchased all the bugged hard drives on the market. Right? –  damko Feb 11 '10 at 12:46
    
In that case, you need to check out the env. conditions - the server room can be either too hot or too cold, weird powergrid behaviours etc. We had a case once, when an entire server farm would go down in a certain order, taking down racks one by one, at exactly the same time every day. Apparently the building had it's back wal to a river, where a ship was docking, and every day, at tide, the ship's radar was levelled up with the server farm floor, and as the radar turned, racks would come down one by one. –  dyasny Feb 11 '10 at 13:32
    
this is interesting. we have some antennas for mobile communications very closed to the lab (~80 meters) at the same level of the shop. Do you think they could have something to do with this? What's the frequency of a radio wave to cause interferences with raid controllers? –  damko Feb 11 '10 at 20:10
    
That's a bit beyond me really. I would open a support ticket for a set of servers with their vendor, and try to get them escalated to the hardware SMEs - this usually leads to a solution, or some senior tech being sent out –  dyasny Feb 12 '10 at 7:11

If I had drives from different vendors on RAID controllers from different vendors in servers from different vendors failing at an abnormal rate, I'd start looking at high temperatures and poor airflow in the server room as a potential cause of the problem.

share|improve this answer
    
I agree but it's an open rack with all the sides open and the room is fresh. Checking the temperature of raid controllers from the management GUI we notice nothing abnormal. –  damko Feb 11 '10 at 14:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.