Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm currently working with a cluster-configured application that stores all data locally for each child node. We're having issues scaling up properly and are looking into traditional SAN and Message-Based cluster configurations.

I'm interested in how to properly configure the data access for children/worker nodes and a single SAN. Would my manager node restrict data access to one child-node at a time, or would all children be able to read/write to the same volume on the SAN at the same time in a performant fashion? It would seem that the best option is to restrict access to one machine at a time through the manager node, but that limits data being accessed from a single child at a single time; which is hardly scalable.

The idea is to be able to launch a job which the manager node would alert all children nodes to run. Each node would grab a portion of data to process from the SAN.

I'm rather new to this area and would greatly appreciate any and all feedback/responses possible.

Thank you.

[SideNote: This is a Windows based server setup. We're thinking of trying out Windows Server 2008 HPC specifically, but are not currently using it.]

share|improve this question

2 Answers 2

up vote 3 down vote accepted

Having worker nodes with direct access to a SAN and some kind of clustered filesystem doesn't scale in the long run. You might be able to get up to a dozen or couple dozen nodes, depending on the filesystem in question, but eventually you will reach a limit and be stuck.

In a more typical scenario, the nodes would have some kind of local storage to which they would stage in the data they need to work on. You'd then have f storage nodes that share data with the workers using some form of networked filesystem (NFS, CIFS, or something more exotic). The workers then copy the subset of data they need to work on to their local storage, either at the start of the job or as they need it, and write their output back either at the end of the job or are they are creating it.

You'll notice my answer is extremely general because, well, this type of thing is extremely application specific. You may have a distributed application that performs very little I/O, writing out a bit of data to the shared filesystem every few minutes. In this case you can write direct to the shared filesystem from all your nodes and not have to worry about performance. On the other hand, if the nodes are churning out 10's of megabytes per second of output, you definitely wouldn't want (or be able to) have them all writing to the shared filesystem at once.

Setting up an HPC cluster is extremely domain specific, there's certainly no one-size-fits-all answer here.

share|improve this answer
    
Right now they are setup with local storage only. But we're looking other solutions that may fit our needs better. The problem with local only storage is availability. If one node goes down, we loose the data it's computing on with it. We are outputting a good bit of data during each job on each node. So all nodes reading/writing to the same SAN doesn't make much sense, unless there was some parallel overlay to its filesystem. –  Nicholas Mancuso Oct 5 '09 at 20:18

I suppose the answer to this question depends on how the application handles multiple workers writing to the same place at the same time. You wouldn't want 2 workers writing to the same file at the same time, nor would you want one worker over-writing files created by another worker (at least, I don't think you would.)

I can't answer for your application specifically, but in the database world we generally allow multiple concurrent reads and a single thread/access/worker write access (whether that be a table-level, row-level, or page-level lock.)

share|improve this answer
    
Right now each worker would not be overwriting each others files. But I'm worried that using a SAN without any parallel filesystem will not be as performant. It could make sense to give each child its own logical volume, but that would seem to detract from the idea of just adding on a compute node. –  Nicholas Mancuso Oct 5 '09 at 19:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.