HPC Data-Access & Storage Design

Question

I'm currently working with a cluster-configured application that stores all data locally for each child node. We're having issues scaling up properly and are looking into traditional SAN and Message-Based cluster configurations.

I'm interested in how to properly configure the data access for children/worker nodes and a single SAN. Would my manager node restrict data access to one child-node at a time, or would all children be able to read/write to the same volume on the SAN at the same time in a performant fashion? It would seem that the best option is to restrict access to one machine at a time through the manager node, but that limits data being accessed from a single child at a single time; which is hardly scalable.

The idea is to be able to launch a job which the manager node would alert all children nodes to run. Each node would grab a portion of data to process from the SAN.

I'm rather new to this area and would greatly appreciate any and all feedback/responses possible.

Thank you.

[SideNote: This is a Windows based server setup. We're thinking of trying out Windows Server 2008 HPC specifically, but are not currently using it.]

Answer 1

Having worker nodes with direct access to a SAN and some kind of clustered filesystem doesn't scale in the long run. You might be able to get up to a dozen or couple dozen nodes, depending on the filesystem in question, but eventually you will reach a limit and be stuck.

In a more typical scenario, the nodes would have some kind of local storage to which they would stage in the data they need to work on. You'd then have f storage nodes that share data with the workers using some form of networked filesystem (NFS, CIFS, or something more exotic). The workers then copy the subset of data they need to work on to their local storage, either at the start of the job or as they need it, and write their output back either at the end of the job or are they are creating it.

You'll notice my answer is extremely general because, well, this type of thing is extremely application specific. You may have a distributed application that performs very little I/O, writing out a bit of data to the shared filesystem every few minutes. In this case you can write direct to the shared filesystem from all your nodes and not have to worry about performance. On the other hand, if the nodes are churning out 10's of megabytes per second of output, you definitely wouldn't want (or be able to) have them all writing to the shared filesystem at once.

Setting up an HPC cluster is extremely domain specific, there's certainly no one-size-fits-all answer here.

Answer 2

I suppose the answer to this question depends on how the application handles multiple workers writing to the same place at the same time. You wouldn't want 2 workers writing to the same file at the same time, nor would you want one worker over-writing files created by another worker (at least, I don't think you would.)

I can't answer for your application specifically, but in the database world we generally allow multiple concurrent reads and a single thread/access/worker write access (whether that be a table-level, row-level, or page-level lock.)