Cloud Computing for HPC

Question

I'm looking into if cloud-computing services (e.g., AWS but others as well) can feasibly supplement the traditional clusters (such as Comet, Stampede2, Pleiades) I've been using to solve fluid dynamics problems.

That is, I'd need to run my code (not OpenFOAM which AWS advertises on one of their plans) and it would need ~1TB RAM ~360 processors which have a fast interconnect (and are in the same location to minimze latency).

The more general question is: does cloud-computing mean that everything is virtualized on some unknown, possibly small machine, or can one request use of specific hardware using cloud computing (where in this case, it's being used in the same way as a traditional cluster but the allocation process may be rather different)?

Answer 1

Of the large public clouds, AWS and Azure claim to have solutions to connect instances with low latency networking. Applications have an MPI implementation to target as usual.

In some cases you can influence instance placement closer, such as with AWS placement groups in a cluster strategy. They aren't going to say either way, but presumably this isn't racks dedicated to HPC. Just racks with the fastest hardware and software accelerated NICs, with single tenant or dedicated metal nodes available.

---

Or, if the app can scale up on a big NUMA system, also consider that. NUMA still has inter-node latencies, but low enough to run a single image.

For x86, eight socket totaling 224 cores is available on the latest Xeons as of 2019. More sockets requires custom interconnects, only seen on rare beasts like HPE Superdome Flex Server. The former is available as a high memory AWS instance, the latter is unlikely to be available to rent in a cloud.

For POWER, 16 socket systems like E980 exist in standard configurations, but with not quite as many cores.

Answer 2

AWS offers high-memory pure metal configurations designed to run HANA DBs (In-memory DBs), and at the time of this writing the smallest of those starts at 6TB RAM and has 224 cores for a total of 448 Logical processors.

If that doesn't satisfy your needs, you'll have to go somewhere else like Virtustream (a Dell Technologies Company) that caters to business-critical cloud systems.

_{^{Disclaimer: I work for Virtustream... ¯\_(ツ)_/¯}}