So basically, why should you do parallel processing over a network of
machines when it can rather be done at much lower cost and much more
reliably on 1 machine that supports numerous cores?
You should do parallel processing over a network of machines when it can't be done at much lower cost and more reliability on one machine that support numerous cores.
There quickly comes a point when multiple processors, many more than a single motherboard can handle, is necessary to handle a computation. Beyond that, there is a sweet spot of power consumption to processing gain that is often found in slower processors. A bank of Xeon 7500 processors uses an enormous amount of power, whereas lower performing CPUs might be able to perform half the processing at 1/3rd the power and thus scale much better.
There's also the concept of single point of failure. Would you want a single device to cause a computation to cease being computed at all? The alternative, having multiple nodes and even multiple datacenters, is much better to handle disaster.