15

First - this is not a shopping question and not about concrete prices but about general feasibility. It makes no sense to look for a manufacturer if the approach is bad.

I am moving my company to new offices in September, and during the move we will expand and consolidate our computing cluster. Right now it is in a data center, but I have a nice room in the basement prepared for it.

I think about cooling. We will likely use around 10kw of power by end of the year. That is a LOT of heat, and cooling will be expensive. I am located in south Poland, close to the German border. This is an area where water is available for a relatively cheap price - "wasting water" is not a concern here. My situation is thus different from, for example, a company in Spain ;)

Physics tells me that to heat 1 liter of water by 1°C I use 1 Calorie (1KCal), and 1kWh is (and we can assume 100% efficiency - water heaters are pretty efficient) 750 Calories. That means that 1kWh is 750 liters heated by 1°C. 10kw and a 20° increase in heat would mean that per hour I need 375 liters. That is 6.25 liters per minute and not THAT much ;) We're talking 270 cubic meters here. Even in summer, the underground pipes really cool down the water a LOT more ;)

Is such an approach feasible? Has anyone done this? We have a 10kw installation for now. Is it feasible to reuse that heat? The alternative is a decent cooling system that WILL use around 2.5kWh to run. Dropping the water would:

  1. Get me a quite cold input compared to the outside air, even in summer (I.e. a lower temperature medium to drop the heat in)
  2. Replace the need to dump heat into the outside air, which may be problematic. If the air is 22°C, that is a LOT to fight off, but OTOH the water will be quite cold. I also would possibly save the investment for the outside part of the cooling circuit.

Now, second question - is there a feasible way to heat a house with that? ;) After all, it is a LOT of energy in that water ;)

If this is a bad idea, I'll stop here. If it is not, I'll start looking for suppliers. Maybe my math is wrong?

7
  • 2
    Interesting idea - as the answers have shown it's been though of before, but, personally, I wouldn't want water anywhere near a hundred-thousand-currency server cluster - regardless of how much money gets thrown at a water-cooling system.
    – tombull89
    Jun 18, 2012 at 17:57
  • 1
    Well, then use it for coutner cooling. In rack cooling system - what you dump the heat in? ;) I Just found en.wikipedia.org/wiki/Deep_water_source_cooling as an idea - now I dont have that, but maybe it works with stuff like heating the house, dropping warm water and getting fresh water etc. Evaporative cooling alone would make things a lot more efficient, energy wise.
    – TomTom
    Jun 18, 2012 at 18:02
  • Water is not typically the medium of choice (because getting 100% pure water is nearly impossible, and impure water tends to conduct electricity). Oil immersion cooling is usually the method of choice... Recycling the waste heat from this system is left as an exercise for the reader, but I'd submit that air exchange using the exhaust from the fans is probably easier/cheaper to implement...
    – voretaq7
    Jun 18, 2012 at 18:03
  • @voretaq7, I've seen Mineral Oil Cooling before, but on a single desktop and din't think it would be done on such a high scale. I was obviously wrong.
    – tombull89
    Jun 18, 2012 at 18:10
  • 1
    10kw is not a high scale. Densities these days go to 40kw PER RACK ( ahve seen back door cooling units for that). I get standard these days 2 processors per 1U - make the maths. THe days of 2.5kw racks are over... ;)
    – TomTom
    Jun 18, 2012 at 18:13

5 Answers 5

4

There are two approaches to this, one I know for sure exists in the market, and another I've only heard about in custom builds.

Market-ready

These solutions use a water loop to provide at-rack cooling. They produce chilled air on the intake side of the rack, and use the water loop to exchange heat between ambient and what it produces. These solutions exist, and don't require any modifications to racked hardware to make work. They sometimes require special rack-doors to ensure correct airflow, but not always; some solutions create a 'curtain' of cold air in front of the server intakes.

With this approach you can easily break the 10KW/Rack barrier for power density.

Custom Builds

This approach brings water into the device themselves. I don't know of any off-the-online-store servers that have water loops built into them, but I hear that the major manufacturers can help you if you really do need 10K devices with such requirements. The advantage here is that you can push higher on the power envelope as you're not relying (as much) on air transmittance of heat, as the water-block itself does the heat transfer from the devices.

Theoretically you can get really high power densities with the custom-build method.

3
  • Sorry, that does not answer the question per se. I KNOW rack solutions exist. Rittal and other sell them - that is an in rack air condition. The question is more about the other side, getting rid of the heat. I prefer not to pay for an AC outside unit that JUST wastes the heat when I can have a more efficient solution. Maybe cold water works as counter-side, maybe it works even better.... when you use the warm water in the house?
    – TomTom
    Jun 18, 2012 at 17:54
  • @TomTom Ah, you're looking for the heat-exchanger side outside the server-room! Beyond my immediate know-how, I'm afraid.
    – sysadmin1138
    Jun 18, 2012 at 18:01
  • In the room I likely go with a rack or side mount install - checking that part at the moment. There are some that use water but make sure it is save (no water ABOVE the servers, a leak goes down to the bottom and there are only pipes, we get data and power from top anyway). THe problem is more - once the liquid is out of the room, how do I get that stuff cooled most efficient ;)
    – TomTom
    Jun 18, 2012 at 18:32
7

As to the cost efficiency of water cooling in general, there's another good question that I answered already: Why datacenter water cooling is not widespread?

As to your specific situation, I would highly recommend looking at ground-loop heat pumps with economizers. It's likely cool enough in your Winter to just run the economizers, using what is basically outside air to cool your data center. In the summer heat pumps are much more efficient than standard AC units, roughly twice as efficient typically.

Using common loop heat pumps can allow you to extract the heat from one area (data center) and pump it into a different area (office space or an attached home even) quite efficiently too (though it's probably more cost effective to go with conventional sources, it depends on energy costs in your locality).

You can use air to dump-water to cool as well, air within the rack (as is typical) and fresh water to cool the air. This water can come from fresh sources and be dumped after "heating"; this used to be common to run commercial size refrigerators because the math worked out just as you put it in the Question. It's not commonly used today because water is more expensive than it used to be and the math no longer works (at least in my corner of the world, and water is pretty cheap here too). It would be more efficient to directly cool the server with water-cooling, however you introduce the risk of leak as well (which carries it's own risk or insurance costs).

The biggest detractor you'll have in implementing any of the aforementioned is that it's expensive equipment. Most businesses don't care if it pays for itself in 10 to 20 years, they only care about this year's bottom line. The payoff period depends on your local prices..

11
  • 1
    Actually no. Even if you take out the server room - ignore the water side etc..... what about using fresh water as heat dump? For a 1mw installation that may be a problem (unless you are on a river), but it would take out half of the energy problem for the AC and provide a SIGNIFICANTLY colder heat dump than outside air. My question is not about widespread - I am willing to invest here. My question is the economy / feasibility. And there ARE kits that do in rack cooling - even with water.
    – TomTom
    Jun 18, 2012 at 17:51
  • 1
    Imperial - Galons are bad enough, but a BTU is not something we use here. It is all about Watts ;)
    – TomTom
    Jun 18, 2012 at 18:08
  • 1
    Using water as the heat sink in a typical refrigeration cycle (typical coolers) makes them 2-3 times more efficient (assuming the water supply is roughly ground temperature, 12ºC). This applies to both closed ground loops (commonly called geothermal heat pumps) and open loops, such as fresh-dump water. Evaporative systems typical are much less efficient, and usually most suitable to steam powered applications (where the local utility supplies steam, commonly from trash incineration).
    – Chris S
    Jun 18, 2012 at 18:22
  • 1
    It is better. Found some claims for cooling wiith ground water that puts cooling at 500 watts (!) for 15kw (!) as long as water in temperature is 14 to 18 degree.
    – TomTom
    Jun 18, 2012 at 21:15
  • 1
    Water provides 1.16W cooling per degree-liter. To cool 10kW you need 11600 degree-liters; most computer will run around 32°C. If the ground water started at 12°C that'd be a 20°C rise (assuming an unrealistic 100% efficiency), so 11600l-C ÷ 200C = 580l (my 650l estimate is more realistic). Your 500 watts number means nothing without a temperature differential and flow rate. For example, if you use a heat pump (Peltier, evap, refrigerant) you can increase the temp differential, but then you'd have to pay for electricity to run that too.
    – Chris S
    Jun 18, 2012 at 22:04
6

TomTom - I very nearly bought a bunch of HP's MCS water-cooled racks for a project a few years ago but chose to change data center instead.

It's not exactly cheap but they stand by it in terms of reliability, for instance they're happy for you to put their Integrity stuff them. It can handle ~36Kw per rack by the way.

Just wanted you to be aware of this stuff as it doesn't exactly leap out of their website.

3
  • Yeah, especialy HP ;) Still there are some around - and I thin ka side cooler is best. Evaluating currently the approaches. My main question is not the server side, it is basically how to dump the heat ;)
    – TomTom
    Jun 18, 2012 at 18:34
  • There's a data centre in east London we use that has some kind of crazy way of doing that in the event of major AC failure - they have a big pond between the buildings filled with lots of regular rain water. In the event of AC failure they can somehow route the heat through it, don't know the particulars but it's not a small pond, maybe 100m x 10m?
    – Chopper3
    Jun 18, 2012 at 18:37
  • 1
    Well, the house has a sing in the back for waste water from toilets etc. - that is not in use anymore. I could pay some people to clean it up, then install a tank there for rainwater. Agreed ;) Actually rainwater may as well reduce the water invoice significantly ;)
    – TomTom
    Jun 18, 2012 at 18:51
0

I am unable to answer all the questions you have asked but there is one host who has based mostly all of their strategy on water-based cooling and that is OVH. All their servers are water-cooled which allows them to offer interesting prices. If I remember the stats correctly, they are the biggest hosting company in Europe and 5th in the world. So yes, it can work.

Is it simple ? No. They manufacture most of their watercooling equipement which induces steep upfront costs. Nonetheless it leads to great energy efficiency.

Can you reuse the heated water? I'm not so sure as you don't want the water to get too high in temperature as it will lose it's efficiency to cool down your servers.

0

LRZ has a large warm water cooled HPC cluster, and they're doing this kind of waste heat recovery.

SuperMUC uses a new, revolutionary form of warm water cooling developed by IBM. Active components like processors and memory are directly cooled with water that can have an inlet temperature of up to 40 degrees Celsius. The "High Temperature Liquid Cooling" together with very innovative system software promises to cut the energy consumption of the system. In addition, all LRZ buildings will be heated re-using this energy.

More information at http://www.lrz.de/services/compute/supermuc/systemdescription/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .