7

Background:

Currently, we use Rackspace cloud servers. We have no intention to stop using them, but would like to look into setting up a cluster of physical servers (probably desktop computers in the $400 range with 8gb memory each) to offset some of our load and work as a secondary, more powerful, less reliable system. To put things in perspective, we can buy comparable desktop computers for the same price as we pay in one month to rent them on Rackspace Cloud.

I understand that this is generally a dumb idea. However, in this particular instance, the server cluster is needed for its computation power. It is not mission-critical, it does not host a consumer-facing website, and if it goes down for a day or two, its not really a problem.

Currently, we have access to business class verizon fios. If I understand correctly, we can get at least 25 dedicated IP addresses with this service, this should be enough.

Requirements:

  • Each server runs Linux Centos 6.3
  • Some of the servers run Python and execute processes from a task queue (Redis or RabbitMQ)
  • Some of the servers are capable of serving static files and Python driven REST APIs
  • Some of the servers host a Cassandra database cluster
  • One or more of the servers are a Redis database servers
  • One or more of the servers are PostgreSQL servers

Questions:

What kind of router or switch is needed?

  • We would like the computers to be able to communicate effectively with each other via internal IP addresses. This is especially important for communicating with servers hosting Redis that need to be able to respond to requests very quickly. Are there special switches or routers that need to be used to connect the servers together?

Are Desktop computers ok for this?

  • We have found that we are mostly RAM-bottle necked, I understand that some servers have highly superior CPUs, but I'm not sure we need CPU power as much as we need RAM, which is cheap in Desktop computers.
  • Will we have problems with the WIFI cards in the desktops or any other unexpected hardware limitation?

What tools should be used to "image" the servers.

  • For example, when we get an installation right for a Redis server or Cassandra node, are there tools that come with Linux Centos 6.3 to image the server to a USB drive or something like that?
  • Or do we need to use some other software for this?

What other things are we missing that we should be concerned about?

2 Answers 2

3
  • You do not need internet IP for that - you can set up a site to site VPN and keep all private. This is what I do with the current 2 locations (my basement in Poland, which has a 2 rack server setup now, and our outside site in chicago). I use 2 routers (virtual in chicago, physical here) to establish private routing.

  • Desktop is ok, but depending what you do you may want to go with E3 Xeons - 32gb memroy is possible there, but more important: ECC memory. I really dont want my simulations having bad result due to memory flipping. As explanation: we assign 1gb / core, this is a queue controlled HPC cluster, doing in our case CPU intensive simulations.

  • If you are RAM bound - look for AMD. Their G32 socket has an 8 core setup that can handle for a relatively low price 64gb ECC memory. Good for virtualization also. I am getting some of thsoe for an upgrade for my main servers (hyper-v platforms, serving also as domain controller etc.) due to the superior RAM.

  • Switch - all nice and dandy. Depending what you do you may want to get one of hte netgear switches that has 48x1gb and 4x10gb - and use the 10gb to link up to your file servers. We just do that - because with the now coming 2x computers (we just add 16 in the blade center) the server side would overload network wise during certain operations.

Otherwise you really need to understand that technology has evolved - switches etc. are reliable today and performant. Nothing special is needed. Network setup will be non-trivial, but not really very complex either.

4
  • Indeed, if you're not engineering for high-availability the hardware problem is much easier.
    – sysadmin1138
    Jul 11, 2013 at 12:20
  • Expecially if you go HPC - availability of nodes is a minor issue. Plus, seriously, desktop hardware is also QUITE available - the rest can be handled in many cases with some spare parts and redundant configuration (2xdns, database mirroring) more (financially) efficient.
    – TomTom
    Jul 11, 2013 at 12:27
  • This is a good answer, but missing an answer to the last question, so I'd like to answer that by saying: puppet. Use some kind of configuration management system. You've already identified that you expect this to be (relatively) unreliable, so having the ability to have one system die in 12 months, replace it with a brand new system and have puppet bring it back in line to what you need automatically will be magic for you.
    – fukawi2
    Jul 11, 2013 at 23:38
  • I left that out on purpose - thanks for filling it up. Not doing Linux my answer would have been very MS specific (SCOM, SCCM).
    – TomTom
    Jul 12, 2013 at 8:47
0

You can buy very fast and reliable (2-socket times 6-Core) servers with 48GB of RAM for around 3000 dollars. This is even with optional extras such as physical ILOM cards, so you can probably trim that down a few hundred.

Going back to your intial estimate for workstation price something similar comes to (48 / 8) * 400 US$ = 2400 U.S. dollars. That's without the cabling, switches and other overhead.

Now I know you probably dont want this answer, but are you sure you want to be doing this? The only reason I see to go the desktop-cluster route is wanting to experiment with new tech (there's nothing wrong with that of course).

1
  • Or for more processing power outside a data center setup - like a SMB office normally is. I was / am running the same for my HPC cluster and we are JUST NOW adding a Dell M1000e blade center - so far most nodes are virtual machines on mostly desktop level hardware which served us well the last years. Heck, a couple oare AMD octo core in wooden frames. For a small business that may be a better value proposition.
    – TomTom
    Jul 11, 2013 at 10:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .