16

Say I had two servers which needed super-low latency (Database, file etc.). Would it be possible to directly connect the two servers with 10GbE, so each server had 1 (in the real world it would have 2) connections to the 'main' network, but 1 network card with an ethernet cable that connected directly to the second server, no switches or routers, just a direct connection

                         Internet/Datacenter
                                 |
                                 |
                                 |
                                 |
                                 |
                                 |
                                 |
                        --------------------
                        |                  |
            ------------|      Switch      |-----------
            |           |                  |          |
            |           --------------------          |
            |                                         |
            |                                         |
            |                                         |
            |                                         |
            |                                         |
            |                                         |
            |                                         |
  Network Card 1 (eth0)                     Network Card 1 (eth0)
            |                                         |
  --------------------                      --------------------
  |                  |                      |                  |
  |     Server 1     |                      |     Server 2     |
  |                  |                      |                  |
  --------------------                      --------------------
            |                                         |
  Network Card 2 (eth1)                     Network Card 2 (eth1)
            |                                         |
            |                                         |
            |               Direct 10GbE              |
            -------------------------------------------

My first question is, would this even be possible? Would they need any unusual/special services configured to let them talk over this network other than a standard file in /etc/sysconfig/network-scripts/? They would both have static IPs on eth1 but how would things like routing work? I'm not an expert on networking so this is probably a n00b-ish question

Second question, is there any point? Would there be any advantages doing this over just letting them communicate over the standard network connection via the switch, or giving them a second dedicated network just for communicating intra-server (Since bandwidth would be used on the standard network by clients accessing the servers). Assuming latency was the priority.

I know there are some issues with this method, like when we came to add a 3rd server we'd ether have to give every server another network card and probably set up some very complicated replication triangle thingy but since this is hypothetical lets ignore that.

And since latency is the key issue, would fiber be better over ethernet (Speed isn't important so long as it can do a couple of Gb/sec)

I phrased this question from a linux POV, because that's my background, but it could apply to any server/device

4
  • 1
    As a side note you should use UDP and not TCP to gain latency at the expense of loosing guarantee of delivery. Depending on the size of your packets jumbo frames could help too by limiting the number of packets.
    – Shadok
    Nov 8, 2011 at 10:31
  • @Shadok UDP is not ideal for database servers, file servers etc. but for monitoring, logging and other non-critical things UDP is a much better option I agree. I think any network setup like this would most likely be something that required guaranteed delivery but UDP packets over the main network have many uses.
    – user80776
    Nov 8, 2011 at 10:35
  • Ok, I was under the impression that latency was your primary concern and tought you possibly had a way to check the integrity or acknowledge delivery of packets in a different way than TCP. One idea I just had would be to implement a simple counter incremented at each packet and then request missing ones again or just drop them, but I'm digressing :)
    – Shadok
    Nov 8, 2011 at 15:21
  • @Shadok - If you did manage to rig a UDP connection to a database it could lead to unpredictable behavior and corruption. Plus you'd basically be turning UDP into TCP by sequencing packets and requesting missing ones. That's what TCP is for. It would be better to fatten up the ethernet to 100g or use a fiber connection
    – 111
    Dec 20, 2013 at 15:13

6 Answers 6

10

There's no reason why you technically can't do this.

I'd probably do something similar, under the circumstances, actually. From a purely linux point of view, it's really easy, just give the connection an IP address with a /30 bitmask, giving you 2 IP addresses, then it's a simple Point-to-Point link.

If you wanted to grow the network, you could get a 10GE switch, and then have a seperate VLAN for traffic between servers. There's some very shiny gear in the Force10 range of switches that can do line-rate 10GE switching, with enormous buffers.

4

I cant comment a Linux point of view but I will just use my knowledge and ask some more questions.

Are you that dependant on low latency and that you need to keep these servers in sync? Are they both running a database or something? 10GbE should suffice most needs of 2 servers to keep them in sync. I would soon rather spend the money on a decent switch in the middle of the 2 instead of going the route you are looking at.

You could with a decent switch tag these ports to prioritise the traffic and even QoS the traffic that needs to be real time.

My thoughts.

3
  • Giving them their own switch/'private network' would be a much better option than a direct connection, that way if we needed to add more servers later it would be a lot easier. Also prioritizing traffic is a good one, I didn't actually realize you could do that, does it depend on the switch features/manufacturer or is it a fairly standard thing?
    – user80776
    Nov 8, 2011 at 10:37
  • QoS on the switch is very much a vendor-dependent thing, some support it, some support it in a very proprietary way, some support it as per the RFC, but you'd probably have to have it on all devices on the network to see any decent effect. Nov 8, 2011 at 11:03
  • +1 because many switches work at what is essentially wirespeed now. A properly configured network will be advantageous in a myriad of ways already discussed, and will also aid fault finding.
    – Dan
    Nov 8, 2011 at 12:09
4

I have actually done this between two laptops. Most modern LAN adapters have auto negotiation between them so you can use a regular lan cable.

Set static IP addresses that are not in the same range as any other subnet you are using - for example, if my systems are on a 192.168.x.x subnet, i use a 10.0.0.x subnet between them. Otherwise, it should just work

3

Security vs. Performance vs. money.

  • If the back channel traffic is high and money is low, use a direct connection. It's done everyday and in many real world situations can have better performance than an already overloaded switch.

  • If the back channel traffic is low and security is medium or low, bond NICs to increase overall Internet throughput - two connections from each server to the internet, multi-home NICS to "isolate" replication traffic (Separate IP spaces makes it easier to firewall, audit, do packet trace diagnostics, etc.).

  • If security is high and plenty of money use a switch. Easier to expand. Easier to diagnose problems.

In the given scenario a switch purchase would not be warranted. Utilizing an existing switch with VLAN segmentation would possible make sense. Although, I can't see any reason to plug into the switch unless the servers are Co-lo'd, i.e., not physically accessible. It's a waste of two switch ports unless packet capture/debugging is active.

2
  1. I'm pretty sure you will need a cross over cable to connect Server 1 to Server 2.
  2. You will have very low latency but latency across a LAN with modern switches is about as close to zero as you can get. I sincerely doubt you will see any measurable performance gain.
2
  • 11
    Gb and 10Gb do auto-MDIX, so you don't need a crossover. That's only for 10/100.
    – MDMarra
    Nov 8, 2011 at 11:53
  • I think your right on that on, @MarkM I don't thing I've ever actually tried it in real life.
    – SBWorks
    Nov 8, 2011 at 12:05
1

There is no advantage of using such setup. the switches today are lightening fast, so you never face any visible latency due to switch. and scalability would be a big issue for you as well. Also, there would be problem of setup of routing as well, as you will have to maintain TWO Separate networks instead of just one.

1
  • 2
    The advantage in my situation was that direct connection eliminated the need to use SSL encryption on a database connection, because it's going over a private wire and not a semi-private switch where passwords could be sniffed.
    – 111
    Dec 20, 2013 at 15:23

You must log in to answer this question.