Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're looking at hooking up 48 small embedded systems with 10/100 Ethernet ports to an Ethernet switch and then have that switch talk to a server upstream via a faster connection. I have a couple of questions about that scenario:

  • What kind of upstream connection is best (fiber, other?)
  • Would it be reasonable to download 1GB/hour from each of the 48 systems concurrently? We'd be using some kind of TCP based protocol of our own design.

Thanks, Andrew

share|improve this question
2  
If I am doing my math right, that is 48 connections transferring 1Gigabyte/hour works out to about ~110Megabit/s. ((2**30*8)/(3600))*(1/2**20)*48. A single 100Mib/s uplink is almost enough assuming traffic is constant. –  Zoredache Feb 25 '11 at 23:32
    
more or less, but there is packet overhead and the ability of the switch CPU to cycle the information of 48 ports at one. If its a switch designed for this, no issues, but if its a set of daisy-chained 12 ports, there will be major issues. Its all in the equipment that you choose. –  MaQleod Feb 26 '11 at 19:42
add comment

migrated from superuser.com Feb 25 '11 at 23:09

This question came from our site for computer enthusiasts and power users.

4 Answers

up vote 10 down vote accepted

No easy way to find out or to help you without more information.

Fibre connections are good for linking routers to other routers or very high end equipment to routers.

If the server and the router support fibre, why not!

However, as for sustained bandwidth and if 1GB/hour is possible... Follow this advice:

Cheap £20 switch = bad
Expensive gear = good (typically)

Hopefully you get what I am saying... You get what you pay for, as for actual usage, depends on many factors (including the drivers on the embedded systems - can they sustain 100Mb/s each at full utilisation?).

I hope this helps and happy to answer any follow up questions.

share|improve this answer
3  
Good answer by Wil. If throughout is an issue, you could use UDP as you have a fairly reliable, short-distance link, but you would then have to implement some form of error/delivery checking protocol on the embedded systems and server and the systems may not have the CPU capacity for that –  Linker3000 Feb 25 '11 at 22:31
    
Ok, thanks. We can spend money on the switch (although I'd love to find a 48 port switch for 20 pounds.) Back of the envelope says it would probably be enough to use 1GbE for the uplink and I can avoid costly fiber connectors. We don't know about the distance yet so that'll come into play as well. –  Andrew Queisser Feb 25 '11 at 22:47
    
Fibre is really good for distance and generally speaking, you only (usually) get good fibre cards... As long as you get a good off loading 1Gb card for the server (doesn't have to be VERY expensive - Intel sell them for around £40-£60), they should be able to sustain very high utilisation. It really just comes down to cost and driver support of the devices. –  William Hilsum Feb 25 '11 at 22:53
add comment

When sizing a switch one thing to keep in mind as to what the "fabric" of the switch can support. Just because a 48 port switch has for example 48 Gigabit ports doesn't mean that the switch can handle all 48 ports pushing 1GB in each direction (Full Duplex) at the same time.

I'm afraid however that even though some switches say they might have a 48GB fabric they might not actually pull that much data.

Also when you talk about how much data you are transferring that usually doesn't account for protocol header overhead. Even forgetting about this overhead and the fact that the port rating is the best case scenerio:

Asking wolfram Alpha: http://www.wolframalpha.com/input/?i=48+Gigabyte+in+1+Hour

Is says 106.7 mbit/s is needed for the uplink.

share|improve this answer
    
Ok, thanks, I'll investigate what "fabric" our 48 port switches have. Some of the specs have pps and overall bandwidth so I'll look into that. I think in your last paragraph you have a factor of 48 too many. There are 48 systems with 1GB/h of net data each, so the total net upstream is 48GB/h. –  Andrew Queisser Feb 26 '11 at 0:00
    
@andrew I was going with bytes for the amount of data and bits for the rate –  Kyle Brandt Feb 26 '11 at 0:23
    
I agree with @Andrew, I'm not sure your math is correct there. 106.7mbps should be what is required for the UPLINK, assuming traffic is consistent. Only 2.22mbps is required per system, I'd guess that this is something the cheapest NEW 48 port switch you can buy would be able to handle, as long as it has a 1Gbps uplink. –  Jed Daniels Feb 26 '11 at 0:23
    
Oh I see my mistake, will fix it in a bit :) –  Kyle Brandt Feb 26 '11 at 0:27
add comment

It should be completely reasonable to get 1GB/hr, but the question will be in your upstream bandwidth. Keep in mind that Fibre is a media, not a guarantee of bandwidth (many mid-range Dell switches have uplinks for Fibre, but those uplinks are only rated at 1Gb). Fibre is used because of the range it's capable of, and the speeds that it can achieve over those distances. If everything was in the same rack, you could just as well use Cat6 copper with a 10Gb ethernet card and achieve good speeds.

That said, you may be limited by your upstream as to how much you are able to transfer. If you're making distinct connections to each host, you may be limited by your upstream bandwidth, even though the switch itself is fast enough. Think of it like a funnel, and you're funneling all of the data through one port (of indeterminate bandwidth, as this isn't specified in your question).

Thanks!

share|improve this answer
    
Right, good point. The upstream bandwidth needed will be in the neighborhood of 48*1GB/h, which translates to 115Mb/s, which doesn't sound too bad. The naive approach would have upstream process(es) with 48 socket connections to the 48 embedded systems so it also becomes a scaling exercise to see how we will distribute the connections among threads/processes/cores. All the gear will preferably be HP, since that's who I work for, although not in the networking or server division. –  Andrew Queisser Feb 25 '11 at 23:38
add comment

First off, why design a new protocol when there are so many well refined ones already in existence? I've never understood the need to re-invent the wheel unless you have a solid reason that requires something so completely different than what already exists. Take a look at existing options first.

Second, if you're going to have all the clients on a 10/100 port switch, your limit is 100mbps minus overhead per client. It doesn't matter what the switch they connect to is capable of, that is the fastest you'll get, period, but that should be no problem to pull 1GB in an hour. If you plan on having 48 clients pushing that much traffic all at once, the switch will require a gigabit connection at the very least, to whatever it talks to further upstream, and even then, you'll never handle all 48 clients at once on anything less than an enterprise grade switch. This cisco would work, but keep in mind the cost of this device is about $1400. It is also stackable so you can add more as you need more.

I've always liked the idea of fiber, but cables are expensive and cleaning and troubleshooting fiber problems is expensive too (standard tester and cleaning equipment is around $2000, more if you want anything other than simple db loss). If you don't know anything about how fiber works or how to clean a fiber ferrule, don't go with fiber, gigabit ethernet will be fine for most people's needs. A gigabit backbone to the server that will be holding the data will suffice.

What you really need to pay attention to is the ability for the server to read off its hard drives for 1GB/hour from 48 machines and the CPU power as well. Your limitations will be on the hardware of the server, not your networking gear.

share|improve this answer
    
Thanks for rounding out the discussion - we are looking at using standard protocols as well as custom but there's no filesystem on the embedded systems and TCP already gives us guaranteed delivery so we may not need much more than a raw socket connection. The fiber thing indeed sounds scary, especially since this stuff will go into a semi-dirty environment. I'm now leaning toward traditional cabling. –  Andrew Queisser Feb 28 '11 at 15:17
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.