6

I'm trying to get my head around how a multi lane x4 SAS connector works in a DAS system such as the Dell MD1000. I understand that each lane is 3Gbps, and that multiple hard drives can share a lane. What I'm having trouble finding information on is how are drives assigned to a lane, and how does that change when you daisy chain two additional md1000's to another md1000?

That's 15 disks per shelf and a total of 45 disks in a three shelf configuration. This would be in a single path configuration BTW, meaning one x4 SAS cable.

Edit 1:

All, first thanks for all the help, but I think you're all heading down a slightly different path than what I'm asking. I get the whole thorughput saturation, that wasn't my question. I know what the theoretical max is simply based on the fact that the server has a single 12g conection per array of shelves. Meaning Server >>>>12g>>>>MD10001>>>>>MD10002>>>>>MD10003 is going to be 12g as it all depends on the servers single connection and each md1000 is daisy chained with a 12Gbps link.

My question is simply if there are 15 drives per shelf and 3 shelves, how do i know which drives go on which lane of a given 4x connector? While it's likely not to matter in the end, i was merely curious.

Also FYI, the enclosure is SAS, but the drives are SATA.

Improve this question
8
  • For whatever it's worth, this seems like an awful lot of disks for a single path configuration. How many heads would roll if there was a SAS controller failure or a loss of connectivity?
    – Skyhawk
    May 8, 2012 at 18:05
  • 45 disks on a single path.. do you care about accessing the data?
    – pauska
    May 8, 2012 at 18:06
  • First, I just joined the company, this wasn't my design. Second, this is for disk to disk to tape. While it would suck if an entire array of shelves went offline, it's not the end of the world. .
    – Eric C. Singer
    May 8, 2012 at 18:13
  • BTW, this is a Dell limitation TMK. Dual paths would only provide redudancy, not throughput.
    – Eric C. Singer
    May 8, 2012 at 18:15
  • 3
    @Eric C. Singer Don't forget you can attach each shelf independantly to a card rather than daisy chain them, which would increase throughput. Striping at the server level if neccesary has very little overhead.
    – JamesRyan
    May 8, 2012 at 18:42

3 Answers 3

Reset to default
5

You'll be massively oversubscribed. A single 4-lane SAS link at 3Gbps == 12Gbps total throughput. There's an expander on each MD1000 enclosure, so your 45 (SAS or SATA) disks will easily saturate that link. That's a theoretical max throughput of 1.5 Gigabytes/second over that connection - 12 Gigabits/sec == 1.5 Gigabytes/sec.

Improve this answer
8
  • I am confused where the 1.5Gb/sec throughput number comes from - is that a limit of the expander?
    – Jeremy
    May 8, 2012 at 18:17
  • 6
    12 Gigabits/sec == 1.5 Gigabytes/sec.
    – ewwhite
    May 8, 2012 at 18:18
  • Ok, so help me through this a little more. I get that 12Gbps is the theoretical "aggregate" maximum. Is that measurment reporting a duplexed speed? Meaning if I'm reading data, and lucky enough that its all spread evenly across all lanes, would i then read at 12Gbps or 6Gbps? Secondly, do you know which disks go on which lane? Is that documented anywhere?
    – Eric C. Singer
    May 8, 2012 at 18:19
  • Sorry, I mis-read your statement (you converted to gigabytes). Anyway, I'm still curious. Is that 12Gbps each way (meaning 24Gbps) or one way?
    – Eric C. Singer
    May 8, 2012 at 18:22
  • 2
    Ok, so I just need you guys to understand that I was just trying to see how Dell scaled the disks across the various lanes. Anyway, 12Gbps as a whole is not slow at all. Think about it this way, three to 4 years ago an average ESX server had what, two 1Gbs links for iSCSI? Fast forward to now, usually two 10g links. So... a single 10g link isn't that bad if you really think about it. 1.5GBps isn't what I would call slow
    – Eric C. Singer
    May 8, 2012 at 18:35
3

I'm going to argue that while the theoretical sequential max throughput of all your disks is greater than the SAS chain can handle, with a backup server you may very well never reach that limit or come close to it.

Let's look at some limits in your system.

Theoretical max throughput of a 7.2k SAS drive ~ 1.2Gbit/sec (150 megabytes per second) Theoretical max throughput of 45 of your SAS drives= (45*1200) = 54Gbit/sec Theoretical max throughput of your SAS chain = 12Gbit/sec

So we're down to 12Gbit/sec so far.

How is your server connected to the DAS? 3Gb SAS? Ok, you've got 12Gbit again.

Your application is a backup server. Does it really have 12Gbit/sec connectivity to all of its backup clients? If so, can each client saturate the backup network (reading from their own disks) to the point where you would actually get 12Gbit/sec coming into the backup server? Probably not. That is a LOT of throughput! Your network would have to support that traffic. The backup server would have to have enough CPU to process all that traffic. Etc. etc.

My point is simply, if you have a couple of 1Gb NICs on this box and are using it for backup, you very well may never need to worry about the bandwdith of the SAS chain, because you'll never hit that limit before you max out your network or the throughput capabilities of your backup clients.

That said, if I could design the system myself, I would give you more SAS bandwidth, but my take away here is it may not be a problem in the real world at all.

Improve this answer
3
  • Don't forget maybe 4x write amplification for RAID when thats on the card not in the enclosure. A valid point about saturation though.
    – JamesRyan
    May 8, 2012 at 18:39
  • 1
    @JamesRyan I'm not familiar with this concept, could you please elaborate?
    – Jeremy
    May 8, 2012 at 18:43
  • To be precise on a RAID 5 for example for each write it is 2 reads (data, parity) then 2 writes (new data, new parity), RAID 6 - 3 reads + 3 writes, RAID 1 - 2 writes. This occurs between RAID controller and drives so in a proper SAN or NAS internally, but in a dumb box like the MD1000 it is across the main link. (Also why software RAID can quickly saturate your bus)
    – JamesRyan
    May 8, 2012 at 18:57
1

All,

So here is the answer, the SAS back plane is basically just a huge bus architecture. It works very much like a hub (not a switch). The md1200's to some extent have no bearing on this. From a generic raid controllers perspective it just see's 45 drives. The lanes aren't being broken apart or anything like that. At least not between the chassis.

SAS, by in large is just a big single cable, the easiest way to think about it is Christmas lights. The cable is the path, and the lights are the HDD's.

Improve this answer
1
  • 2
    That's how table-table routing in SAS 2+ works, more ore less. Multiple links between initiator (SAS HBA) and target (storage drive) can be aggregated. Also, there can be multiple paths to the same target (say you had a HBA with 2 miniPorts, and two enclosures; you link from the HBA to each enclose, and a "cross-over" between enclosures; you now have a multi-path layout). It should be noted, this is not how SAS 1 works.
    – Chris S
    May 7, 2013 at 13:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .