156

This is a software design question

I used to work on the following rule for speed

cache memory > memory > disk > network

With each step being 5-10 times the previous step (e.g. cache memory is 10 times faster than main memory).

Now, it seems that gigabit ethernet has latency less than local disk. So, maybe operations to read out of a large remote in-memory DB are faster than local disk reads. This feels like heresy to an old timer like me. (I just spent some time building a local cache on disk to avoid having to do network round trips - hence my question)

Does anybody have any experience / numbers / advice in this area?

And yes I know that the only real way to find out is to build and measure, but I was wondering about the general rule.

edit:

This is the interesting data from the top answer:

  • Round trip within same datacenter 500,000 ns

  • Disk seek 10,000,000 ns

This is a shock for me; my mental model is that a network round trip is inherently slow. And its not - its 10x faster than a disk 'round trip'.

Jeff attwood posted this v good blog on the topic http://blog.codinghorror.com/the-infinite-space-between-words/

6
  • 11
    Sometimes yes, sometimes no. What network? What disk? Feb 22, 2011 at 1:46
  • 1
    The other interesting data from top answer: 1 MB sequential read from network vs. disk. I suspect the "round trip" time omits any significant data transfer.
    – Paul
    Feb 22, 2011 at 20:48
  • Paul: Depends on your MTU, I'm sure. (1MB MTU? Awesome!) Feb 22, 2011 at 22:01
  • 2
    I'd love to see some of these answers reconsidered in the light of 10Gbps network gear being widely available.
    – chicks
    Jul 8, 2015 at 16:11
  • gigabit network vs raid 5?
    – derelict
    Jul 22, 2015 at 21:18

13 Answers 13

162

Here are some numbers that you are probably looking for, as quoted by Jeff Dean, a Google Fellow:

Numbers Everyone Should Know

L1 cache reference                             0.5 ns
Branch mispredict                              5 ns
L2 cache reference                             7 ns
Mutex lock/unlock                            100 ns (25)
Main memory reference                        100 ns
Compress 1K bytes with Zippy              10,000 ns (3,000)
Send 2K bytes over 1 Gbps network         20,000 ns
Read 1 MB sequentially from memory       250,000 ns
Round trip within same datacenter        500,000 ns
Disk seek                             10,000,000 ns
Read 1 MB sequentially from network   10,000,000 ns
Read 1 MB sequentially from disk      30,000,000 ns (20,000,000)
Send packet CA->Netherlands->CA      150,000,000 ns

It's from his presentation titled Designs, Lessons and Advice from Building Large Distributed Systems and you can get it here:

The talk was given at Large-Scale Distributed Systems and Middleware (LADIS) 2009.

Other Info


It's said that gcc -O4 emails your code to Jeff Dean for a rewrite.


3
  • 1
    Some presentations have different values indicated in the brackets. I assume the one's in the bracket were incorrect and he updated the values. Feb 22, 2011 at 12:56
  • 3
    Is this all pre-SSD era? see here for further up-to-date numbers.
    – matanox
    Jun 13, 2016 at 19:26
  • I actually used these numbers to build a presentation showing why SSD drives pay for themselves, to convince our office manager that yes we needed faster machines to work on. Included the numbers for the technical info but geared it towards non-tech management as much as possible.
    – brichins
    Apr 14, 2017 at 21:09
23

There are a lot of variables when it comes to network vs. disk, but in general, disk is faster.

The SATA 3.0 and SAS buses are 6 Gbps, vs. a networks 1Gbps minus protocol overhead. With RAID-10 15k SAS, the network is going to seem dog slow. In addition, you have disk cache and also the possibility of solid state harddrives, which depending on the scenario, could also increase speed. Random vs. Sequential data access plays a factor, as well as the block size in which data is being transferred. That all depends on the application that is being used to access the disk.

Now, I have not even touched on the fact that whatever you are transporting over the network is going to or coming from disk anyway...so.......again, disk is faster.

10
  • 1
    Points for mentioning RAID which gives you parallel reads, something which you are unlikely to get on a network anytime soon. Of course, if we are talking about local laptop hard drives, then the combo of fast SAN and fast network might well be faster. Especially with the SSDs in that SAN. Feb 22, 2011 at 1:47
  • 13
    Networks are inherently parallelizable -- what are you talking about? It's incredibly trivial to read from multiple systems on a network in aggregate; this is the entire point behind systems like Hadoop and MPI, not to mention the obvious BitTorrent. Feb 22, 2011 at 3:39
  • 2
    With SONET/SDH you can have 38Gbps still faster than SAS. And network aggregation can be done with something like en.wikipedia.org/wiki/Link_aggregation Feb 22, 2011 at 4:30
  • 11
    @Jake When talking about 6 Gbps, you might want to make a clear distinction between interface bandwidth and the rate at which a disk can actually supply data.
    – NPE
    Feb 22, 2011 at 9:28
  • 4
    i did say in my question that I was talking about a remote in memory database compared to a local on disk cache
    – pm100
    Feb 22, 2011 at 18:41
11

Well, that depends on whether the network resource has the data you are requesting readily available (in memory or similar) or if it would just, in turn, read it from a disk.

In any case, throughput may be higher in some cases but I believe latency will be higher.

3
  • You mean that seek time on the disk is greater than a 10Gbit/s request? Feb 22, 2011 at 2:02
  • 2
    @Mircea, he means that the 10Gbit network has to get it's data from somewhere, so it will be limited to the latency of that source, plus the latency of the network.
    – Chris S
    Feb 22, 2011 at 2:48
  • The storage could be a RAM disk. See: en.wikipedia.org/wiki/Solid-state_drive#DRAM-based Feb 22, 2011 at 4:25
3

The experience I have with this is that when you're on a 1Gbit connection and you try to download a file your harddisk is usually the bottleneck. A thing you have to keep in mind though is that you have to set up a connection first, which also takes time. So for sending big chunks of data network might actually be faster than disk.

2
  • 1
    Unless the disk is also the bottleneck on the other side of the network connection...
    – Argote
    Feb 21, 2011 at 23:51
  • @Argote: True, but if the server software was written correctly, it'll buffer into memory before writing to the disk. Feb 22, 2011 at 1:32
2

IMX the disk is still faster. The theoretical transfer rate of the network is high but in practice you don't get close to that.

About two years ago I had hard drive trouble on my laptop and the DMA went out. This made the hard drive dramatically slower, and in particular slower than network. But when I switched to another computer I was back to my original state of HDD faster than Internet.

2

My experience with gigabit networks is, given the right server, that you can beat local performance in terms of throughput and latency. See Network Tests: Are We Getting Gigabit Performance?

For all practical purposes I would recommend treating network & local storage as equivalent and only use memory caches.

Standard caveat as you mentioned is true in that there are no general rules; and that actually most of the time one should be working with well configured servers and be using metrics to evaluate the best method of data transfer.

If you are using a low end machine with a slow hard drive then it will almost certainly be quicker to use a gigabit network connection to a server with a fast storage array.

Equally if you are working with two machines of near identical hardware then the latency and network overhead would make local storage quicker; it's common sense really.

2

It depends. If your I/O is primarily random access then its flat throughput is probably not that great compared to the network bandwidth that could be available. However, most network traffic is ultimately generated by processes that involve I/O. If the working set of whatever process is generating the network traffic fits into cache then it won't be constrained by disk bandwidth. If it thrashes the cache then disk will become a bottleneck.

I work on data warehouse systems, and the canonical DW query is a table scan. If your query hits more than a few percent of the rows in the fact table (or partition) then a table or partition scan using sequential I/O will be more efficient than a random access query plan using index lookups and seeks.

Networked storage (i.e. SANs) tends not to perform well on streaming workloads unless it is tuned appropriately. If the SAN is being used for a general purpose consolidation environment it will almost certainly be tuned quite sub-optimally for a streaming, spiky load like a data warehouse. I have seen a vendor white paper suggest that you need about 3x the number of disks to get the same throughput on a SAN that is not tuned for streaming I/O as for one that is.

My experience tallies with that. In fact, I have never deployed a data warehouse onto a consolidation environment where I could not run the same ETL process significantly quicker on my desktop PC. I've also had sales reps from a major vendor of SAN equipment say off the record that a lot of their customers use direct attach storage for the DW system because SANs aren't fast enough.

Networked storage is at least an order of magnitude more expensive per IOPS than direct attach storage for random access workloads and closer to two orders of magnitude more expensive for streaming.

1

Yes, in general, networks are now get faster than hard-drives, but this may chnage over time.

I think, therefore I am

When an application is running it means the host machine is working, while working over network needs a common protocol, checking for peer availability, channel security... and if the peers use different platforms, it's harder to achieve what you can do on a single machine.

I prefer to look at this in the terms of trade-offs rather than who is the strongest...

1
  • 5
    I doubt, therefore I might be. Feb 22, 2011 at 2:49
1

You have to describe an exact use case for this comparison. Harddrives have seek time + transfer rate and cache. Networks have latency, transfer rate and protocol overhead...

I think your original cache memory>memory>disk>network still stands true in general though

0

The disk is connected with the CPU via SCSI, SAS or IDE bus. Which is a internal network running a specific protocol - SCSI or ATAPI. Ethernet is designed to work on longer distances and can be much slower than SAS/SCSI/IDE. So which one is faster, dependes on which technologies are you comparing. If you compare a 20 years old laptop HDD with a 10Gbps in RAM storage, the winner will always be the networking. And when you buy a storage you have to compare it versus price and manageability.

0

Well, there's Light Peak which is aiming for 100GBps networking speed, which is getting close to RAM speeds. Of course, the network can only deliver data as fast as the sender can generate the data, i.e. if the sender is reading the data from a hard disk then the receiver will only get the data at the same speed as the disk read, even with a superfast network.

0

One thing to keep in mind is that it depends on the network. Say, for example, you're responsible for performance on a web site. This web site is of course connected to a database server over a local network, and is also connected to web surfers over the internet, which is also a kind of network.

In many cases, a dedicated link may be set up between the web server and the database server via static IPs and a crossover cable or automdx to keep latency down and provide a dedicated link for the traffic, since you want it to be very fast. The database server does all kinds of work to keep as much of the db as possible in memory, and in many cases often succeeds for the entire contents plus a few indexes. Queries to this database are going to be just as fast or even faster than queries to disk.

On the other hand, certain web technologies (asp.net webforms viewstate, I'm looking at you) like to push a lot of information to and from the client web browser as a cache (of sorts). If this is a local lan connection (and in asp.net webform's defense this is true much of the time) it's not all that bad, but on the public internet this can absolutely kill performance, such that you are often much better off pushing this to a database or local disk instead.

0

Personally, I think there are several factors to consider. For instance, how fast is the memory or disk you are accessing locally vs. the one you would be accessing via network? If the remote data was on very fast SSD and faster than gigabit networking installed end to end, the remote might be faster for large streaming files.

However if you were randomly accessing small units of data and the network wasn't flawless or had many hops and more than just you accessing it, I'd bet that a local cache is faster even of its on a whirly mechanical disk drive almost 100% of the time. But you bring up an interesting point and how long will will need local storage of anything if network speeds continue to grow?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .