Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This is a software design question

I used to work on the following rule for speed

cache memory > memory > disk > network

With each step being 5-10 times the previous step (e.g. cache memory is 10 times faster than).

Now, it seems that gigabit ethernet has latency less than local disk. So, maybe operations to read out of a large remote in-memory DB are faster than local disk reads. This feels like heresy to an old timer like me. (I just spent some time building a local cache on disk to avoid having to do network round trips - hence my question)

Does anybody have any experience / numbers / advice in this area?

And yes I know that the only real way to find out is to build and measure, but I was wondering about the general rule.

edit:

This is the interesting data from the top answer:

  • Round trip within same datacenter 500,000 ns

  • Disk seek 10,000,000 ns

This is a shock for me; my mental model is that a network round trip is inherently slow. And its not - its 10x faster than a disk 'round trip'.

share|improve this question

migrated from stackoverflow.com Feb 22 '11 at 0:47

This question came from our site for professional and enthusiast programmers.

9  
Sometimes yes, sometimes no. What network? What disk? –  John Gardeniers Feb 22 '11 at 1:46
1  
The other interesting data from top answer: 1 MB sequential read from network vs. disk. I suspect the "round trip" time omits any significant data transfer. –  Paul Feb 22 '11 at 20:48
    
Paul: Depends on your MTU, I'm sure. (1MB MTU? Awesome!) –  Matt Simmons Feb 22 '11 at 22:01
add comment

13 Answers 13

up vote 64 down vote accepted

Here are some numbers that you are probably looking for, as quoted by Jeff Dean, a Google Fellow:

Numbers Everyone Should Know

L1 cache reference                             0.5 ns
Branch mispredict                              5 ns
L2 cache reference                             7 ns
Mutex lock/unlock                            100 ns (25)
Main memory reference                        100 ns
Compress 1K bytes with Zippy              10,000 ns (3,000)
Send 2K bytes over 1 Gbps network         20,000 ns
Read 1 MB sequentially from memory       250,000 ns
Round trip within same datacenter        500,000 ns
Disk seek                             10,000,000 ns
Read 1 MB sequentially from network   10,000,000 ns
Read 1 MB sequentially from disk      30,000,000 ns (20,000,000)
Send packet CA->Netherlands->CA      150,000,000 ns

It's from his presentation titled Designs, Lessons and Advice from Building Large Distributed Systems and you can get it here:

The talk was given at Large-Scale Distributed Systems and Middleware (LADIS) 2009.

Other Info


It's said that gcc -O4 emails your code to Jeff Dean for a rewrite.


share|improve this answer
    
+1 Very interesting! –  9dan Feb 22 '11 at 9:35
    
Some presentations have different values indicated in the brackets. I assume the one's in the bracket were incorrect and he updated the values. –  David d C e Freitas Feb 22 '11 at 12:56
add comment

There are a lot of variables when it comes to network vs. disk, but in general, disk is faster.

The SATA 3.0 and SAS buses are 6 Gbps, vs. a networks 1Gbps minus protocol overhead. With RAID-10 15k SAS, the network is going to seem dog slow. In addition, you have disk cache and also the possibility of solid state harddrives, which depending on the scenario, could also increase speed. Random vs. Sequential data access plays a factor, as well as the block size in which data is being transferred. That all depends on the application that is being used to access the disk.

Now, I have not even touched on the fact that whatever you are transporting over the network is going to or coming from disk anyway...so.......again, disk is faster.

share|improve this answer
1  
Points for mentioning RAID which gives you parallel reads, something which you are unlikely to get on a network anytime soon. Of course, if we are talking about local laptop hard drives, then the combo of fast SAN and fast network might well be faster. Especially with the SSDs in that SAN. –  Michael Dillon Feb 22 '11 at 1:47
7  
Networks are inherently parallelizable -- what are you talking about? It's incredibly trivial to read from multiple systems on a network in aggregate; this is the entire point behind systems like Hadoop and MPI, not to mention the obvious BitTorrent. –  jgoldschrafe Feb 22 '11 at 3:39
2  
With SONET/SDH you can have 38Gbps still faster than SAS. And network aggregation can be done with something like en.wikipedia.org/wiki/Link_aggregation –  Mircea Vutcovici Feb 22 '11 at 4:30
4  
@Jake When talking about 6 Gbps, you might want to make a clear distinction between interface bandwidth and the rate at which a disk can actually supply data. –  NPE Feb 22 '11 at 9:28
3  
i did say in my question that I was talking about a remote in memory database compared to a local on disk cache –  pm100 Feb 22 '11 at 18:41
show 5 more comments

Well, that depends on whether the network resource has the data you are requesting readily available (in memory or similar) or if it would just, in turn, read it from a disk.

In any case, throughput may be higher in some cases but I believe latency will be higher.

share|improve this answer
    
You mean that seek time on the disk is greater than a 10Gbit/s request? –  Mircea Vutcovici Feb 22 '11 at 2:02
1  
@Mircea, he means that the 10Gbit network has to get it's data from somewhere, so it will be limited to the latency of that source, plus the latency of the network. –  Chris S Feb 22 '11 at 2:48
    
The storage could be a RAM disk. See: en.wikipedia.org/wiki/Solid-state_drive#DRAM-based –  Mircea Vutcovici Feb 22 '11 at 4:25
add comment

IMX the disk is still faster. The theoretical transfer rate of the network is high but in practice you don't get close to that.

About two years ago I had hard drive trouble on my laptop and the DMA went out. This made the hard drive dramatically slower, and in particular slower than network. But when I switched to another computer I was back to my original state of HDD faster than Internet.

share|improve this answer
add comment

My experience with gigabit networks is, given the right server, that you can beat local performance in terms of throughput and latency. See Network Tests: Are We Getting Gigabit Performance?

For all practical purposes I would recommend treating network & local storage as equivalent and only use memory caches.

Standard caveat as you mentioned is true in that there are no general rules; and that actually most of the time one should be working with well configured servers and be using metrics to evaluate the best method of data transfer.

If you are using a low end machine with a slow hard drive then it will almost certainly be quicker to use a gigabit network connection to a server with a fast storage array.

Equally if you are working with two machines of near identical hardware then the latency and network overhead would make local storage quicker; it's common sense really.

share|improve this answer
add comment

The experience I have with this is that when you're on a 1Gbit connection and you try to download a file your harddisk is usually the bottleneck. A thing you have to keep in mind though is that you have to set up a connection first, which also takes time. So for sending big chunks of data network might actually be faster than disk.

share|improve this answer
1  
Unless the disk is also the bottleneck on the other side of the network connection... –  Argote Feb 21 '11 at 23:51
    
@Argote: True, but if the server software was written correctly, it'll buffer into memory before writing to the disk. –  amphetamachine Feb 22 '11 at 1:32
add comment

Yes, in general, networks are now get faster than hard-drives, but this may chnage over time.

I think, therefore I am

When an application is running it means the host machine is working, while working over network needs a common protocol, checking for peer availability, channel security... and if the peers use different platforms, it's harder to achieve what you can do on a single machine.

I prefer to look at this in the terms of trade-offs rather than who is the strongest...

share|improve this answer
3  
I doubt, therefore I might be. –  John Gardeniers Feb 22 '11 at 2:49
add comment

You have to describe an exact use case for this comparison. Harddrives have seek time + transfer rate and cache. Networks have latency, transfer rate and protocol overhead...

I think your original cache memory>memory>disk>network still stands true in general though

share|improve this answer
add comment

It depends. If your I/O is primarily random access then its flat throughput is probably not that great compared to the network bandwidth that could be available. However, most network traffic is ultimately generated by processes that involve I/O. If the working set of whatever process is generating the network traffic fits into cache then it won't be constrained by disk bandwidth. If it thrashes the cache then disk will become a bottleneck.

I work on data warehouse systems, and the canonical DW query is a table scan. If your query hits more than a few percent of the rows in the fact table (or partition) then a table or partition scan using sequential I/O will be more efficient than a random access query plan using index lookups and seeks.

Networked storage (i.e. SANs) tends not to perform well on streaming workloads unless it is tuned appropriately. If the SAN is being used for a general purpose consolidation environment it will almost certainly be tuned quite sub-optimally for a streaming, spiky load like a data warehouse. I have seen a vendor white paper suggest that you need about 3x the number of disks to get the same throughput on a SAN that is not tuned for streaming I/O as for one that is.

My experience tallies with that. In fact, I have never deployed a data warehouse onto a consolidation environment where I could not run the same ETL process significantly quicker on my desktop PC. I've also had sales reps from a major vendor of SAN equipment say off the record that a lot of their customers use direct attach storage for the DW system because SANs aren't fast enough.

Networked storage is at least an order of magnitude more expensive per IOPS than direct attach storage for random access workloads and closer to two orders of magnitude more expensive for streaming.

share|improve this answer
add comment

The disk is connected with the CPU via SCSI, SAS or IDE bus. Which is a internal network running a specific protocol - SCSI or ATAPI. Ethernet is designed to work on longer distances and can be much slower than SAS/SCSI/IDE. So which one is faster, dependes on which technologies are you comparing. If you compare a 20 years old laptop HDD with a 10Gbps in RAM storage, the winner will always be the networking. And when you buy a storage you have to compare it versus price and manageability.

share|improve this answer
add comment

Well, there's Light Peak which is aiming for 100GBps networking speed, which is getting close to RAM speeds. Of course, the network can only deliver data as fast as the sender can generate the data, i.e. if the sender is reading the data from a hard disk then the receiver will only get the data at the same speed as the disk read, even with a superfast network.

share|improve this answer
add comment

One thing to keep in mind is that it depends on the network. Say, for example, you're responsible for performance on a web site. This web site is of course connected to a database server over a local network, and is also connected to web surfers over the internet, which is also a kind of network.

In many cases, a dedicated link may be set up between the web server and the database server via static IPs and a crossover cable or automdx to keep latency down and provide a dedicated link for the traffic, since you want it to be very fast. The database server does all kinds of work to keep as much of the db as possible in memory, and in many cases often succeeds for the entire contents plus a few indexes. Queries to this database are going to be just as fast or even faster than queries to disk.

On the other hand, certain web technologies (asp.net webforms viewstate, I'm looking at you) like to push a lot of information to and from the client web browser as a cache (of sorts). If this is a local lan connection (and in asp.net webform's defense this is true much of the time) it's not all that bad, but on the public internet this can absolutely kill performance, such that you are often much better off pushing this to a database or local disk instead.

share|improve this answer
add comment

Personally, I think there are several factors to consider. For instance, how fast is the memory or disk you are accessing locally vs. the one you would be accessing via network? If the remote data was on very fast SSD and faster than gigabit networking installed end to end, the remote might be faster for large streaming files.

However if you were randomly accessing small units of data and the network wasn't flawless or had many hops and more than just you accessing it, I'd bet that a local cache is faster even of its on a whirly mechanical disk drive almost 100% of the time. But you bring up an interesting point and how long will will need local storage of anything if network speeds continue to grow?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.