Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Problem: I need to copy a large block of data from a remote location into system memory as quickly as possible.

Scenario: I have a data processing system. The system is built via shell scripts on-the-fly using multiple components that are pulled in from remote locations.

One of those components is a large block of data stored as groups of files.

The requirement I have is to retrieve that large block of data from a remote location and install it into system memory as quickly as possible. This is a requirement so that the system which relies on this data can start using it for processing as soon after boot time as possible.


Question: "What would be the most efficient container for my data?"


Solutions already tried/considered:

  • ISO file: requires tools for creation and reading that are not typically native
  • TAR file: extracting can take a lot of time
  • Remote filesystem mounted as local: slow because contents need to be copied into memory
  • LVM snapshot: gear more toward backups, not built for speed on restore

Notes:

  • Data loss is not a primary concern.
  • The remote file transfer procedure is not a primary concern as I already have an adequate tool.
  • The system is currently using Ubuntu Linux.
share|improve this question
2  
Have you tried ZFS send/receive? –  EEAA Jun 3 '13 at 22:21
    
I have not, although I was just looking at ZFS in general. I will take a look at send/receive specifically. –  SunSparc Jun 3 '13 at 22:26
1  
ZFS seems to be a great budding filesystem with some nice features. For this project ZFS is not a good fit, at least at this time. Furthermore, I did not see any indication of ZFS providing overly efficient data transfer containers. –  SunSparc Jun 6 '13 at 20:43
1  
ZFS is hardly budding. btrfs? That's a budding filesystem. ZFS has been around for quite some time and is very stable. Anyway, it's hard to get more efficient than ZFS send/receive - it transfers changed bytes only, and compressed if you elect. –  EEAA Jun 6 '13 at 21:14
1  
Sorry, by budding I simply meant that it does not seem to yet be as ubiquitous as other filesystems. But that is beside the point. The transfer I am working on will not be a comparative transfer but an initial transfer, each time. Thus there would be no "speed-up" benefits due to comparison. So many transfer systems and containers seem to be geared toward backups, which is not what I am working on. –  SunSparc Jun 6 '13 at 21:40

3 Answers 3

"The remote file transfer procedure is not a primary concern as I already have an adequate tool."

If you already have the file transferred, I suggest using mmap(2).

share|improve this answer
    
This answer is intriguing to me. I am looking into whether this will work for my situation. –  SunSparc Jun 3 '13 at 22:31
    
After having reviewed mmap it seems that it is quite a bit more low level and surgical than I am looking for. I already have a memory location to put the data via mount tmpfs. I am more looking for a container that is quick to unload into the tmpfs. So far, snapshots seem like they may be the best option. –  SunSparc Jun 4 '13 at 17:28

You should consider an image file with a file system that contains your data (put a loop device over the file with losetup and mount the loop device). The fastest way would probably be a compresed read-only file system like squashfs.

This would even allow some tricks if not all the data is needed simultaneously. Instead of mounting the loop device you could put a DM device on top of it, mount a network file system (or network block device) with the image file, put a second loop device on top of the network version of the file and combine both loop devices with the DM device.

Let's assume you have to copy 500 MiB of data. You start copying it. As soon as the first 100 MiB have been transferred you create the loop devices and the DM device. The DM device points to the loop device of the local file for the first 100 MiB and to the other one for the rest. After e.g. each transferred 10 MiB block you suspend the DM device and reload it with the border shifted by another 10 MiB.

The risk is: If accesses go to the network version then that data is transferred twice. So if that happens often then the data transfer will take longer (the whole process may finish earlier though, depending on its access characteristics).

Edit 1:

See this answer of me to another question for an explanation how to use DM devices this way (without suspend/reload/resume though).

share|improve this answer
    
The answer is not easy for me to quickly grok. I will have to come back to it. –  SunSparc Jun 3 '13 at 22:35
    
The image file (ISO) is what we have decided to go with at this point. While usage of partial data as it comes in is a nice idea, in theory, for this particular system, in order to begin processing, the entire block of data is required. –  SunSparc Jun 6 '13 at 20:41
up vote 0 down vote accepted

My initial research into the ISO container was apparently incomplete. The ISO container seems to be the most efficient, for purpose of being able to quickly get to the contents. This is based on what my research has been able to uncover, and could of course change.

Packaged in an ISO I am able to:

  • store the data remotely
  • retrieve it very quickly via multipart transfer
  • store it locally, directly into memory
  • mount it quickly

Using this container I have been able to get the entire process down to under 1 minute, which is an acceptable tolerance level for this project.

Creating this container is done easily in Ubuntu with a command similar to the following:

mkisofs -o /tmp/data.iso /opt/data/

**Note that this requires genisoimage which is easily installed via apt-get.

To store the file directly into memory I created a ramdisk in the /tmp filesystem:

mount -t tmpfs -o size=3G tmpfs /tmp/data

Retrieving the container can be done relatively quickly with a multipart transfer utility. I used one called axel in this manner:

axel -a -n 128 -o /tmp/data/data.iso https://s3.amazonaws.com/bucket/data.iso

Finally we mount the file to a local filesystem:

mount -o loop -r /tmp/data/data.iso /opt/data/

The mounting process is nearly instantaneous, which allows the system to quickly begin using the data for processing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.