Problem: I need to copy a large block of data from a remote location into system memory as quickly as possible.
Scenario: I have a data processing system. The system is built via shell scripts on-the-fly using multiple components that are pulled in from remote locations.
One of those components is a large block of data stored as groups of files.
The requirement I have is to retrieve that large block of data from a remote location and install it into system memory as quickly as possible. This is a requirement so that the system which relies on this data can start using it for processing as soon after boot time as possible.
Question: "What would be the most efficient container for my data?"
Solutions already tried/considered:
- ISO file: requires tools for creation and reading that are not typically native
- TAR file: extracting can take a lot of time
- Remote filesystem mounted as local: slow because contents need to be copied into memory
- LVM snapshot: gear more toward backups, not built for speed on restore
- Data loss is not a primary concern.
- The remote file transfer procedure is not a primary concern as I already have an adequate tool.
- The system is currently using Ubuntu Linux.