9

On a Linux system, is there any way to concatenate a series of files into one exposed file for reading and writing while not actually taking up another N bytes of disk space? I was hoping for something like mounting these files via loopback/devmapper to accomplish this.

I have a problem where there are split binary files that can get quite large. I don't want to double my space requirements with massive disk IO just to temporarily read / write contents from these files by cating them all together into one enormous file.

I found this project here, but it seems to have a very specific use case and also depends on perl

  • I guess FUSE was invented to solve this kind of problem. Probably you have to write your own module but that should not be too difficult. – Hauke Laging Mar 14 '13 at 5:09
13

You can do that by combining loop devices and device mapper, thanks to "everything is a file". But with one limitation: The file size cannot be changed (as you cannot increase block devices by writing to them). So if you want to append data then you have to create a bigger block device and overwrite the appended dummy data.

# for testing, Create 4 files
echo foo | dd of=block0 cbs=512 conv=block
echo bar | dd of=block1 cbs=512 conv=block
echo bat | dd of=block2 cbs=512 conv=block
echo baz | dd of=block3 cbs=512 conv=block
# Create a loop device for each of the 4 files
losetup /dev/loop0 block0
losetup /dev/loop1 block1
losetup /dev/loop2 block2
losetup /dev/loop3 block3
# Create a device map named "test" using those loop devices
(
    echo "0 1 linear /dev/loop0 0"
    echo "1 1 linear /dev/loop1 0"
    echo "2 1 linear /dev/loop2 0"
    echo "3 1 linear /dev/loop3 0"
) | dmsetup create test
$EDITOR /dev/mapper/test # use overwrite mode only

For extending the file you may create a big sparse file and use that as additional device.

1

This could be done using named pipe in Linux.

Assume you have files named file0, file1, file2, file3, file4, file5

# create a name pipe
$ mkfifo mynewfile
# cat to named file
$ cat file{0..5} > mynewfile &

In C programatically you can do

mkfifo(mynewfile , 0777);
system("cat file{0..5} > mynewfile");

Then use then use mynewfile as us you would read from normal file.

mynewfile is FIFO file

0

For reading, you can less multiple files and then use the :n and :p options to go through them.

For writing, without accessing the files directly you will not be able to write to them.

You can vim multiple files as well, and it will just go through the order they were called (ie. vim fileA fileB fileC - fileB opened after fileA closed, fileC opened after fileB closed).

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.