Create a block device in RAM

Question

One of my day-to-day tasks is to create a bootable image of a USB pendrive with our software on it. The image must consist of two partitions, first one formatted with FAT32 and second with EXT3. On the first we put Linux kernel and a small startup script that the bootloader on the device we are going to boot with reads on startup.

On the second partition we put our rootfs.

Now, here's how I do it:

1. insert a USB pendrive into my Linux comp. Figure out its device file (here let's assume /dev/sdx)

2. zero out the first 600 MB of it: dd if=/dev/zero of=/dev/sdx bs=1M count=600

3. Create the two partitions, first 10 MB and second 500 MB: sfdisk /dev/sdx --unit S << EOF ,20000,c ,1000000,83 EOF

   mkfs.vfat /dev/sdx1 mkfs.ext3 /dev/sdx2

(that's assuming sector size is 512 bytes!)

4. copy the kernel and startup script to /dev/sdx1
5. Unpack pre-compiled RootFS to /dev/sdx2
6. Copy the ready-made image from USB pendrive back to our HDD: dd if=/dev/sdx of=usb.img bs=1M count=520
7. Compress it and send to the people who test this software.
8. the people then uncompress, flash their USB disks with dd if=usb.img of=/dev/sdx insert into their devices and test.

Now, all of this works. The obvious improvement is to create a script that would do this all automatically. This can also be done, but one manual step will still remain: insert the USB pendrive.

I am thinking instead of the actual USB disk we could use a block device in RAM. This not only would eliminate the manual step of inserting the USB disk, but would also be faster.

So what I need is a plain block device in RAM. One that can be split into partitions just like a physical USB pendrive can.

I just researched this topic and seems like the only options are

1. tmpfs and ramfs, which obviously don't suit me
2. ramdisk and the 'brd' kernel module, which, when loaded, creates a number of 'partitions': /dev/ram[0-15]

and each of them can individually be formatted and used, but like I said - I want a plain block device in RAM which I would then split into two partitions of 10MB and 500MB myself, and from where I would be then able to dd an image to a file on my HDD.

What are my options?

Answer 1

Just use brd and create one brd device (ram0). Use that device in place of your usb drive. You can partition it using sfdisk, use the partitions and then use dd to dump its contents to file.

There is no need to use one filesystem per brd device.

Or (though little hacky) you can use tmpfs, create image file and use that as loop device. That might be the easiest way to accomplish what you want. As a bonus, you have that image ready and can upload it straight away. No need to dd.

# Create mountpoint for tmpfs
mkdir /tmp/tmpfs
# Mount tmpfs there
mount -t tmpfs none /tmp/tmpfs
# Create empty file of 600MB 
# (it creates 599MB hole, so it does not 
#  consume more memory than needed)
dd if=/dev/zero of=/tmp/tmpfs/img.bin bs=1M seek=599 count=1
# Partition the image file
cfdisk /tmp/tmpfs/img.bin 
# Create loop block device of it (-P makes kernel look for partitions)
losetup -P /dev/loop0 /tmp/tmpfs/img.bin 
# Create filesystems
mkfs.vfat /dev/loop0p1 
mkfs.ext4 /dev/loop0p2
# Now it's your turn:
#   mount loop0p1 and loop0p2 and copy whatever you want and unmount it
# detach the loop device
losetup -d /dev/loop0
# May i present you with your image ... 
ls -al /tmp/tmpfs/img.bin

Modify to suit your needs.

Answer 2

This is an analog to Fox's answer, but ram disk is used instead of tmpfs. In my tests with vdbench, ram disk shows higher IOPS than tmpfs (33k IOPS vs 29k IOPS on my machine in three consecutive tests, i.e. 6 tests overall), and it is easier to setup too.

# the following creates 1 device /dev/ram0 of size 600M. More elaborate parameters
# description you can see with `modinfo brd | grep parm`, ATM there's just three.
modprobe brd rd_nr=1 rd_size=614400
# Partition the image file. The following creates two GPT partitions: one of size
# 300M, and the other one takes all space that's left
sgdisk -Z --new 1::+300M --new 2 /dev/ram0
# Create filesystems
mkfs.vfat /dev/ram0p1
mkfs.ext4 /dev/ram0p2
# Now you can mount ram0p1 and ram0p2 as you would with a usual block device and copy
# your stuff there. After you `umount` it, you get a partitioned `/dev/ram0` block
# device with your files. You can make an image of it with `dd if=/dev/ram0 of=img.bin`
# for example. After you're done, to remove the ram device execute `rmmod brd`.

Answer 3

nbdkit-memory-plugin can be used for a userspace implementation:

$ nbdkit -U /tmp/sock memory 10G
# nbd-client -unix /tmp/sock /dev/nbd0

This is implementated as a sparse array so it can be any virtual size. Add allocator=zstd to do compression.