Creating swap files faster

Question

I'm using Amazon EC2 and wish to be able to quickly generate large swapfiles (~10+GB) on instance startup. Unfortunately, I/O speed on my instances (c1.xlarge) is slow enough (20 MB/s) that this operation takes 10+ minutes, which is unacceptable for my usage.

I am aware that the swapfiles must be pre-alocated to use, so that I can't use sparse files.

However, is there some command to allocate blocks w/out spending the large amount of time zero-ing out the blocks? Also, if this command exists, am I correct in assuming that a page in the swapfile is zero'd out before a user process has access to it (mitigating security concerns)?

Answer 1

You didn't indicate what method you were trying to avoid.

Traditionally, you would issue a dd command that would in turn pump out a zero'd file of the appropriate size, then run mkswap, add the entry to /etc/fstab, and then swapon to activate it. I've attached a rather hastily-written example shell script that I'm sure has errors (it's late where I'm at and the fstab entry is far from perfect)

#!/bin/bash
# --- allocate 10Gbyte of swap space as 10 separate 1Gbyte files
# --- that are brought online sequentially during processing
for swpidx in 01 02 03 04 05 06 07 08 09 10
do
  dd if=/dev/zero of=/swapfile.$swpidx bs=16738 count=65536
  mkswap /swapfile.$swpidx
  echo "/swapfile.$swpidx    swap    swap    default    0 0" >> /etc/fstab
  swapon -a
done
swapon -s

However, it sounds like you are trying to avoid this method. The fastest solution that I could provide would be to use a swap partition, which does not require the zero-out process, and can be brought online in minutes. If your instance is running LVM and you have an existing volume group that you could carve a partition out of, that would work just as well, and the allocation could be completed in just a few minutes.

I think I should mention that carving out a swap space of this size is a bit unusual, even for a server; and I only say that because most servers have several gigs of RAM attached when dealing with programs/data of that size. Not to pry or anything, but are you really needing that much swap space?

Another thing you may wish to consider is re-tuning your workload, rather than trying to dynamically allocate swap space. While it's great to have that much "on-demand", as you yourself pointed out, it will quickly become a bottleneck due to the slow I/O throughput on your server instance. By the time you exhaust your memory and you're essentially "living in swap", you'll find that the 20Mbyte/sec transfer rate turns your instance into a 386SX.