2

I am currently in charge of a server running Red Hat and a bioinformatics webapp that deals with huge files, some of which are over 100GB when uncompressed. The act of decompressing these files is done by several different programs, all of which use the system temporary directory, /tmp. When a huge file is being decompressed, /tmp fills up and halts the operation in progress, causing downstream errors in the webapp. I have to go in and delete the problem file from /tmp.

The server's filesystem is setup like this:

/*output of df-h follows*/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_root-LogVol01 83G   34G   45G  43% /
tmpfs                  95G     0   95G   0% /dev/shm
/dev/sda1             485M   81M  379M  18% /boot
/dev/sdb1             8.0T  6.2T  1.5T  82% /data

/*output of blkid follows*/
/dev/sda1: UUID="5f489589-0678-46c1-9f0f-e4e66c6a9e04" TYPE="ext4" 

/dev/sda2: UUID="k2rP0k-1YhK-72D9-fRrQ-BxUW-3gaC-6KF0nh" TYPE="LVM2_member" 

/dev/sdb1: UUID="4e54bee3-6450-446f-af80-70ca6268e12f" TYPE="ext4" 

/dev/mapper/vg_root-LogVol01: UUID="b6f228dc-fa6c-43c1-b88e-3b43a75e980b" TYPE="ext4" 

/dev/mapper/vg_root-LogVol00: UUID="80c6863d-c9f8-4abe-a353-a6a6818dc82d" TYPE="swap"

/*output of fstab*/
/dev/mapper/vg_root-LogVol01 /   ext4    defaults   1 1
UUID=5f489589-0678-46c1-9f0f-e4e66c6a9e04 /boot   ext4   defaults   1 2
/dev/mapper/vg_root-LogVol00 swap    swap    defaults    0 0
/dev/sdb1   /data   ext4    defaults    1 1
tmpfs        /dev/shm   tmpfs   defaults        0 0
devpts      /dev/pts    devpts  gid=5,mode=620  0 0
sysfs       /sys        sysfs   defaults        0 0
proc        /proc       proc    defaults        0 0

I know the basics of Linux, but I don't know much about filesystems. What I would like to do is enact a permanent solution to allocate more space to the temporary directory. I would be open to setting the TMPDIR environment variable to point to another filesystem with more space. Or, if it's possible, I'd be equally satisfied allocating more of the existing space to LogVol-01.

My question is how, given my current filesystem details, to permanently allocate more space to the temporary directory. As you can see by the df output, I have disk space to spare. I didn't set the server up, but I am now in charge of it and have root access, for better or worse!

7
  • 1
    The df -v is missing the right-most column. Can you show /etc/fstab?
    – cjc
    Mar 13, 2012 at 22:11
  • Thanks for the answers, folks! They have been very helpful. I will post the output of fstab as soon as I get into work tomorrow.
    – DeeDee
    Mar 13, 2012 at 22:38
  • To everyone having trouble with the mount points, the "Mounted on" field in the df output is being shown on the next line (probably copied from a terminal that was too small). The first line is the / mount point. The second line is tmpfs not /tmpfs (the / that should have been on the previous line also had the space after it removed) and is mounted at /dev/shm, etc. Unfortunately I cant edit as serverfault doesnt let you fix whitespace issues.
    – phemmer
    Mar 14, 2012 at 0:03
  • @Matt @swasheck @cjc @Patrick OK, here are the results of /etc/fstab: code /dev/mapper/vg_root-LogVol01 / ext4 defaults 1 1 UUID=5f489589-0678-46c1-9f0f-e4e66c6a9e04 /boot ext4 defaults 1 2 /dev/mapper/vg_root-LogVol00 swap swap defaults 0 0 /dev/sdb1 /data ext4 defaults 1 1 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0
    – DeeDee
    Mar 14, 2012 at 13:32
  • Yes, @Patrick, good catch on the df -v output. I edited my post to fix that issue. Thanks for that!
    – DeeDee
    Mar 14, 2012 at 13:48

2 Answers 2

4

Update:

Since you've now confirmed that /tmp is not a symbolic link and that there are no mounts at /tmp. It's basically saying that / is filling up. How do you fix that? 2 options.

  1. I'd insert another hard disk and mount it at /tmp by inserting an appropriate entry in /etc/fstab.
  2. Or as a temporary measure move /tmp to another drive like /dev/sdb1 which currently has hundreds of GB free. It's simply a matter of creating a tmp directory on /dev/sdb1

i.e.

sudo mkdir /data/tmp
sudo chmod 1777 /data/tmp
sudo rm /tmp
sudo ln -s /data/tmp /tmp

If you'd prefer to add more storage (which is a better long term solution), there are lots of tutorials on the net on how to go about adding extra drives to Linux.

How to add an extra hard drive to Linux


Original answer...

tmpfs could be the culprit. It's ram based and it'll use swap when it needs to as well. That means it'll keep growing and growing until no more ram can be used for tmpfs. You're ending up with an incomplete decompress.

What you need to do then is edit /etc/fstab and comment out (with a leading '#') the line beginning with '/tmp' and reboot or unmount /tmp (with 'sudo umount /tmp')

Looks like you've got the whole drive using LVM so it's safe to not mount a replacement for /tmp.

Edit:

Wish I could see a default fstab file from redhat, or yours. Anyhow... It's possible that /tmp doesn't exist in fstab because it could be a symbolic link.
To find out run "ls -ld /tmp". If it's a symbolic link you'll see it listed with /tmp -> /dev/shm or something.

If it is actually a link to /dev/shm or to a directory that is mounted as tmpfs, Fix that with:

sudo rm /tmp
sudo mkdir /tmp
sudo chmod 1777 /tmp

EDIT2: On second thought, it could other reasons.

  1. Either /tmp is filling up, or
  2. The decompression tools aren't working too well on your large archives. 100GB uncompressed data is probably a large archive! What tools are you using to extract the archive. What's it's format? I know some versions of tar have issues, or
  3. The filesystem type does not allow for some files to to be extracted because they exceed the filesystem limit. What type is the target filesystem? is it ext3 or 4 or something else?
  4. Or the filesystem is really just full.
7
  • I think tmpfs is mounted as /dev/shm and not as /tmp. It's hard to tell unless the OP shows us /etc/fstab.
    – cjc
    Mar 13, 2012 at 22:26
  • 1
    /dev/shm is still shared memory though.
    – swasheck
    Mar 13, 2012 at 22:27
  • @swasheck whats that got to do with anything? He's not extracting to /dev/shm
    – phemmer
    Mar 13, 2012 at 23:51
  • and tmpfs is just the filesystem type. it's possible /tmp is a symbolic link to /dev/shm which is mounted as tmpfs though.
    – hookenz
    Mar 13, 2012 at 23:54
  • @Patrick if /tmp is mounted on/linked to /dev/shm then it is extremely relevant. Sorry that my comment was so vague. End of a busy day.
    – swasheck
    Mar 14, 2012 at 14:25
2

I'm pretty sure that /tmp and tmpfs are RAM-based, so decompressing them and storing them in RAM may be the reason that things grind to a halt. The OS starts swapping like crazy so you begin to really notice the performance hit.

Must this operation occur in /tmp?

Adding an actual answer:

http://wiki.centos.org/TipsAndTricks/ExpandLV

6
  • I think tmpfs is mounted in /dev/shm, actually. It's hard to tell from the OP's copy and paste of the df output. It's not clear how /dev/sdb is mounted, also.
    – cjc
    Mar 13, 2012 at 22:13
  • 1
    you're absolutely correct
    – swasheck
    Mar 13, 2012 at 22:15
  • And I believe that tmpfs can also use swap under low memory.
    – hookenz
    Mar 13, 2012 at 22:17
  • But swapping would still cause a performance hit, right?
    – swasheck
    Mar 13, 2012 at 22:20
  • Yes, it would. Though, I think the OP is saying that the compute processes are actually crashing due to a lack of space rather than just performing slowly.
    – cjc
    Mar 13, 2012 at 22:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .