Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have some filenames (session files) that can't be written ("No space left on device"). Other filenames (same directory) are fine. Disk is not full. The filesystem is ext3

PHP is giving an error when trying to create the files, but the error can be reproduced on the command line:

# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory
# touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
touch: cannot touch `/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1': No space left on device
# touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0
# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0
# ls -al /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
ls: /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory

Note the different filenames. There doesn't seem to be any pattern, but it does affect certain filenames only, and they don't seem to be ever able to be written to.

To possibly complicate things further, this is an OpenVZ server, but you can't write to/create those filenames from either the virtual server or the hardware node.

There are a lot of files in that directory (18,456,002 at the time of writing), but there is no inode problem, and the disk is definitely not full.

As requested, df outputs:

[root@web1 session_data]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/simfs            2.7T  2.1T  495G  81% /
/dev/root             2.7T  2.1T  495G  81% /path/to/session_data
[root@web1 session_data]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/simfs           726761472 22843560 703917912    4% /
/dev/root            726761472 22843560 703917912    4% /path/to/session_data

(I've edited the path to the session data, but it really is mounted separately like that, to enable sharing of session data between multiple virtual servers)

share|improve this question
2  
session.save_path from php.ini, df -h and df -i, please. –  GioMac Sep 17 '13 at 12:24
    
Did you mean "there is no inode problem" –  jftuga Sep 17 '13 at 12:33
    
yes, sorry - there is no inode problem –  Rob Ferrer Sep 17 '13 at 12:41
    
I've added the DF output. I hope I'm replying to the right comment - This is my first time on serverfault! –  Rob Ferrer Sep 17 '13 at 12:46
    
Welcome to the serverfault. Looks like php is not cleaning up sessions. What is your filesystem? –  GioMac Sep 17 '13 at 12:59

1 Answer 1

Ext3 does hashing on file/dir-names.
With so many many files in one directory I wouldn't be surprised if you are running in hash-collision issues.
Having all the filenames of the same length and have the same structure makes the likelihood of a collision even worse.

If there is a collision the filename is effectively unusable, as you noticed.

I have no further experience with this sort of thing so I can't give you any more detailed advice.
(We ran into a similar problem in the office once and the resident Linux guru used some tunefs magic to make the problem go away. I never got the exact details what he did. Just that it had to do with hash-collisions on the filenames.)

You can apparently set the hashing behavior via tunefs. There is also a ext3 extension called "dirindex" that really helps dealing with directories containing so many files.

share|improve this answer
    
Thanks, it's good to hear the hash-collision theory from another source. I'd love to hear something definite from someone, but in the meantime we'll look at reducing the number of files in the directory (we found our garbage collection wasn't functioning as it should have been, which is a start). –  Rob Ferrer Sep 17 '13 at 15:10
    
I hadn't seen your comment about the hashing when I wrote my answer. I can get more info from our Linux guy, but he is on holiday at the moment. Another 2 weeks before he is back. As your are debugging: Maybe change the paths to /sessiondata/<last 4 chars of filename>/<filename> or something like that. Is usually rather trivial to implement and greatly reduces the amount of files per directory (at the expense of more directories and a slight overhead on generating the paths). Of course that is assuming you don't need to re-generate those paths in 500 different places. –  Tonny Sep 17 '13 at 18:04
    
I've thought about that, but that would mean losing the existing session files, which means customers losing their baskets (which they complain about). We found a problem in our garbage collection which we've now fixed which should drastically reduce the number of files. Hopefully that should fix it for now, until we get a chance to overhaul the session handling. –  Rob Ferrer Sep 17 '13 at 19:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.