Tag Info

Hot answers tagged

30

# tune2fs -l /dev/sda1 | grep -i 'block size' Block size: 1024 Replace /dev/sda1 with the partition you want to check.


25

The -C flag will display a progress bar. Performance differences depending on how fsck is called. And very cool, if e2fsck is already running, you can send a USR1 signal for it to start displaying a progress bar. USR2 to stop. Example: killall -USR1 e2fsck From FSCK(8): -C Display completion/progress bars for those filesys- tems ...


24

Would it be possible to backup all of the other files from this file system to a temporary storage location, reformat the partition, and then restore the files?


24

Check for files on located under mount points. Frequently if you mount a directory (say a sambafs) onto a filesystem that already had a file or directories under it, you lose the ability to see those files, but they're still consuming space on the underlying disk. I've had file copies while in single user mode dump files into directories that I couldn't ...


20

Whilst a major cause of this problem is ext3 performance with millions of files, the actual root cause of this problem is different. When a directory needs to be listed readdir() is called on the directory which yields a list of files. readdir is a posix call, but the real linux system call being used here is called 'getdents'. Getdents list directory ...


17

The data=writeback mount option deserves to be tried, in order to prevent journaling of the file system. This should be done only during the deletion time, there is a risk however if the server is being shutdown or rebooted during the delete operation. According to this page, Some applications show very significant speed improvement when it is used. For ...


15

Upgrade to ext4 or some other modern filesystem that uses extents. Since ext3 uses the indirect blocks scheme rather than extents, deleting large files inevitably entails lots of work.


14

Its performance doesn't scale well with multiple processing cores due to the specific implementation. Prior versions were actually fairly prone to corruption (and I've encountered this multiple times), though that's supposedly resolved in current versions. Its fsck can actually end up causing more corruption on an already heavily corrupted filesystem. ...


14

Unfortunately, there is no such driver (or if there is the author(s) chose not to publish.) All the existing drivers are of questionable quality - I have had data corruption issues with all of them. I would stick with ntfs-3g. Edit: Just to clarify - by "all the existing drivers" I do mean all the existing drivers. That includes the commercial ones. None ...


11

You should consider XFS. It supports a very large number of files both at the filesystem and at the directory level, and the performance remains relatively consistent even with a large number of entries due to the B+ tree data structures. There's a page on their wiki to a large number of papers and publications that detail the design. I recommend you give ...


11

The most interesting answer was originally buried in a comment on the question. Here it is as a first class answer to make it more visible: Basically no method from here worked, so we developed our own. Described it in here: http://www.depesz.com/index.php/2010/04/04/how-to-remove-backups/ – depesz Apr 6 '10 at 15:15 That link is an incredibly ...


11

Options for quickly accessing and backing up millions of files Borrow from people with similar problems This sounds very much like an easier sort of problem that faces USENET news servers and caching web proxies: hundreds of millions of small files that are randomly accessed. You might want to take a hint from them (except they don't typically ever have ...


10

See what 'df -i' says. It could be that you are out of inodes, which might happen if there are a large number of small files in that filesystem, which uses up all the available inodes without consuming all the available space.


10

The differences between the various and sundry ext[0-9] filesystems are more about features, and less about structure. The advantage is that they are fairly compatible with each other. For instance, you can mount an ext3 partition as ext2 and everything should work just fine. Similarly, you can mount an ext3 partition as ext4. Whenever you do either of these ...


10

I really hope you have already taken the affected partition offline so to prevent data been written on top of the deleted data. One useful tool that has worked for me in the past is extundelete. Remount the affected partition as read only and then use extundelete to recover the data. I suppose it is pointless to reiterate that prevention is always better ...


10

There is no per directory file limit in ext3 just the filesystem inode limit (i think there is a limit on the number of subdirectories though). You may still have problems after removing the files. When a directory has millions of files, the directory entry itself becomes very large. The directory entry has to be scanned for every remove operation, and ...


10

Really you should not choose a "creative" solution if you have a simple, effective, and more correct way to do it. Just because the migration isn't easy to implement, the creative solution, in my experience, usually ends with a much bigger headache down the road and ending up having to do it the "right" way anyway. It sounds like you already know better ...


10

First advice If you cannot afford to lose any data (I mean once a user entered new data, if that cannot be lost in the coming seconds) and because you do not have something like a UPS, then I would not remove the write barrier, neither would I switch to writeback. Removing write barriers If you remove write barrier, then in case of crash or power loss, the ...


9

Corruption can also occur on most modern disks due to in-disk re-ordering. Modern disks typically do re-ordering of requests that are used to speed up performance (by re-ordering writes to make the entire list of requests less seeky), this is called Tagged Command Queueing. It is possible the write to the journal on the disk is delayed because its more ...


9

You're both wrong (maybe?)... ext3 is coping the best it can with having its underlying storage removed so abruptly. Your SSD probably has some type of onboard cache. You don't mention the make/model of SSD in use, but this sounds like a consumer-level SSD versus an enterprise or industrial-grade model. Either way, the cache is used to help coalesce ...


9

Provided you have a distro that supports the dir_index capability then you can easily have 200,000 files in a single directory. I'd keep it at about 25,000 though, just to be safe. Without dir_index, try to keep it at 5,000.


9

Try this tutorial Basically you can use the commandline tool ext3grep to search through sections of the filesystem. I have not tried this myself YMMV.


8

I've seen those errors before, but not during the install process. It means that the drive got enough errors that the OS took it to read-only mode. If you could find the full logs, there'd probably be some I/O errors that retried and worked before the full-on failure errors you saw. Something with actual blocks mentioned. It's a storage system error. ...


8

When it comes to recovering data from a hosed disk or system, straight ext3 will be slightly easier than LVM+ext3, simply because it adds a bit of additional complexity and your recovery tools need to be LVM aware. However, LVM gives you significantly increased flexibility and is nearly always justified in my opinion. It can also make backups a lot easier ...


8

A 64 bit kernel and large quantities of RAM will allow the fsck to finish nice and fast. Alternately, there's now an option in e2fsck that'll tell it to store all of it's intermediate results in a directory instead of in RAM, which helps immensely. Create /etc/e2fsck.conf with the following contents: [scratch_files] directory = /var/cache/e2fsck (And, ...


8

Taken from: Linux Filesystem Primer EXT2 Recommended to move to EXT3 Not Journaled POSIX access control EXT2 file system is the predecessor to the EXT3 file system. EXT2 is not journaled, and hence is not recommended any longer (customers should move to EXT3). EXT3 Most popular Linux file system, limited scalability in size and number of files ...


8

You could use the "debugfs" tool to view file info on the command line or interactivley. either use: # debugfs /dev/<spartition> # stat /path/to/file or # debugfs -R "stat /path/to/file" /dev/<partition> for example: # debugfs -R "stat /etc/passwd" /dev/sda5 Inode: 435914 Type: regular Mode: 0644 Flags: 0x0 Generation: 979004472 ...


8

There's no 100% accurate way really, but there's a way to give a good guess. There is a python library chardet which is available here: http://chardet.feedparser.org/ e.g. See what the current LANG variable is set to: $ echo $LANG en_IE.UTF-8 Create a filename that'll need to be encoded with UTF-8 $ touch mÉ.txt Change our encoding and see what ...


8

Be VERY careful how you select the directory split. "a/b/c" sounds like a recipe for disaster to me... Do not just blindly go making a several directory deep structure, say 100 entries in the first level, 100 entries in the second level, 100 entries in the third. I've been there, done that, got the jacket and had to restructure it when performance went in ...



Only top voted, non community-wiki answers of a minimum length are eligible