Tag Info

Hot answers tagged

39

# tune2fs -l /dev/sda1 | grep -i 'block size' Block size: 1024 Replace /dev/sda1 with the partition you want to check.


37

Check for files on located under mount points. Frequently if you mount a directory (say a sambafs) onto a filesystem that already had a file or directories under it, you lose the ability to see those files, but they're still consuming space on the underlying disk. I've had file copies while in single user mode dump files into directories that I couldn't ...


31

The -C flag will display a progress bar. Performance differences depending on how fsck is called. And very cool, if e2fsck is already running, you can send a USR1 signal for it to start displaying a progress bar. USR2 to stop. Example: killall -USR1 e2fsck From FSCK(8): -C Display completion/progress bars for those filesys- tems ...


26

Whilst a major cause of this problem is ext3 performance with millions of files, the actual root cause of this problem is different. When a directory needs to be listed readdir() is called on the directory which yields a list of files. readdir is a posix call, but the real linux system call being used here is called 'getdents'. Getdents list directory ...


24

Would it be possible to backup all of the other files from this file system to a temporary storage location, reformat the partition, and then restore the files?


18

The data=writeback mount option deserves to be tried, in order to prevent journaling of the file system. This should be done only during the deletion time, there is a risk however if the server is being shutdown or rebooted during the delete operation. According to this page, Some applications show very significant speed improvement when it is used. For ...


17

Upgrade to ext4 or some other modern filesystem that uses extents. Since ext3 uses the indirect blocks scheme rather than extents, deleting large files inevitably entails lots of work.


14

Its performance doesn't scale well with multiple processing cores due to the specific implementation. Prior versions were actually fairly prone to corruption (and I've encountered this multiple times), though that's supposedly resolved in current versions. Its fsck can actually end up causing more corruption on an already heavily corrupted filesystem. ...


14

Unfortunately, there is no such driver (or if there is the author(s) chose not to publish.) All the existing drivers are of questionable quality - I have had data corruption issues with all of them. I would stick with ntfs-3g. Edit: Just to clarify - by "all the existing drivers" I do mean all the existing drivers. That includes the commercial ones. None ...


12

The most interesting answer was originally buried in a comment on the question. Here it is as a first class answer to make it more visible: Basically no method from here worked, so we developed our own. Described it in here: http://www.depesz.com/index.php/2010/04/04/how-to-remove-backups/ – depesz Apr 6 '10 at 15:15 That link is an incredibly ...


12

See what 'df -i' says. It could be that you are out of inodes, which might happen if there are a large number of small files in that filesystem, which uses up all the available inodes without consuming all the available space.


12

Options for quickly accessing and backing up millions of files Borrow from people with similar problems This sounds very much like an easier sort of problem that faces USENET news servers and caching web proxies: hundreds of millions of small files that are randomly accessed. You might want to take a hint from them (except they don't typically ever have ...


12

First advice If you cannot afford to lose any data (I mean once a user entered new data, if that cannot be lost in the coming seconds) and because you do not have something like a UPS, then I would not remove the write barrier, neither would I switch to writeback. Removing write barriers If you remove write barrier, then in case of crash or power loss, the ...


11

You should consider XFS. It supports a very large number of files both at the filesystem and at the directory level, and the performance remains relatively consistent even with a large number of entries due to the B+ tree data structures. There's a page on their wiki to a large number of papers and publications that detail the design. I recommend you give ...


10

You could use the "debugfs" tool to view file info on the command line or interactivley. either use: # debugfs /dev/<spartition> # stat /path/to/file or # debugfs -R "stat /path/to/file" /dev/<partition> for example: # debugfs -R "stat /etc/passwd" /dev/sda5 Inode: 435914 Type: regular Mode: 0644 Flags: 0x0 Generation: 979004472 ...


10

Just stumbled on this page when trying to track down an issue on a local server. In my case the df -h and du -sh mismatched by about 50% of the hard disk size. This was caused by apache (httpd) keeping large log files in memory which had been deleted from disk. This was tracked down by running lsof | grep "/var" | grep deleted where /var was the partition ...


10

The differences between the various and sundry ext[0-9] filesystems are more about features, and less about structure. The advantage is that they are fairly compatible with each other. For instance, you can mount an ext3 partition as ext2 and everything should work just fine. Similarly, you can mount an ext3 partition as ext4. Whenever you do either of these ...


10

There is no per directory file limit in ext3 just the filesystem inode limit (i think there is a limit on the number of subdirectories though). You may still have problems after removing the files. When a directory has millions of files, the directory entry itself becomes very large. The directory entry has to be scanned for every remove operation, and ...


10

Provided you have a distro that supports the dir_index capability then you can easily have 200,000 files in a single directory. I'd keep it at about 25,000 though, just to be safe. Without dir_index, try to keep it at 5,000.


10

Online resizing an ext4fs from 1 TB to 2 TB, on an RAID1 lvm2 took about 1h40min on a pair of seagate barracuta 2TB SATA drives. This was the root partition. During this time, the CPU never went over 10% (cumulating md2_raid1 and md2_resize). The load was about 3. If your resize2fs processes is eating 100% of your CPU and your HDD does not work, it really ...


10

I really hope you have already taken the affected partition offline so to prevent data been written on top of the deleted data. One useful tool that has worked for me in the past is extundelete. Remount the affected partition as read only and then use extundelete to recover the data. I suppose it is pointless to reiterate that prevention is always better ...


10

Really you should not choose a "creative" solution if you have a simple, effective, and more correct way to do it. Just because the migration isn't easy to implement, the creative solution, in my experience, usually ends with a much bigger headache down the road and ending up having to do it the "right" way anyway. It sounds like you already know better ...


10

Corruption can also occur on most modern disks due to in-disk re-ordering. Modern disks typically do re-ordering of requests that are used to speed up performance (by re-ordering writes to make the entire list of requests less seeky), this is called Tagged Command Queueing. It is possible the write to the journal on the disk is delayed because its more ...


9

You're both wrong (maybe?)... ext3 is coping the best it can with having its underlying storage removed so abruptly. Your SSD probably has some type of onboard cache. You don't mention the make/model of SSD in use, but this sounds like a consumer-level SSD versus an enterprise or industrial-grade model. Either way, the cache is used to help coalesce ...


9

Try this tutorial Basically you can use the commandline tool ext3grep to search through sections of the filesystem. I have not tried this myself YMMV.


9

There's no 100% accurate way really, but there's a way to give a good guess. There is a python library chardet which is available here: https://pypi.python.org/pypi/chardet e.g. See what the current LANG variable is set to: $ echo $LANG en_IE.UTF-8 Create a filename that'll need to be encoded with UTF-8 $ touch mÉ.txt Change our encoding and see ...


8

from man page for version 1.41 -C fd This option causes e2fsck to write completion information to the specified file descriptor so that the progress of the filesystem check can be monitored. This option is typically used by programs which are running e2fsck. If the file descriptor number is negative, then absolute value of the ...


8

I've seen those errors before, but not during the install process. It means that the drive got enough errors that the OS took it to read-only mode. If you could find the full logs, there'd probably be some I/O errors that retried and worked before the full-on failure errors you saw. Something with actual blocks mentioned. It's a storage system error. ...


8

When it comes to recovering data from a hosed disk or system, straight ext3 will be slightly easier than LVM+ext3, simply because it adds a bit of additional complexity and your recovery tools need to be LVM aware. However, LVM gives you significantly increased flexibility and is nearly always justified in my opinion. It can also make backups a lot easier ...



Only top voted, non community-wiki answers of a minimum length are eligible