I have a bit of a mystery. I have an Ubuntu 17.04 system (upgraded from 16.04 LTS) using ext4 as it's main filesystem. I used wget and curl to download a 2.3GB iso, but i cannot mount it. In fact, I cannot do any operation on it: md5sum, wc, cat, mount -o loop, etc... without getting an "operation not permitted". I can "rm" it, though. I am root, and the perms on the file are 644. I cannot do an "lsattr" nor "chattr" on it without "operation not permitted". I have proven that it's exactly related to the filesize as I did this: dd if=/dev/zero of=/tmp/test.iso bs=1M count=2047 and I am able to read test.iso as it's 1M less than 2G, but if i change it to 2048, I am unable to read the file. I understand that ext4 has a bare minimum limit of 16GB, but I am way under that. The files appear to be created just fine.
I did a thorough search before posting and nothing is related to my problem. No, it's not FAT. It's ext4. I see no errors in dmesg after accessing the file, as this would show apparmour errors. I doubt it's an immutable perm as I don't even have permission to lsattr on it. I have an almost identical Ubuntu 17.04 system and it works fine (except that was installed fresh instead of via upgrade from 16.04).
If I write to /dev/shm instead of /tmp, I can md5sum a 3GB file just fine... so there's something related to the way it's mounting /. /dev/shm is obviously a different mountpoint and not ext4.
# ls -l asm8.iso -rw-r--r-- 1 root root 2253135872 Jan 15 20:27 asm8.iso
# md5sum asm8.iso md5sum: asm8.iso: Operation not permitted
# strace -f md5sum asm8.iso ... open("asm8.iso", O_RDONLY) = -1 EPERM (Operation not permitted) ...
# lsattr asm8.iso lsattr: Operation not permitted While reading flags on asm8.iso
# mount |grep " / " /dev/mapper/asci--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
# tune2fs -l /dev/mapper/asci--vg-root ... Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Filesystem state: clean Filesystem OS type: Linux ...
I compared the working system, and the only difference is that it doesnt have uninit_bg, and has 2 extras: 64bit,metadata_csum. I researched these flags and they don't seem to be relevant to the issue.
Does anyone have any ideas? If you want to see the output of any command, I will gladly share them. Thanks in advance.
Here is the extra info:
# df -h Filesystem Size Used Avail Use% Mounted on udev 7.9G 0 7.9G 0% /dev tmpfs 1.6G 114M 1.5G 8% /run /dev/mapper/asci--vg-root 992G 643G 299G 69% / tmpfs 7.9G 0 7.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup /dev/sda1 472M 116M 332M 26% /boot tmpfs 1.6G 0 1.6G 0% /run/user/999 tmpfs 1.6G 0 1.6G 0% /run/user/1003 tmpfs 1.6G 0 1.6G 0% /run/user/0 tmpfs 1.6G 0 1.6G 0% /run/user/1008
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on udev 2046118 419 2045699 1% /dev tmpfs 2051789 712 2051077 1% /run /dev/mapper/asci--vg-root 66035712 3469883 62565829 6% / tmpfs 2051789 1 2051788 1% /dev/shm tmpfs 2051789 5 2051784 1% /run/lock tmpfs 2051789 16 2051773 1% /sys/fs/cgroup /dev/sda1 124928 315 124613 1% /boot tmpfs 2051789 5 2051784 1% /run/user/999 tmpfs 2051789 5 2051784 1% /run/user/1003 tmpfs 2051789 5 2051784 1% /run/user/0 tmpfs 2051789 5 2051784 1% /run/user/1008
# cat /etc/fstab # /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # /dev/mapper/asci--vg-root / ext4 errors=remount-ro 0 1 # /boot was on /dev/sda1 during installation UUID=20926db6-5e54-4907-912a-5fe996f45adc /boot ext2 defaults 0 2 /dev/mapper/asci--vg-swap_1 none swap sw 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
It's not a directory issue as the files are created fine in the same directory: I can create the 2 files.
root@asci /tmp # dd if=/dev/zero of=/tmp/file2047.iso bs=1M count=2047 2047+0 records in 2047+0 records out 2146435072 bytes (2.1 GB, 2.0 GiB) copied, 6.46013 s, 332 MB/s root@asci /tmp # dd if=/dev/zero of=/tmp/file2048.iso bs=1M count=2048 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 60.2936 s, 35.6 MB/s root@asci /tmp # ls -ls /tmp/file*.iso 2096132 -rw-r--r-- 1 root root 2146435072 Jan 19 05:52 /tmp/file2047.iso 2097156 -rw-r--r-- 1 root root 2147483648 Jan 19 05:53 /tmp/file2048.iso
Even though md5sum, cat and wc all say "operation not permitted", it seems as though I can run file and stat on them:
# file *.iso file2047.iso: data file2048.iso: writable, regular file, no read permission
Here's stat. I don't see notable differences.
# stat *.iso File: file2047.iso Size: 2146435072 Blocks: 4192264 IO Block: 4096 regular file Device: fd00h/64768d Inode: 7380361 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-01-19 05:52:30.925760162 +0100 Modify: 2018-01-19 05:52:30.921760136 +0100 Change: 2018-01-19 05:52:30.921760136 +0100 Birth: - File: file2048.iso Size: 2147483648 Blocks: 4194312 IO Block: 4096 regular file Device: fd00h/64768d Inode: 7380363 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-01-19 05:52:36.765797317 +0100 Modify: 2018-01-19 05:53:37.034180299 +0100 Change: 2018-01-19 05:53:37.034180299 +0100 Birth: -
I'm not an apparmor expert, but rather than go through a learning curve, I just disabled it and you can see that it's still a problem. I suspected it wasn't the issue because I think that "dmesg" should report apparmor blocking messages and there was no new dmesg output during my tests.
# service apparmor stop root@asci /tmp # service apparmor teardown * Unloading AppArmor profiles ...done. root@asci /tmp # update-rc.d -f apparmor remove root@asci /tmp # wc -c file2048.iso wc: file2048.iso: Operation not permitted
OMG. What finally fixed it was "apt remove apparmor; reboot". Now it works. So it must've been an apparmor profile. I don't use it, so something in the default apparmor settings is preventing the viewing of files over 2G. Does anyone have any idea what setting that might be? I will have to reinstall now to know.