2

I have a bit of a mystery. I have an Ubuntu 17.04 system (upgraded from 16.04 LTS) using ext4 as it's main filesystem. I used wget and curl to download a 2.3GB iso, but i cannot mount it. In fact, I cannot do any operation on it: md5sum, wc, cat, mount -o loop, etc... without getting an "operation not permitted". I can "rm" it, though. I am root, and the perms on the file are 644. I cannot do an "lsattr" nor "chattr" on it without "operation not permitted". I have proven that it's exactly related to the filesize as I did this: dd if=/dev/zero of=/tmp/test.iso bs=1M count=2047 and I am able to read test.iso as it's 1M less than 2G, but if i change it to 2048, I am unable to read the file. I understand that ext4 has a bare minimum limit of 16GB, but I am way under that. The files appear to be created just fine.

I did a thorough search before posting and nothing is related to my problem. No, it's not FAT. It's ext4. I see no errors in dmesg after accessing the file, as this would show apparmour errors. I doubt it's an immutable perm as I don't even have permission to lsattr on it. I have an almost identical Ubuntu 17.04 system and it works fine (except that was installed fresh instead of via upgrade from 16.04).

If I write to /dev/shm instead of /tmp, I can md5sum a 3GB file just fine... so there's something related to the way it's mounting /. /dev/shm is obviously a different mountpoint and not ext4.

# ls -l asm8.iso
-rw-r--r-- 1 root root 2253135872 Jan 15 20:27 asm8.iso
# md5sum asm8.iso
md5sum: asm8.iso: Operation not permitted
# strace -f md5sum asm8.iso
...
open("asm8.iso", O_RDONLY)              = -1 EPERM (Operation not permitted)
...
# lsattr asm8.iso
lsattr: Operation not permitted While reading flags on asm8.iso
# mount |grep " / "
/dev/mapper/asci--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
# tune2fs -l /dev/mapper/asci--vg-root

...
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Filesystem state:         clean
Filesystem OS type:       Linux
...

I compared the working system, and the only difference is that it doesnt have uninit_bg, and has 2 extras: 64bit,metadata_csum. I researched these flags and they don't seem to be relevant to the issue.

Does anyone have any ideas? If you want to see the output of any command, I will gladly share them. Thanks in advance.

Here is the extra info:

# df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                       7.9G     0  7.9G   0% /dev
tmpfs                      1.6G  114M  1.5G   8% /run
/dev/mapper/asci--vg-root  992G  643G  299G  69% /
tmpfs                      7.9G     0  7.9G   0% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                      7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda1                  472M  116M  332M  26% /boot
tmpfs                      1.6G     0  1.6G   0% /run/user/999
tmpfs                      1.6G     0  1.6G   0% /run/user/1003
tmpfs                      1.6G     0  1.6G   0% /run/user/0
tmpfs                      1.6G     0  1.6G   0% /run/user/1008
 # df -i
Filesystem                  Inodes   IUsed    IFree IUse% Mounted on
udev                       2046118     419  2045699    1% /dev
tmpfs                      2051789     712  2051077    1% /run
/dev/mapper/asci--vg-root 66035712 3469883 62565829    6% /
tmpfs                      2051789       1  2051788    1% /dev/shm
tmpfs                      2051789       5  2051784    1% /run/lock
tmpfs                      2051789      16  2051773    1% /sys/fs/cgroup
/dev/sda1                   124928     315   124613    1% /boot
tmpfs                      2051789       5  2051784    1% /run/user/999
tmpfs                      2051789       5  2051784    1% /run/user/1003
tmpfs                      2051789       5  2051784    1% /run/user/0
tmpfs                      2051789       5  2051784    1% /run/user/1008
# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
/dev/mapper/asci--vg-root /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=20926db6-5e54-4907-912a-5fe996f45adc /boot           ext2    defaults        
0       2
/dev/mapper/asci--vg-swap_1 none            swap    sw              0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0

It's not a directory issue as the files are created fine in the same directory: I can create the 2 files.

root@asci /tmp
# dd if=/dev/zero of=/tmp/file2047.iso bs=1M count=2047
2047+0 records in
2047+0 records out
2146435072 bytes (2.1 GB, 2.0 GiB) copied, 6.46013 s, 332 MB/s

root@asci /tmp
# dd if=/dev/zero of=/tmp/file2048.iso bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 60.2936 s, 35.6 MB/s

root@asci /tmp
# ls -ls /tmp/file*.iso
2096132 -rw-r--r-- 1 root root 2146435072 Jan 19 05:52 /tmp/file2047.iso
2097156 -rw-r--r-- 1 root root 2147483648 Jan 19 05:53 /tmp/file2048.iso

Even though md5sum, cat and wc all say "operation not permitted", it seems as though I can run file and stat on them:

# file *.iso
file2047.iso: data
file2048.iso: writable, regular file, no read permission

Here's stat. I don't see notable differences.

# stat *.iso
  File: file2047.iso
  Size: 2146435072      Blocks: 4192264    IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 7380361     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-01-19 05:52:30.925760162 +0100
Modify: 2018-01-19 05:52:30.921760136 +0100
Change: 2018-01-19 05:52:30.921760136 +0100
 Birth: -
  File: file2048.iso
  Size: 2147483648      Blocks: 4194312    IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 7380363     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-01-19 05:52:36.765797317 +0100
Modify: 2018-01-19 05:53:37.034180299 +0100
Change: 2018-01-19 05:53:37.034180299 +0100
 Birth: -

I'm not an apparmor expert, but rather than go through a learning curve, I just disabled it and you can see that it's still a problem. I suspected it wasn't the issue because I think that "dmesg" should report apparmor blocking messages and there was no new dmesg output during my tests.

# service apparmor stop

root@asci /tmp
# service apparmor teardown
 * Unloading AppArmor profiles
   ...done.

root@asci /tmp
# update-rc.d -f apparmor remove

root@asci /tmp
# wc -c file2048.iso
wc: file2048.iso: Operation not permitted

OMG. What finally fixed it was "apt remove apparmor; reboot". Now it works. So it must've been an apparmor profile. I don't use it, so something in the default apparmor settings is preventing the viewing of files over 2G. Does anyone have any idea what setting that might be? I will have to reinstall now to know.

  • can you share your df -h, df -i, /etc/fstab? – Tux_DEV_NULL Jan 16 '18 at 8:19
  • thanks. i just edited my post and added the output of those. :) – Mike Weller Jan 16 '18 at 17:31
  • I am not seeing anything right way. Can you file file asm8.iso or stat asm8.iso? Could it be directory permission issue? Also check if you have any apparmor profile that can cause this. – Tux_DEV_NULL Jan 17 '18 at 9:36
  • Thanks so much Tux_DEV_NULL. You are absolutely right on the apparmor causing it! After actually removing it from my system, I was able to read the file more than 2G. Any idea which default Ubuntu apparmor setting would cause this? I will have to reinstall apparmor to reproduce now. – Mike Weller Jan 19 '18 at 5:10
  • Glad you figured out the issue. Sorry, do not use apparmor regularly so can't comment. Maybe look up about a profile that deals with file permissions and attributes. – Tux_DEV_NULL Jan 19 '18 at 8:20
3

I have solved my problem. The reason for not being able to access files over 2GB in size is due to my antivirus software (McAfee for Linux - Endpoint Security for Linux Threat Prevention). It was not due to apparmor, funky filesystem options, permissions, disk space or older filesystems.

Here is how I came to my conclusion. The troubleshooting steps might be interesting. Immediately following a reboot, I can quickly create a 2G file and probe it like so:

root@asci /tmp
# dd if=/dev/zero of=/tmp/file2049.iso bs=1M count=2049
2049+0 records in
2049+0 records out
2148532224 bytes (2.1 GB, 2.0 GiB) copied, 2.05888 s, 1.0 GB/s

root@asci /tmp
# md5sum file2049.iso
4555da35a7064cc499ba1e3f6fa1993a  file2049.iso

Within a minute, the md5sum no longer works.

To rule out crontabs, I disabled crontab, and also wrote a 1-liner script to output 0 upon successful read and 1 if unsuccessful. Notice that it breaks at the 40'th second (indicating most likely not a cron job):

# while [ 1 == 1 ]; do echo -n "`date` - ";dd if=asm8.iso bs=1M count=30 2>&1|grep -c "Operation not permi"; sleep 1;date >> /var/log/ps.log; ps -efww >> /var/log/ps.log; done
Fri Jan 19 18:57:28 CET 2018 - 0
Fri Jan 19 18:57:30 CET 2018 - 0
Fri Jan 19 18:57:31 CET 2018 - 0
Fri Jan 19 18:57:32 CET 2018 - 0
Fri Jan 19 18:57:33 CET 2018 - 0
Fri Jan 19 18:57:34 CET 2018 - 0
Fri Jan 19 18:57:35 CET 2018 - 0
Fri Jan 19 18:57:36 CET 2018 - 0
Fri Jan 19 18:57:37 CET 2018 - 0
Fri Jan 19 18:57:38 CET 2018 - 0
Fri Jan 19 18:57:39 CET 2018 - 0
Fri Jan 19 18:57:40 CET 2018 - 1
Fri Jan 19 18:57:41 CET 2018 - 1
Fri Jan 19 18:57:42 CET 2018 - 1
Fri Jan 19 18:57:43 CET 2018 - 1
Fri Jan 19 18:57:44 CET 2018 - 1

I only dd the start of the file, as that's all I need to test it, as it takes 48 seconds to read the entire thing. I logged the ps output to /var/log/ps.log. To do a diff on the ps output by comparing what changed between the 39th second and 40th second, I did this:

# cat ps.log |grep "Fri Jan 19 18:57:39 CET 2018" -A10000 |grep "Fri Jan 19 18:57:40 CET 2018" -B10000 > /tmp/ps39
# cat ps.log |grep "Fri Jan 19 18:57:40 CET 2018" -A10000 |grep "Fri Jan 19 18:57:41 CET 2018" -B10000 > /tmp/ps40
# cd /tmp

Comparing the ps outputs between the 39th and 40th second:

# diff ps39 ps40
 Fri Jan 19 18:57:40 CET 2018
266c266
 root      2412  1276 97 18:57 ?        00:00:19 /opt/isec/ens/threatprevention/bin/isectpd
275,276c275,278
 root      2632  2412  1 18:57 ?        00:00:00 /opt/isec/ens/threatprevention/bin/isectpd
> root      2635  2412 11 18:57 ?        00:00:00 /opt/isec/ens/threatprevention/bin/isectpd
> root      2663  2518  0 18:57 pts/1    00:00:00 ps -efww

So, that's it! Once the isectpd process started up at the 40th second, it disabled access to my 2GB file.

Once I did this:

# systemctl stop isectpd

It started working. Obviously, this is a bandaid until I find out how to allow > 2GB files from McAfee. If anyone has any experience with this, feel free to chime in.

Cheers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.