Why is an nfs server mounted as rw returning read only filesystem errors

Question

I am working on a Ubuntu Server 64bit. I have mounted an nfs as rw, but whenever I try to edit anything on the mountpoint in question, I get a read-only filesystem error

my etc/fstab reads:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/mapper/product-root /               ext4    errors=remount-ro 0       1
UUID=1cae6948-ca9d-4f0b-aec6-9e7a60bbbad8 /boot           ext2    defaults        0       2
/dev/mapper/product-swap_1 none            swap    sw              0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0
vnxnfs1.company.com:/NFS2         /mnt/nfs2                        nfs     defaults       0       0

mount returns:

/dev/mapper/product-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
vnxnfs1.company.com:/NFS2 on /mnt/nfs2 type nfs (rw)

cat /proc/mounts returns:

rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=16465812k,nr_inodes=4116453,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=6590172k,mode=755 0 0
/dev/mapper/product-root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
/dev/sda1 /boot ext2 rw,relatime,errors=continue 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
vnxnfs1.company.com:/NFS2 /mnt/nfs2 nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.x.y.z,mountvers=3,mountport=1234,mountproto=udp,local_lock=none,addr=172.x.y.z 0 0

Files on the mnt read just fine but whenever I try to change anything I get an error:

touch: cannot touch `/mnt/nfs2/path/to/test_file.txt': Read-only file system

I can connect to the nfs in question from other servers and read and write just fine. The only problem is on this server. I have tried mount -o remount,rw vnxnfs1.company.com:/NFS2 /mnt/nfs2 as answers to related questions suggest, but to no avail.

Sorry for the data dump, I just tried to include anything that could yield clues.

EDIT:

more details

I have been testing from the root user on the machine with troubles, and writing to nfs works from root as well as user from other (nfs-write-is-working) server.

from working server: ls -la /mnt/nfs2/

total 76
drwxrwxrwx 12 root root   1024 2013-04-18 10:14 .
drwxr-xr-x 24 root root   4096 2013-04-17 10:42 ..
drwxrwxr-x  2 user user   80 2013-04-18 10:14 archives
drwxrwxrwx  2 user user   1024 2012-08-15 10:13 assignee_dealiasing_temp_folder
drwxrwxr-x  9 user user   1024 2013-03-11 13:50 data_runs
drwxrwxrwx  2 user user   2048 2013-04-19 14:54 db_transfer_dumps
drwxrwxrwx  2 root daemon 1024 2013-08-22 01:00 .etc
drwxrwxrwx  2 root root   8192 2012-08-16 14:47 lost+found
drwxr-xr-x 13 user user   1024 2013-08-16 13:37 projects
drwxr-xr-x 35 user user   2048 2013-05-10 16:34 reports
drwxrwxrwx  6 root root   1024 2013-04-17 16:47 saved_logs

from the vnxnfs1 server_export server_2 -list

server_2 :
export "/NFS3" rw=172.16.0.0/24 root=172.16.0.0/24
export "/NFS2" rw=172.16.0.0/24:172.16.9.0/24 root=172.16.0.0/24:172.16.9.0/24
export "/NFS1" rw=172.16.0.0/24:172.16.9.0/24 root=172.16.0.0/24:172.16.9.0/24
export "/ISO" rw=172.16.9.16:172.16.9.18:172.16.9.20 root=172.16.9.16:172.16.9.18:172.16.9.20
export "/NFS0" rw=172.16.0.0/24 root=172.16.0.0/24
export "/" anon=0 access=128.221.252.100:128.221.253.100:128.221.252.101:128.221.253.101

Answer 1

'Read-only file system' error indicates that the file system is exported as read only. The rw mount option tells client that 'WRITE' is allowed to be sent to the server. On regular unix systems, check /etc/exports file:

/NFS2 172.x.y.z(rw)

on appliances, check the documentation to export as read-write.

Answer 2

Export of a parent folder (ro) will override a child export (rw).

Example exports:

/NFS_ROOT *(ro)
/NFS_ROOT/SHARE1 *(rw)

SHARE1 will be read-only because NFS_ROOT is read-only. You have constraints on an exported root directory which may be causing the problem.

Answer 3

As Brian said, a parent export can override a child export. But you can solve this by adding priorities to your exports. So, using Brian's example, this would solve the problem:

/NFS_ROOT *(ro,fsid=2)
/NFS_ROOT/SHARE1 *(rw,fsid=1)

Answer 4

In my case the problem was an extra space. It says rw, but the space before the bracket apparently breaks it silently:

/NFS2 172.x.y.z (rw)

Answer 5

Check for and remove a space before the (rw):

Wrong format:

/u02/backup 10.174.68.0/24 (rw)

Right format:

/u02/backup 10.174.68.0/24(rw)