Specifically, I want to ensure POSIX advisory locking (file locking) is working correctly in simfs in a Linux/Ubuntu VM used for continuous integration testing. We've had file corruption that only occurs to a SQLite DB file when there are concurrent writes by 30 processes. This is only being used in testing by one project, but we'd like to help track down the problem so others won't run into it.
According to the SQLite team and documentation, concurrent writes are only supported when POSIX advisory locks are working in the filesystem/OS. The test I have that uses SQLite works in v3.7.7 of SQLite in OS X, but the same test corrupts the DB file in v3.7.9 of SQLite in the Ubuntu VM provided by TravisCI (and hosted by Blue Box). The SQLite team did not indicate that there were any concurrency issues fixed between those two versions, since concurrency is dependent on the OS/filesystem's POSIX advisory locks working.
Additional information about the environment that I'm trying to investigate:
$ sqlite3 -version 3.7.9 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e $ uname -r 2.6.32-042stab061.2 $ cat /proc/version Linux version 2.6.32-042stab061.2 (root@rh6-build-x64) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Aug 24 09:07:21 MSK 2012 $ lsb_release -a Distributor ID: Ubuntu Description: Ubuntu 12.04.2 LTS Release: 12.04 Codename: precise (home dir it that is exhibiting the problem is within the / mount.) $ cat /proc/mounts /dev/simfs / simfs rw,relatime 0 0 ... $ mount /vz/private/6062841 on / type simfs (rw) ...
I have a ticket in with those that provide the VM here where they stated that they are not using network filesystems, which commonly are associated with POSIX lock-related issues because of the complexity involved with implementing POSIX locks in such environments. In addition to the info above, though this press release would seem to indicate OpenStack is being used, the path above shows 'vz' in the mount, making it seem OpenVZ is being used.
As for tools to help diagnose POSIX lock failures, the only one that I've heard about is a ping-pong test that is part of called smbtorture which tests POSIX locking with Samba, but I'm not using Samba in this case, so I'm not sure that would help.
If there is no command-line test available, how would I go about testing that it is working if I all I have available to me is limited access to the VM (as sudo doesn't require password as my user, but the commands that should output something using sudo don't work, so I think it is overriden)? Are there commands that I could have the VM administrator run to gather more info to help resolve this problem?