I've confirmed the issue over here. I don't know why Linux refuses to dump core contents to an HGFS share (Arch Linux kernel 2.6.32 with open-vm-tools 2010.01.19 here), but I do have a solution.
Linux 2.6.19 and higher will let you pipe core dumps through an arbitrary program, so create a shell script that copies its stdin to a file on your HGFS share, e.g.:
# Where do you want the core to go?
tee $COREFILE >/dev/null
Of course you may wish to implement some logic for
$COREFILE so that each subsequent core dump doesn't just overwrite the last.
Save your script as
/usr/local/bin/core.sh, then set the file's executable bit and configure core_pattern as follows:
# chmod +x /usr/local/bin/core.sh
# sysctl -w kernel.core_pattern='|/usr/local/bin/core.sh'
Linux will pipe any core dumps through your shell script, which won't have any problem writing to the HGFS share itself.
If you're wondering, you can't simply put the
tee command directly in kernel.core_pattern, because in kernels older than 2.6.24 you can't specify arguments to a pipe command with this sysctl. For the same reason, unfortunately I can't think of a good way for you to incorporate the core_pattern template specifiers into your core dump file names using this method, if you're tied to kernel 2.6.21.