In my experience, the big issues are:
We've defined "project" directories and use XFS project quotas to enforce quotas. We apply both soft and hard quotas, and have scripts to alert users by email when their soft quotas are reached. This is braindead easy to do with XFS.
Permissions always seem to be a huge problem. We now use ACLs to ensure that all members of the group have access to all files that are created, but problems arise when we share the directories to computers that don't understand those ACLs, or when programs create files with explicitly narrow permissions. We end up with subdirectories that don't have the correct ACLs, and then users complain that they can't edit some items in their group folder that someone else created. This leads to everyone getting frusterated and running
chmod -R 777 * on everything they own. It seems that most people expect a group folder to have a single set of permissions which apply equally to all descendant filesystem objects.
To enforce this, we have a cron script called the permission smasher that periodically walks through all project directories and fixes up the permissions (both standard and ACLs) of all filesystem objects. Everything gets the same permission levels (although it maintains the eXecutable bit), and the permissions are defined in a config file that is version controlled. This gives us excellent assurance that users aren't unknowingly circumventing access controls to give access to the wrong people.
On our Mac OS X Servers, this is quite a bit easier. HFS has ACLs that can override the ACLs of all descendant filesystem objects. Even better, they're applied on the server side, so clients that don't understand HFS ACLs (linux NFS clients, for example) work as expected too. We simply define the ACL on the group directory itself, and all of its contents are always available uniformly to the entire group.
Another tricky issue is who is ultimately responsible for data that is stored in group/project directories. Data tends to accumulate and rot in them, because no single person is responsible for keeping it tidy. When the quota is reached, it is sometimes difficult to find the appropriate person to notify. To solve this, we try to designate a single user as the maintainer for each directory.