I'm putting together a guide on this at the moment for the very reason that there is not a lot of good info out there about how best to configure the exact architecture you describe above.
My config is similar to Chirs_O's config described above.
In a situation where I'm the only admin who needs to manage files under each site I'll typically change file ownership to a specific user to ensure I can easily scp/ftp files from local to each site.
I typically use the following command
chown -R myuser:myuser /site1.com
This allows me to transfer files with no permissions issues and is also a good secure setup for serving these files.
The downside to this config is that you'll hit all sorts of problems with plugin installs, wordpress updates etc. (and image uploads).
Before I do plugin installs, wordpress updates I typically change ownership of a site from myuser to www-data using the same command
chown -R www-data:www-data /site1.com
I then install whatever plugins I need and then reset permissions back to myuser.
Some people leave www-data as the main owner but I understand this is not a great security setup but I've come across a permissions config for nginx/php5-fpm yet that provides ease of file transfer under individual ftp/ssh accounts while allowing WordPress plugin installs/updates to be done without the need for setting 777 permissions on specific folders which is a pain and has it's own security issues.
I'm currently exploring an alternative config where I setup individual vhosts directly within each users individual accounts e.g.
This ensures the ftp/ssh account for that user is restricted to their own home account. There currently would still be the need to change ownership back to www-data for wordpress updates etc. but I'm trying to figure out if there is a secure way that these user accounts can be added to the www-data group to allow a best of both world's scenario. Will let you know if I get a working config.