How this works appears to change depending on whether you connected to the share using a "cifs://" or "smb://" URL (i.e. whether the location you type in to the Finder's "Connect to server" dialog begins with "cifs" or with "smb").
If you connect using a "cifs://" URL, then OS X will attempt to use the "UNIX extensions" described in other answers here, and files copied or moved to the share will retain the permissions they had on the source drive, no matter what the various permission masks/modes settings are for the share. I think this is probably due to a bug in Samba (I'm using 3.5.6 from Debian Squeeze - 2:3.5.6~dfsg-3squeeze8 if anyone is interested).
If however you connect using an "smb://" URL, the UNIX extensions will not be used, and the permissions will be dictated by the following settings for files:
force create mode
create mask
security mask
force security mode
and these for directories:
force directory mode
directory mask
directory security mask
force directory security mode
Oh, and don't forget:
map hidden
map archive
map system
There may be more that I've forgotten, but these are the most common.
See the manpage for smb.conf for details of how all these settings work.
You can use the "unix extensions" setting to disable all connections, whether made using cifs:// or smb:// URLS, to behave as the more basic smb:// connections usually do. The advantages of using the UNIX extensions are include that things like links (symbolic or hard) may work. Unfortunately this is a global setting, and can't be set per-share (at least with the Samba version I have here).