There's no built-in mechanism in any version of Windows Server to do what you're describing. The quota functionality only deals with the total size of files owned by a given user, not with the sizes of indvidual files. The "file screen" functionality in Windows Server 2003 R2 File Server Resource Manager only concerns itself with file extensions, not with sizes.
It would take a fairly low-level hook into the filesystem driver stack in order to implement what you're looking for. I really doubt you're going to find an off-the-shelf utility to do what you're looking for.
You could always run a script that looks for files over that size and deletes them, but that's not going to stop users from saving files there (and potentially thinking that the files are safe and sound).