If you're coming at this w/o much experience I think you'll end up with the costs of a Windows versus open-source solution being a wash. Your trade off in using an open source solution is going to be, to some extent, your time versus money.
What you're already familiar with, obviously, matters a lot too. If you're not familiar with administering either solution then your costs automatically go up, regardless of what solution you choose
A Linux distro (I'd choose CentOS over Fedora, but that's a personal decision-- CentOS is built for a longer support cycle, whereas Fedora isn't) and Samba will give you a very reasonable solution, as well. There will be less up-front software licensing cost. For a plain-vanilla file server the functionality is more than adequate. Something like OpenFiler may be easier for you to get started with, too.
Depending on how many Windows clients you have you may end up with a better management TCO by using Windows Server, Active Directory, etc. That means more up-front cost. An Active Directory domain gets you Group Policy and lots of cool centralized management features (automated software deployment, centralized enforcement of user settings, etc). Using Windows Server gets you WSUS and control of deployment of Windows updates. Obviously, all these things have a "price of admission".
Hosting an SFTP server on both OS's is reasonably straightforward.
You mention wanting a "nice user/group system", which will be somewhat difficult with any sufficiently large number of Windows client computers w/o something like Active Directory or a Samba domain to provide single-sign-on. You're going to be stuck in a situation, without a domain, of having to create individual user accounts on each machine. While this might seem "good enough", I'd argue that you should take the time, right out of the gate, to get things like single-sign-on, roaming user profiles, storage of user document / data files on the server setup so that, when it comes time to add / move / replace computers your road to hoe is much easier. (Add / move / changes aside, having users able to go to any computer, logon, and get a base level of functionality turns a lot of "fires" into less critical issues because, at least, the user can get work done when their "primary computer" is "down.)
re: the hardware - You can do software RAID 5 on both Windows Server and in Linux w/ drives of mixed sizes. Practically speaking, you get the capacity of (N - 1) disks multipled by the capacity of the smallest disk in the set. (A 36GB, 73GB, and 146GB disk in a RAID-5 will give you 72GB usable space.) You can do some pretty crazy things w/ Linux software RAID like, say, putting two volumes of a RAID-5 set onto a bigger disk, but you're defeating the fault-tolerance purpose of RAID when you do that.
Think about backup and restore, in all of this, too. Whenever you deploy a file server you need to be thinking about how you're going to backup and restore it. If you've never been a sysadmin before, the Tao of Backup is a decent read (even though it is a sales pitch).