Is there a way to add a domain policy to a Windows 2008 domain that sets the attachments size limit for Outlook (various versions) and Windows Mail?
You're really better off doing this on your SMTP server, because it's not client-dependent.
The maximum message size in Outlook 2010 can be controlled with registry modification even for non-Exchange clients that are just using SMTP to deliver messages. Outlook 2007 and older versions do not have this functionality and rely totally on Exchange's size limiting functionality for their own limits.
I'm not aware of any attachment limiting size functionality in Windows Mail. Being that it's a home-user oriented product it really doesn't surprise me that there wouldn't be any such limit. Typically an ISP's mail server would provide the limiting functionality for home users.
As best practice, you should be doing this on Exchange but i agree why waste time uploading (especially with limited bandwidth) to a hosted SMTP server when it will be rejected anyway.
Open the registry editor and browse to:
Then add a new DWORD value: MaximumAttachmentSize
Unlimited message size uses a value of 0. For a specific message size (recommended) enter the value in Kb.
This is configured on your MS Exchange server, not via GPOs.
Setting Message Size Limits in Exchange 2010 and Exchange 2007. It has been already asked here.