If you're navigating between subdomains on the same domain (like www.YOURDOMAIN.com to app.YOURDOMAIN.com) you can get the desired results with a domain level asp.net authentication cookie using SetAuthCookie and some web.config changes.
This article was helpful when I setup a similar configuration Single Sign-on in ASP.NET and Other Platforms - CodeProject
Step 1 - Set Cookie
Even though the SharePoint app uses Windows auth, you can still call this method to create a .NET auth cookie for your current domain and auto authenticate the user when they navigate to your forms auth app. So, once your user logs into your SharePoint site apply the cookie (possibly via a custom web part or page/masterpage code-behind) like so:
// this will create a persistant cookie
// (meaning closing/re-opening the browser won't remove the cookie)
FormsAuthentication.SetAuthCookie(
SPContext.Current.Web.CurrentUser.LoginName,
true);
Step 2 - Set Identical <machineKey> values
In the web.config of both applications you'll need to set identical values for your <machineKey> (inside <system.web>). Such as:
<machineKey
validation="SHA1"
validationKey="[128 character random string]"
decryption="AES"
decryptionKey="[64 character random string]" />
Step 3 - Set <forms> values
You'll also want to set values in the <forms> tag so that when the auth cookie is created it will use your settings for domain, cookieless
in your SharePoint web.config apply the following
<authentication mode="Windows">
<forms domain="YOURDOMAIN.COM"
cookieless="UseCookies"
enableCrossAppRedirects="true" />
</authentication>
in your forms-auth app set these in your web.config along with whatever you have already (probably at least your loginUrl and timeout property values are set also)
<authentication mode="Forms">
<forms domain="YOURDOMAIN.COM"
cookieless="UseCookies"
enableCrossAppRedirects="true" />
</authentication>