Basic Authentication is a term that generally refers to authentication within the HTTP protocol.
Forms based authentication is handled within context of a web-based application. This usually involves a form which sets some kind of session identifier with a cookie, and then when the form is processed information is associated with that session on the server side about the users state.
There really isn't any direct relationship between form based authentication which is basically tracked via the session cookie, and the HTTP-based authentication which is actually directly within the HTTP headers.
how can it be possible that IIS would prevent the use of Basic Auth on a site that uses a custom form auth for its users?
It has nothing to do with IIS preventing basic auth, it has to do with the two not being compatible. If you do your initial authentication with a form, then the associated login state will be stored in a session. But the software handling basic authentication doesn't normally know anything about cookies or sessions, all it knows about is HTTP authentication. When you let IIS perform the authentication stem, the authentication happens before your application is even touched.
If you use the built-in facilities of IIS for Basic authentication then you basically have to use that only.
But, it should be possible to implement HTTP authentication within your application by having your application send and parse the correct HTTP headers. For this you would leave IIS set to forms-based authentication, and then you simply do everything within your application. In that way it should be possible to have your application send out the proper headers depending on the session state.