If I recall correctly, this is fine, the different versions coexist peacefully, but I just want to double-check there's no possibility of problems (I'm installing the newer version of the .net framework on an older production server in preparation for upgrading a critical asp.net 1.1 app to 3.5 in the near future).
As long as the two applications are running in a different AppPool, it's like having two separate servers. No problems.
As Moshe mentioned, it is the AppPool that matters with IIS. With .NET in general, 1.1 and 3.5 can co-exist without interference because they are running on different cores. .NET 4.0 is also completely separate from the earlier versions. Microsoft has a pretty decent article that covers the versions and dependencies:
The relationships between the .NET Framework versions 2.0, 3.0, and 3.5 differs from the relationships between versions 1.0, 1.1, and 2.0. The .NET Framework 1.0, 1.1, and 2.0 are completely separate from one another, and one version can be present on a computer regardless of whether the other versions are present. When versions 1.0, 1.1, and 2.0 are on the same computer, each version has its own common language runtime, class libraries, compiler, and so forth. Application developers can choose which version to target. For more information, see Side-by-Side Execution, Targeting a Specific .NET Framework Version or Profile, and Using MSBuild to Target Specific Versions of the .NET Framework.