Is the memory which is transferred to swap due to swappiness still cached in the physical memory (until it is invalidated there)?
Yes, the kernel will know the pages are available in both places so it can reallocate the RAM if it needs to (knowing the pages are already copied to disk). If the pages are read back into RAM later they remain on disk too (unless swap space is congested and it need to be used to page out something else).
The amount of data in pages available in both RAM and in swap at any given time can be found in
The kernel is unlikely to write something to swap unless it is about to immediately reallocate the RAM they occupy though. A high swappiness values doesn't mean it will swap things out needlessly - it just means it will be more likely to swap pages out to make room for buffers/cache/similar when needed instead of clearing older blocks from cache.