Here's what I know so far. I'll put this here in the hopes that it's useful, but refinements are welcome.
The sum of swap space and RAM should be available for use by /tmp files. You can read more about it in the man pages for tmpfs, mount_tmpfs and swap.
Here are some relevant docs:
Oracle Solaris 11.1 Administration: Devices and File Systems
http://docs.oracle.com/cd/E26502_01/html/E29006/fsswap-29713.html
At the end of the tmpfs man page I found this:
df(1M) output is of limited accuracy since a tmpfs file sys-
tem size is not static and the space available to tmpfs is
dependent on the swap space demands of the entire system.
Oracle support document 1007407.1 suggests that the implementation reports out of space if this condition becomes true:
((RAM + swap - process-space) * .9) - space-from-files-in-tmp = 0
That seems to indicate that some sort of fudge factor is involved.
My interpretation of that formula is that running processes are allowed to get much closer to the true limit of the memory+swap. Space allocated in /tmp will start to fail when the free space in (RAM+swap) gets close to 10%. If you have large RAM (like 128G) and small default swap (like 4G), then this means that /tmp could report that it is full before any pages are allocated from the swap space at all. Nothing is wrong with that, it's just a curious twist on the traditional virtual memory algorithms and assumptions.