My understanding (from Wikipedia) is that the x64 instruction set supports executing 16-bit protected mode code from long mode, but cannot execute real mode code without being switched out of long mode because long mode lacks virtual 8086 mode. Therefore, it stands to reason that real mode DOS apps can't be run in Win64 w/o software emulation or dynamic translation. However, why was support for Win16 protected-mode apps excluded when support for them seems (at least at first glance) to be reasonably implementable and is included in newer versions of Win32? Was it just a matter of demand not being high enough to justify implementation costs (and the win32 version was already implemented), or is there a good technical reason?
I don't think there are any actual Win16 protected mode apps. I understand Windows/286 and above ran in protected mode (called "standard mode" or "enhanced mode" by Microsoft) but the apps were technically real mode apps.
I remember reading that it came as a surprise to Microsoft that the Windows team figured out how to run real mode code under a protected mode operating system. But a similar solution for running real mode code in long mode does probably not exist.
So the question really comes down to a) why not switch to protected mode (or real mode) to run Win16 apps and b) why not include an emulator (like on other non-x86 NT platforms).
The answer to a) is obvious, I think, because switching between long mode and other modes is not really something the CPU supports without a reboot. OS/2 1.x had the same problem with protected mode and real mode and offered only a very inelegant solution to the problem.
The answer to b) is more difficult, but I think it comes down to a decision by Microsoft based on these three points (I love lists):
Few people still run 16 bit apps.
Those who do can run 32 bit Windows to run them.
Third party products can cover the remaining market.
And indeed there are several solutions to running 32 bit Windows (and hence 16 bit apps) under 64 bit Windows, including Microsoft's own Virtual PC.
All-in-all this was simply a decision to stop supporting compatibility with (what Microsoft consider) a legacy platform, kind of like the removal of 16 bit OS/2 compatibility in Windows XP. (Those were 16 bit protected mode apps.)
@AndrewJBrehm's answer is good, but two points I would add is that removing 16-bit support makes maintenance of the 64-bit product easier and more secure. Imagine the number of test cases built up over the years for the 16-bit subsystem. Any time you make a change you would have to ensure that nothing gets broken. More subsystems especially (relatively) rarely used ones like the 16-bit subsystem are good targets for attackers.