I'm trying to determine if Microsoft Data Protection Manager will function on a Hyper-V host running Windows 2008 R2 Server Core. I've found this document in the documentation, but it still unclear which components of DPM will work. Or if I can install the management tools on a separate VM. Or do I install DPM into a VM and then install the protection agent onto the host?
The short answer is no, you cannot install DPM on Server Core. One of the big reasons for this is that Server Core supports some, but not all of the same .NET namespaces as a full installation of Windows Server.
Here's a better list of supported OSes for you: http://blogs.technet.com/b/askcore/archive/2008/05/02/data-protection-manager-supported-operating-systems.aspx
It is very likely that a future version of DPM will support being installed on Core. All of these products are fairly new and ever-evolving.
Now on the other hand, if you're talking about installing DPM on a Server 2008 R2 Standard or Enterprise server that is running as a virtual machine on a Server Core Hyper-V host, then sure, why not? Just remember to use pass-through disks and such to keep disk performance acceptable, etc.