I'm about to install SQL Server 2008 R2 on a Server 2008 R2 box running RDS. Do I need to install it using install mode like I would an application? Having asked that what determines whether you use install mode or not? Would I need to install an antivirus in install mode? Thanks for helping out a beginner.
Yes you should install SQL Server while the RDS server is in install mode. Any software that writes to the HKCU registry hive needs to be installed in install mode. SQL Server writes to the HKCU registry hive and therefore should be installed in install mode.
As a caveat, if you default to always installing software in install mode there's no harm for software that doesn't write to HKCU.
Also, here's a good article on the subject. It's a little dated but still holds true as far as I know.