I am using RealVNC server Enterprise on the Mac.
Real VNC Enterprise Viewer on PC.
I am able to connect it properly but colors are weird, 16 colors or only blue color is showing up (see below)
How do I resolve this?
When using a VNC viewer you need to ensure you use protocol v3.3 when connecting to a Mac. Version 4.x will, at best, give unreliable results.
One application I've found that works well for VNC, RDP and a bunch of other protocols is Terminals. I have mine set up with shortcuts to each of the servers, wither they be Windows or Mac. With its tabbed windows life gets to be just a little bit easier.
The Mac wouldn't happen to be an XServe, would it? If so, connect to the Lights-Out-Management (or LOM) address instead of the OS's IP. I've seen this behavior before and that was the solution.