In the Node API docs, I think that "hostname" is actually the wrong term. They really mean IP Address. If you use a virtual host name on a server with a single IP address (the typical VPS for example), you will still get responses on both host names.
So setting the hostname on server.listen wont work for you unless you have multiple IP addresses and can assign the sub-domain to one of them.
I think that you have two choices to make this work.
The first would be to restrict Node to only listen on localhost (internal) and use a reverse proxy to handle the incoming request. The proxy would forward the request to the internal node listener. This is not always totally robust as the reverse proxy can be rather picky about static resources such as image files. Additionally, most reverse proxies are "blocking" and so may undo the benefits of using Node. It is also possible to use Node with the http-proxy library to do the proxying.
The second approach is to use some middleware to handle the VHOST. Express is commonly used for this though the capability is actually provided by the Connect middleware library which you could use directly. See: https://github.com/visionmedia/express/tree/master/examples/vhost