If I understand your question correctly, your infrastructure looks like this:
In this instance, you need to create an additional CNAME record subdomain.yourdomain.com and point it at ServerB-IP like the original A record.
Then on IIS on Server B, you need to configure it to serve a different site depending on the Host Header Name. See the following link for a full explaination:
Apologies in advance if I've guessed incorrectly, and you're not in a Windows/IIS environment.
There's another guide for this here:
If I've misread your question and your website is currently hosted entirely on Server A, but you want only a specific subdomain to redirect to server B, then you need to create a second A Record, and set it to subdomain.mydomain.com pointing at the IP for Server B. Then you install IIS on Server B and setup the subdomain web content.