We use NIS to allow all our users to connect to any of our hosts. However, for one particular host, I want to restrict the users that can log in to a subset of the NIS users.
Is this possible?
Yes, you can setup a netgroup and only place the users you want in that netgroup. Then on the host server that you want to restrict access you modify your nsswitch.conf accordingly and then place the netgroup in your /etc/passwd and /etc/shadow. ie:
I've done this before but can't quiet recall all the specifics. But netgroups is definitely the way you want to go. Netgroups also work well in LDAP for this purpose, if you ever decide to migrate to it some day.
Keep in mind the approach for Linux vs. Unix may differ slightly.