It sounds like this question is about firewalling rather than IPSEC. If you are only allowing IPSEC secured connections then you would simply only have an IPSEC SA (Security Association) for the chosen client. This would not necessarily need to be tied to a specific IP and would be a good choice if you do not know the IP that you will be accessing the server from. Any client you do not have an SA for will not be able to get access.
If you know your IP address it is more likely what you want is to install either a software firewall on the client server or a hardware firewall in front of it that only allow access from your nominated IP.
You could also combine the approach so that your management traffic is encrypted in transit using IPSEC and the server is protected to only accept traffic from your host by a firewall. IPSEC encryption may not be needed if the management already uses an encrypted channel (e.g. SSH or HTTPS).