The first thing to check is:
- Is you virtual host on the bridge network ( probably xenbr0 ) or on the private nat network ( probably virbr0 ) ?
You probably want to be on the bridged network in this kind of situation ( access to the vm from the outside ). On the bridged network you can give your VM an ip address in the same subnet as the host system ( or on a vlan you configured on your host, but I suppose this is too complicated for what you want to do ).
Then if the Host network is the public network (hopefully not), you can directly assign the virtual server its public address. Else you have to configure the router/firewall above your host system in order to nat the public ip address you want to use to the internal address on your Host network.
If this is not clear please provide more information about your ip addressing.