hostname.local or the FQDN. You really SHOULD be able to use the FQDN. If you can't, it means your DNS server isn't set up correctly. All Apple OS X Servers need DNS to be set up correctly on the DNS server side. This is critical to getting almost all of the services set up correctly.
Since the OS has already been installed, verify that DNS is working by doing:
sudo changeip -checkhostname
You should get:
Primary address = 10.10.10.10
Current HostName = xserve.example.com
DNS HostName = xserve.example.com
The names match. There is nothing to change.
Of course, substituting the IP address and hostname for their actual values.
I'd also recommend checking out http://consultants.apple.com/
** Update **
Your DNS is not set up correctly*. This is mandatory to set up OS X Server. Your DNS server (or whoever provides DNS) has to have a DNS record of the fqdn pointing to your server. Once that is done, run
changeip - 126.96.36.199 188.8.131.52 oldhost newhost.
oldhost would be server.local and
newhost would be whatever your FQDN is. This MUST be done before you can go any further.
Since the server seems like a fresh setup, I would suggest even trying a re-install which will give you the chance to set it up correctly during the installation process which is a bit easier than trying to go back in and doing it afterwards.
A bit of advice though, Apple markets OS X Server to be "simple" and that it will "just work". While that may be true, you still need to know the ins and outs of it for when something stops working and you're not sure what to do. If you don't, hire someone who does.
*.local is not a FQDN. It is used for Apple's Bonjour service, which is a different thing altogether. Important to know the difference.