The reverse DNS records should generally be handled by your ISP (they "own" the IP number, so the reverse DNS for it gets delegated to them). Contact your ISP's support to find out how to have them set it up.
If your server is behind an NAT (network address translation, i.e. it has a private internal address as well as its public address), you'll also need to set up internal DNS (both forward and reverse) with the internal numbers instead of the public numbers. You can do this in OS X's DNS service by creating a forward zone and machine record for your server, and it'll create the reverse info automatically.
Suppose your server's full DNS name is server.example.com, and its internal IP address is 192.168.1.5. You'd create a zone named "server.example.com." (be sure to include the trailing period), and add an entry to its Nameservers list with both the Zone and Nameserver Hostname set to "server.example.com.". Then, add a Machine (A) record to that zone, with the Machine name "server.example.com." (again, the trailing period is very important), and the IP address 192.168.1.5.
Once that's set, start the DNS service, and configure your server's network preferences to use 192.168.1.5 as its DNS server. If you have other computers on the network with manual IP setups, change them similarly. If you have a DHCP server running, set it to hand out 192.168.1.5 as the DNS server. That should do the trick.