OS X Server (at least 10.8, I haven't looked at 10.7) doesn't seem to have a way to create a TXT record that's not accompanied by an A record. To create A and TXT records for the same domain name, make sure the DNS pane is in "Show All Records" mode (use the "action"/gear pop-up menu to switch it), then choose add (plus sign) pop-up menu > Add Machine Record, and fill in the Host Name, IP Addresses (A), and Text (TXT) fields appropriately. Unfortunately, it won't let you save the entry if you fill in the Text field but leave the address blank.
I was able to get it to serve a bare TXT record by adding an A+TXT "machine" record, then editing the zone file (/var/named/db.zonename) to remove the A record. This seemed to work, and it didn't even get overwritten when I edited the zone in Server.app, but I wouldn't trust this in production... Also, my attempt to simply add a TXT entry to the zone file manually caused servermgrd (the daemon that Server.app talks to go get/set server settings) to crash repeatedly. If you do any manual messing with the zone files, I recommend backing up /var/named first.