I believe you can do it with
serveradmin, but the syntax is horrid. The best way I know of is to edit the zone files directly (i.e. just append records to the end of them). How the zone files are organized depends on the version of OS X server you're using:
Through 10.6, there were two sets: /var/named/db.zonename. and /var/named/zones/db.zonename.zone.apple. The server admin interface manages the second files, so only add to them if you're adding record types it understands, and in the exact format it expects (and I mean exact -- I remember getting messed up once because I added records with the wrong number of spaces between fields). You can add anything you want to the first set of files, as long as it's in proper
named format (but it won't be visible in the admin interface).
Lion 10.7 keeps everything in a single set of zone files, /var/named/db/zonename. I haven't experimented with it yet, but I'd assume you can at least add entries that the admin tools will understand.