You could use RewriteRule's [F] flag to deny the access to all but the permitted IP(s). See the following for an example (I didn't test this but I think it's correct). If you want to use the allow directive, Another way to do it is to use Rewrite directives to set an environment variable using the [E=...] flag of RewriteRule and then do Allow from env=environment_var_goes_here (see http://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#allow)
RewriteEngine on
RewriteCond %{REQUEST_URI} ^/admin.*
RewriteCond %{REMOTE_ADDR} !^x\.x\.x\.x$
RewriteRule .* - [F]
Note: I wrote that example to match any URL starting with /admin, but if you want to just match /admin, remove the .* and replace it with a $.