Solution: I have to put the allow/deny directives inside the first Directory directive(which also happens to be for the root). I'm guessing it's because it has a AllowOverride None that does't allow any children to specify allow/deny?
<Directory /> Options FollowSymLinks AllowOverride None Order allow,deny Allow from all Deny from xxx.xx.xxx.xx </Directory>
This configuration still allows access to all IPs after apache restart
<VirtualHost *:80> ServerName www.xxx.com DocumentRoot /var/www/vhosts/xxx <Directory /var/www/vhosts/xxx> Options Indexes FollowSymLinks AllowOverride none Order deny,allow Deny from all Allow from 127.0.0.1 </Directory> </VirtualHost>