Solution: I have to put the allow/deny directives inside the first Directory directive(which also happens to be for the root). I'm guessing it's because it has a AllowOverride None that does't allow any children to specify allow/deny?
<Directory /> Options FollowSymLinks AllowOverride None Order allow,deny Allow from all Deny from xxx.xx.xxx.xx </Directory>
Original:
This configuration still allows access to all IPs after apache restart
<VirtualHost *:80>
ServerName www.xxx.com
DocumentRoot /var/www/vhosts/xxx
<Directory /var/www/vhosts/xxx>
Options Indexes FollowSymLinks
AllowOverride none
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Directory>
</VirtualHost>