I'm expanding my comment into an answer in order to provide the detailed configuration in response to the OP's further inquiry. My recommendation is moving your config into VirtualHost blocks to allow for increased flexibility. Only the essential bits are shown here:
Require all denied
default can be any name or the server's IP address.
domain.com is the domain you want your server to respond to.
Explanation: When called with the IP address, apache elaborates the request with the default VirtualHost. The first VirtualHost defined is the default, regardless of its ServerName. Of course, setting the ServerName to the IP address also works, and will leave the default server free to be used otherwise. The location block denies every request to the whole VirtualHost.
Edit: I later saw that the question is tagged
apache-2.2, while my answer uses
apache-2.4 syntax. On
apache-2.2 the line
Require all denied must be substituted with two lines:
Order deny,allow and
Deny from all. Additionally, a
NameVirtualHost *:80 is required before the
Update: The question has been updated so I think it's best to provide the configuration for
apache-2.2, including all necessary lines. The same explanation as above applies.
Allow from all
Deny from all
Comment: I agree that there are other valid solutions; however, this is how I like to organize my config files, with security restraints inside
Location blocks so I know where to look for them.