LazyOne is exactly right. As long as you have your
NameVirtualHost directives sorted out, then the first
VirtualHost defined for the ip address requested is used, if no other
ServerAlias matches exist.
Therefore, if you want to serve a 404 for each ip address that doesn't hit a host header match, create a
VirtualHost (per ip) without a
ServerAlias directive, and position it in the config so that it is loaded first. Something like:
Redirect 404 /
Redirect 404 /
Redirect / http://www.website.com
#note how there is no alias here
Update: Most of the following is cribbed from the apache2 docs, here and here.
The main config file, generally named httpd.conf, is loaded first. But if you're using a binary package on a debian-based system, there's a good chance it'll be called apache2.conf. Other config files are added using the
Include directive in the main config. Multiple uses of the
Include directive are allowed.
Include directives may use fnmatch-style wildcards to load several config files at once, in alphabetical order.
To clarify further (hopefully), your main config is loaded first. As
Include directives are encountered, they are loaded in the order they appear in the main config. If an individual
Include uses a wildcard, each config file that matches is loaded in alphabetical order.
On a debian server, apache2.conf may look something like this:
# Include module configuration:
# Include all the user configurations:
# Include ports listing
# Include generic snippets of statements
# Include the virtual host configurations:
In other words, any file ending in
.conf in mods-enabled/ is loaded before httpd.conf, which is loaded before ports.conf, which is loaded before any file in conf.d/, etc.