First of all, I want to way that apt-get remove does not remove the config files. It just removes the indicated package (and any possible dependent package).
After installing apache2 package, you can make sure the package is installed using a command like:
$ sudo dpkg -l | grep apache2
ii apache2 2.2.20-1ubuntu1.1 Apache HTTP Server metapackage
ii apache2-mpm-worker 2.2.20-1ubuntu1.1 Apache HTTP Server - high speed threaded model
ii apache2-utils 2.2.20-1ubuntu1.1 utility programs for webservers
ii apache2.2-bin 2.2.20-1ubuntu1.1 Apache HTTP Server common binary files
ii apache2.2-common 2.2.20-1ubuntu1.1 Apache HTTP Server common files
So, there are several packages related to apache installed not just "apache2" package. You can check the installed files of a package using a command like:
$ sudo dpkg -L apache2.2-common | less
When I executed the above command, I found that the configuration files are installed by this package not "apache2" package. So, you need to make sure that this package is installed in order to find the configuration files after removing them manually.