You are loading the module into the Apache process, not into a specific virtual host. The "LoadModule" directive can only be used in the "server config" (think "global") context, and affects every process spawned by the first Apache process.
The primary advantages to using DSOs are (a) you can add additional modules without having to rebuild Apache, and (b) you can load only those modules that you are actually using, which may slightly cut down on Apache's memory footprint. Distributors like DSOs because they can make all features available in a modular fashion. If you're building your own Apache AND you have a stable set of requirements, there's no reason not to build everything statically. You may even get slightly faster startup times, since there's no dynamic binding going on, but this usually isn't relevant (since Apache tends to be a long-lived process, the startup time isn't terribly important).