I have configure Virtualmin to load its library using 20M RAM but it still takes about at least 5-10secs to load every page.
How much memory does your system have? If it's less than 256MB you should not configure Virtualmin to pre-load libraries, because memory is at a premium, and pushing things into swap via pre-loading would make things slower rather than faster.
See the documentation about low memory systems here: http://www.virtualmin.com/documentation/system/low-memory
I can't think of anything else that would make Virtualmin that slow. Some pages that use a lot of data, like the usage graphs and such, might take a few seconds to load fully, but generally speaking 1 or 2 seconds is the max on a reasonably well-equipped system.
Have you checked CPU usage to see what's taking so long? If it takes 5-10 seconds per page, you ought to be able to spot the offending process or processes in top.
In short, it's not normal behavior, so there's not really anything I can tell you to do to make it faster...there is no secret "go fast" button in Virtualmin. It normally is pretty fast (though not as fast as most Webmin pages, which don't pull in and process so many different files and data sources on every page load).