I'd like to use hg or git for repo with size in about 5gb. Are they work proper on repo with such a big size?
A repository with 5 GB is large, but not outrageously large. You don't say if the 5 GB is for the history alone, the working copy, or a combined figure?
For comparison, I can tell you that the OpenOffice repository has has a history of 2.0 GB (about 270,000 changesets) and a working copy of 2.3 GB. Mercurial runs okay on that size:
With a cold cache is takes 2.4 sec — a little longer, but not too bad. There are 69,000 files in the working copy.
In general, you can expect both Git and Mercurial to slow down as the repository grows. The complexity of the operations differ:
I have never used hg but facebook recently sent a post about their git repo slowing down. I think their issue was all their "apps" are in one repo
It's worth a read through for people trying to help them out