There are probably several things working against you:
First, upgrade your version. 3.1.2 is out and 3.1.4 is in beta (at the time of this posting).
Second, have you looked in the log files VirtualBox generates? It will likely give you (or those trying to help) clues as to what is happening. They usually live where the virtual machine XML configuration file lives.
Third, as suggested, remove the existing network adapter in your VM configuration and re-add it. Or even better, add a different network adapter type (Maybe Intel instead of PCNet or vice versa).
Fourth, try statically assigning an IP address in your guest as to minimize the host networking issues.
Fifth, make sure the various VirtualBox adapters in your HOST are enabled and configured right. I am not entirely certain what it looks like in OS X land, but under Windows VirtualBox adds a "VirtualBox Host-Only Network" that if you disable or generally muck with, it causes no end to weird problems. On that same line, it would be worth checking to make sure none of your private subnets are conflicting with each other either.
