There is nothing to "compile" in Webmin. It is written in Perl, with some client-side JavaScript and Java. setup.sh is an install script that puts the files in the right places, enables the webmin service on boot, etc.
There is an uninstall script generated when you run setup.sh, which will reside in the Webmin directory (wherever you told Webmin to install itself when you ran setup.sh), called "uninstall.sh". But, if the install hung very early in the process this script may not have been generated (it has to be generated because the locations are not known until you tell it where to put things, and it'd be annoying to have to know where you installed it when you want to uninstall it). I suspect such a script exists, though. There isn't a lot that can go wrong during the install; you've not given us any clues about when in the process it hung. I just haven't seen a Webmin install fail in years, except in cases of pathological server problems (failing memory, failing disks, no disk space, not enough memory, etc.).
I would recommend you install Webmin using the apt repository Jamie provides at http://www.webmin.com, or if you're also using Virtualmin or Cloudmin, use the repositories at http://www.virtualmin.com. Using native packages just makes more sense, and provides many additional tools for keeping up with what is installed on your server, and makes removing stuff easier and cleaner. It also makes your install more predictable; if I sit down at a Debian system where Webmin has been installed from our package, I know Webmin lives in /usr/share/webmin. If I sit down at a system installed from the tarball, I have to guess where it lives, or go dig into the initscript to see where it points to.
I agree with sybreon...you're trying to solve the wrong problem here. The problem is you don't know why Webmin failed to install. Installing it again will just keep failing until you understand why it's failing and resolve that problem.
Anyway, to answer your question, if you don't have an uninstall.sh script in the Webmin directory:
Remove the Webmin directory. This is whatever you told it to use. Maybe /usr/local/webmin, maybe /opt/webmin, maybe /usr/share/webmin. Probably the first of those.
Disable the Webmin service on boot using the Debian update-rc.sh script (I think that's how it's done on Debian, anyway)
Remove the Webmin initscript (/etc/init.d/webmin)
Remove any Webmin cronjobs; or, if you're going to reinstall Webmin, you can just leave them and double check to be sure there aren't any extraneous ones using the crontab editor in Webmin. I think it will skip adding cronjobs if they already exist, even if it doesn't know it's an upgrade.
That's pretty much it. Webmin isn't compiled, doesn't touch any system directories except creating the startup script, and doesn't leave stuff lying around at random on your system.
