Magento has a bit of a reputation for being a resource hog.
We're just about to go live with a Magento site, running on a 512MB Slicehost VPS.
Depending on the amount of memory on your VPS, and the expected traffic for the site, you may need to tune both mysql and apache. I found quite a few articles in the Slicehost forums, and elsewhere on the 'net with regard to this. Performance tuning may not be so much of an issue, if it's a low traffic site, but it's worth keeping in mind, the default settings for both apache and mysql leave plenty of room for improvement should you run into issues with memory usage initially. These sorts of decisions are not a one-size-fits-all thing, so I can't give you a bunch of magic numbers to enter info a conf file to get better performance.
I tried nginx initially, as an alternative to apache but in the end decided to go with apache, as I know it a lot better than nginx. I also didn't find the performance gains some people site as reasons for using nginx, although I will attribute that to my lack of knowledge with that server, rather than its abilities.
Once you're ready to go live with the site, make sure caching is turned back on within the admin section of Magento. During development, it's better to turn it off as this ensures any changes you are making are visible instantly, and you're not looking at cached content. It also means you don't get as good performance from Magento, so make sure you turn it back on.