Please don't take offense to this but I strongly suggest you bring in a local area IT consulting firm that specializes in systems and network administration. I also came from a programming background many moons ago and learned many hard lessons on the do's and don'ts of managing a networked server environment. I (thankfully) had alot of mentors and help over the years, because without it, who knows what kind of smoldering wreckage would be left behind.
Moving right along now to your original question: I see two mistakes, one being Linux: don't get me wrong, I love Linux and use it in all kinds of various roles, but as a sole server in a small company that (again, no offense) doesn't have a full-time sysadmin is asking for trouble. Finding competent Linux administrators (and it's even harder to find ones that follow best practices) is not easy. Down the road, if you leave or you hire a new person to take over your duties, who's going to look after it?
Assuming you're under 75 end-users, I would strongly recommend Microsoft Small Business Server 2011 Standard on solid tier-1 hardware (like Dell, HP, IBM) with a 3-year on-site/4-hour replacement warranty. Have at least a RAID 1 mirrored array for the data (and another for the system if you can afford it). Get at least 8GB of RAM, 12GB is better. Invest in an offline/off-site backup: you can start with a couple of external drives or a tape drive, but something you can take off-site with you every night.
I'm also not sold on your suggestion for a custom "all-in-one" database: there are so many better, more viable software options out there, that unless you have some very specific niche requirements that only a custom solution can provide, you'd be much better off using a well-supported 3rd-party offering. You have to resist the "I can write something" programmer urges and think about supporting this solution long-term.
And finally, I think you and your employer need to decide what you role is going to be at this company. It sounds like you're new there and while you're right, they likely do need to upgrade their systems, you don't want to bite off more than you can chew and fail to provide whatever it was that you were hired there to do.
There's a lot of opinions floating around right now, so I'm going to take a step back and hopefully provide some platform-agnostic advice that will be of use to you regardless of what you end up going with:
Do a complete inventory of all systems and devices; check warranty status of hardware (if it's a Dell, IBM, etc. you should be able use the service tag to get a warranty check; if it's a white box server, they may still have some sort of identifier, but you'll have to call to find out what the status is most likely).
Do a complete inventory of data: Don't trust that they have no data on their C: drives; they probably do, actually they probably have PST files all over the place of old mail. Find out what's critical, what's being backed up, what's not being backed up, how it's being backed up and whether anything is taken off-site or not. FIX THIS FIRST. RIGHT NOW. If they have no backup setup, go buy an external USB drive at a Big Box store for now and use NTBackup (it's likely on that server already) and do a full backup and take it off-site with you. If they have backup in place, go do a test restore (see below).
Check patch levels on all systems (get #2 sorted out first!): not just Windows Updates, but Java and all Adobe products especially and update accordingly (might want to do #4 first so you know what machines are higher-priority than others. i.e. that workstation for the part-time staff member could stand a botched update much more than the accountant who cuts the pay checks).
Talk to your users: find out what's working well, what's not working, get a feel for everyone's level of change tolerance, their comfort level with IT (you may be recruiting a helper to get things in order), and any wish lists they may have. Understand their business processes; as a sysadmin, your priority should be ensuring that the systems the business depends on to function are working in good order and to do that, you need to know how everyone uses those systems.
After #1, you should have an idea of how the network's setup. Look for any old hubs that can be replaced; you'll want at least 10/100 everywhere, switch-wise. Check the firewall/router (make sure there is one), check for any open wi-fi access points, etc.
If you do go the Linux route, stick with a distro that's well-supported by the community (Ubuntu would be a good choice) and set it up on whatever hardware you can afford (as you know, a LAMP box could be an off-lease P4 workstation for now) and as isolated from the currently-working system as possible. As a learning exercise (and could pay huge dividends in a disaster recovery scenario), try to get the core applications that are running on the current server working on another Windows box first -- use your full backup you did in #2 to do a test restore; have fun with that :)
As for your test setup, you can opt to buy something beefy with lots of RAM and then you can virtualize (ESXi is free, so is XenServer, so is VirtualBox) but if the current server is Windows 2003 or older, you can likely get that FoxPro application working on an off-lease Windows XP workstation for cheap.
Now pat yourself on the back; you now have good backups; you also did a test restore and now have a better understanding of how everything works together. You also likely have a (long) priority TODO list that'll keep you busy for the foreseeable future.
Oh and when that's all done, you now have a test environment you can start building your Utopian "dream" system... or maybe take a vacation :)