Which version of SharePoint 2010 are you using? Foundation, Standard, or Enterprise. Can't remember is Std has it, but I know Enterprise has the ability to open XLSX files within the browser window and display the contents (used extensively within my office). The user can then choose to download and edit the file from there. Pretty sure this is the default behaviour when you click a XLSX doc link within a document library. However, you need to configure web viewing properly for it to not error out.
Instead of clicking the link, try pulling up the menu for the document item, and select the "Edit in Microsoft Excel" option. If that opens the doc in excel properly, then it's probably a config issue with the "Excel Services Application Web Service Application". Have your SP admins take a look at it within Central Administration and see if they can configure it properly.