PCI passthrough is supported on the devices you have. You'll need to update the firmware of all devices. Do this with all devices connected and by running the HP Service Pack for ProLiant bootable DVD. Your PSOD error may have been resolved in a later build of ESXi. Don't just use what you've downloaded from VMware. You'll want to update to the latest build from the patch download site.
I think using the Smart Array P812 controller is a mistake in a passthrough configuration, though. By dedicating it to a single virtual machine, you've added complexity to the setup with no performance or manageability advantage. In addition, you'll need to monitor the storage hardware from the file server VM as well as at the ESXi level.
A better solution would be to use one or both controllers (since the P812 can address your internal disks, too) and create multiple HP logical drives, placing the file server's data in standard VMDK files. With that, you'll have a single management plane.
But ensure all updates are in place before trying the passthrough again.