I got same thoughs about mixing Perc hardware RAID and ZFS, I was thinking about splitting backplane or doing other nasty procedures. From one side I needed hardware raid volumes, from other side I needed to import existing ZFS pool.
I have readed that H730 can be set up for either HBA or RAID mode. However that's not true, at least for latest firmware.
I have tested on R730, H730p mini mono and ESXI 7- everything works perfect. No need to create virtual disk. Passthrough works pretty cool!
Also, on ESXi 7 no more RDM passthrough needed!
Step 1. Upgrade PERC firmare to latest 25.5.7.005. Set it to RAID mode.
Step 2. Create Virtual Disk for needed hardware raid layout.
Step 3. Mark needed ZFS disks as Non-Raid ones.
Step 4. Im not sure if You can disable write cache for Non-Raid disks via BIOS, but I have suceeed disabling non-raid cache through OpenManage (of course cache still enabled for HW RAID):
H730p openmanage settings
Step 5. Non-raid disks and RAID disks will be normally visible on ESXi. Of course no problems with hotplugging.
Detected devices by vSphere
Step 6. Create/import FreeNAS VM. Edit VM settings, click Add Hard Disk -> New Raw Disk. Non-VMFS raw devices will be shown, and You can add them one-by one as passthrough:
New one-click RDM passthrough
Remember to leave Disk Compatibility: Physical
Step 7. Boot FreeNas, all passthrough disks will be visible as directly attached, including disks serial numbers, APM managment, etc:
PERC h730p nonraid disks in FreeNAS
Step 8. Use shell and test smartctl:
Disks are directly visible by FreeNas. Also, I recommend upgrading to TrueNas 12.0 - iSCSI speed and stability improvments are really big step forward. No more ctl_datamove aborted errors.
You can also hotplug any other disk, set it to nonraid via iDrac and then just on-live Edit VM options and add new RAW device, it will be detected and attached to FreeNAS immediatly without rebooting of course.
I have readed too much on forums that's not possible... not sure if new firmware added such capability, but actually I am more than happy. System is very flexible that way.
From one side iDrac is fully helath-monitoring created RAID Virtual disk (incl. patrol read and all), and from other side non-perc RAID disks are monitored via FreeNAS. Also, because thats 1:1 passthrough, of course disks temperatures are monitored too.
Small tip: Don't forget to disable "Delayed ACK" when using ESXi software iscsi client. Feed FreeNas with decent amount of RAM. After disabling Delayed Ack, and upgrading NAS to v12.0 (From 11.3 Freenas to TrueNAS) actual iSCSI backup times dropped from 11+ hours to 3.5 hrs.
It will look like this from iDrac perspective:
However, remember that Write Cache can't be disabled for Non-Raid disks via iDrac, there is no such setting in controller settings. So OpenManage (confirmed), or bios maybe.