Hello Nobby,
The targetting of the Inventory Task should be against the ESX/ESXi host as this is the host you intend to gather the inventory from though the actual inventory task once run will run against the associated vCenter in which the host was discovered via network discovery, if you look at the vCenter/Host/Guest tree the Inventory task will run against the appropriate vCenter.
I am not sure how you did the discovery, did you discover via network or via vCenter. Did you try running the Inventory Task targetting the Hosts and what was the results?
I would highly suggest reaching out to Technical Support as they can provide more indepth troubleshooting methods to quickly identify where the process is failing.
Kind Regards,