Can I suggest you locate the Microsoft SDK help file MSI.CHM that is installed with Wise, and this can provide the answers to many MSI questions, including your first one.

First of all, a 1603 code is not necessarily an error. A 1603 code from windows installer can mean that a file that needs to be replaced is in use at the time of install, so a reboot will be required to complete the installation.  The other meaning of a 1603 code is a generic "a fatal error occurred during installation", but this requires you to look further into the logs for the actual cause of the error. Typically, the string Return Value 3 in the log is usually an indication of a failed action, as successful actions return a value of 1.

The WiseCleanup action is documented in the Windows Installer Editor PDF file (I know is it not customary for users to read the product documentation, but sometimes it can be time well spent.)

To save you turning to page 496, here is the extract:

If you are getting consistent errors involving the above, my initial suspicion would be that your DLL call is badly formed and is causing the install to crash.