The help file MSI.CHM has a lot of information about custom actions, and their types, plus the return codes - for example the topic "Logging of Action Return Values" describes the numeric values entered into the MSI log (if logging is enabled) for different return scenarios.

If your intention, however, is to handle the return code programmatically, then these "standard" custom actions will not be appropriate. You would still need to have additional code to analyse the exit code of your executed program, so you would find it simpler to use a custom action such as "Run VBScript from Embedded", which would be sequenced in deferred after the installfiles action, and which would execute the EXE that your program installed, handle the return code itself, and take appropriate action if the code indicates a problem - whether that action is to continue after displaying an error message, or roll back, it's entirely up to you.

Should you just leave the native logic in windows installer to handle this, a failed install of the EXE **should** normally result in an exit code which would cause the install to roll back.