Try googling on the error - there are many people who have encountered the same error, and many discussion threads, such as this one:
http://community.flexerasoftware.com/archive/index.php?t-154479.html

What application are you repackaging by the way, and if you are repackaging, why do you need a transform?
Do you have the correct .NET framework installed for this application?

Updated information:
I've been going through the logs and it seems like the problem has to do with missing network connections.
If I run the installation manually I need to specify two network paths, to a file server and to an application server, during the installation process.
It Seems like the utils the application is looking for is on one of these servers, and my log says:
"Error 1606.Could not access network location ..." and "Error 1314.The specified path servershareis unavailable."
What is the best way to repackage an application that asks for a network path to an external server during installation?
I map the needed servershares in front of the installation, and add the value manually during a manual installation, but I don't know if I need to do this as a "script action" when the installation is silent?
Any advice, please?

I can see I have posted this to the wrong forum, moving the post to Wise Application Packaging forum

If you open the SDK help file MSI.CHM, and search on Custom Action In-Script Execution Options you will find that custom actions can be run in either system or user context.
In order for a custom action to be able to access network resources, it needs to run in user context, as it then has access to the same resources as the current user.
If the action runs in system context, then it has no access to network resources, as the local system account has no domain privileges.

Do bear in mind, however, that a custom action running in user context has the same "access rights" as the user, so if the user happens to be a standard user, for example, the custom action will not be able to make changes to HKLM, or write to program files, as the standard user only has read privileges there.
So depending on what you need to accomplish, you may need to run one custom action in user context, and another in system context.

I'm sorry, there was a spelling mistake in my mst file, so the mst did not run when I wrote the last post.
When the spellingmistake was taken care of, the original errormessag came back: "1001 InstallUtilLib.dll unknown error".
This is a part of the log:

MSI (s) (C8:3C) [13:52:30:655]: Executing op: IniWriteRemoveValue(Section=General,Key=UNICODELOGIN,Value=TRUE,Mode=0)  WriteIniValues: File: [i]application[/i].ini, Section: General, Key: UNICODELOGIN, Value: TRUE  MSI (s) (C8:3C) [13:52:30:655]: Executing op: IniWriteRemoveValue(Section=General,Key=RootUrl,Value=http://[i]application_server_share[/i],Mode=0)  WriteIniValues: File: [i]application[/i].ini, Section: General, Key: RootUrl, Value: http://[i]application_server_share[/i]  MSI (s) (C8:3C) [13:52:30:655]: Executing op: ActionStart(Name=ISSelfRegisterFiles,,)  Action 13:52:30: ISSelfRegisterFiles.  MSI (s) (C8:3C) [13:52:30:701]: Executing op: CustomActionSchedule(Action=ISSelfRegisterFiles,ActionType=3073,Source=BinaryData,Target=ISSelfRegisterFiles,CustomActionData=M:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\~5B.tmp)  MSI (s) (C8:C0) [13:52:30:701]: Invoking remote custom action. DLL: M:\WINDOWS\Installer\MSI5E.tmp, Entrypoint: ISSelfRegisterFiles  InstallShield 13:52:30: Initializing Property Bag...  InstallShield 13:52:30: Getting file count from property bag  InstallShield 13:52:30: File Count : 1  InstallShield 13:52:30: Sorting Based On Order...  InstallShield 13:52:30: This setup is running on a 32-bit Windows...No need to load ISBEW64.exe  InstallShield 13:52:30: Registering file M:\Program Files\[i]vendor_name[/i]\Shared\https50.ocx (32-bit)  InstallShield 13:52:30: Begin Comitting Property Bag  InstallShield 13:52:30: Write KeyList count  InstallShield 13:52:30: Finished Comitting Property Bag  MSI (s) (C8:3C) [13:52:30:967]: Executing op: ActionStart(Name=_F33A3DBE5EC2CB759D64A895403FD778.commit,,)  Action 13:52:30: _F33A3DBE5EC2CB759D64A895403FD778.commit.  MSI (s) (C8:3C) [13:52:30:967]: Executing op: CustomActionSchedule(Action=_F33A3DBE5EC2CB759D64A895403FD778.commit,ActionType=3585,Source=BinaryData,Target=ManagedInstall,CustomActionData=/installtype=notransaction /action=commit /LogFile= "M:\Program Files\[i]vendor_name[/i]\[i]application_name[/i]\Bin\BWMSAPICategory52_NET.dll" "M:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{3BEE632D-E423-4EBD-B983-406B73B6BCFB}\_isconfig.xml")  MSI (s) (C8:3C) [13:52:30:967]: Executing op: ActionStart(Name=_F33A3DBE5EC2CB759D64A895403FD778.rollback,,)  Action 13:52:30: _F33A3DBE5EC2CB759D64A895403FD778.rollback.  MSI (s) (C8:3C) [13:52:30:967]: Executing op: CustomActionSchedule(Action=_F33A3DBE5EC2CB759D64A895403FD778.rollback,ActionType=3329,Source=BinaryData,Target=ManagedInstall,CustomActionData=/installtype=notransaction /action=rollback /LogFile= "M:\Program Files\[i]vendor_name[/i]\[i]application_name[/i]\Bin\BWMSAPICategory52_NET.dll" "M:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{3BEE632D-E423-4EBD-B983-406B73B6BCFB}\_isconfig.xml")  MSI (s) (C8:3C) [13:52:30:967]: Executing op: ActionStart(Name=_F33A3DBE5EC2CB759D64A895403FD778.install,,)  Action 13:52:30: _F33A3DBE5EC2CB759D64A895403FD778.install.  MSI (s) (C8:3C) [13:52:30:983]: Executing op: CustomActionSchedule(Action=_F33A3DBE5EC2CB759D64A895403FD778.install,ActionType=3073,Source=BinaryData,Target=ManagedInstall,CustomActionData=/installtype=notransaction /action=install /LogFile= "M:\Program Files\[i]vendor_name[/i]\[i]application_name[/i]\Bin\BWMSAPICategory52_NET.dll" "M:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{3BEE632D-E423-4EBD-B983-406B73B6BCFB}\_isconfig.xml")  MSI (s) (C8:40) [13:52:30:983]: Invoking remote custom action. DLL: M:\WINDOWS\Installer\MSI5F.tmp, Entrypoint: ManagedInstall  Info 2835.The control ErrorIcon was not found on dialog SetupError.  Error 1001.InstallUtilLib.dll: Unknown error.  MSI (s) (C8!DC) [13:54:23:248]:  MSI (s) (C8:40) [13:54:23:310]: Leaked MSIHANDLE (48) of type 790531 for thread 5852  MSI (s) (C8:40) [13:54:23:310]: Note: 1: 2769 2: _F33A3DBE5EC2CB759D64A895403FD778.install 3: 1  Info 2769.Custom Action _F33A3DBE5EC2CB759D64A895403FD778.install did not close 1 MSIHANDLEs.  Action ended 13:54:23: InstallFinalize. Return value 3.

Since you are clearly running a vendor MSI written with Installshield, I strongly recommend that you investigate the application packaging library at appdeploy.com to see if your particular app is covered.
There are also forums on this site where issues predominantly affecting Installshield apps are discussed - as well as the forums on the Installshield site (now Accresso ??).

The Installshield action ISSelfRegisterFiles also raises some questions in my mind, as it's not regarded as good practice to use self registration with files.

Out of interest, does the basic vendor MSI install OK without your transform?

...to your AppDeploy post www.appdeploy.com/messageboards/tm.asp

thank you, folks, I'll continue this thread on the appdeploy post:
http://www.appdeploy.com/messageboards/tm.asp?m=56566&mpage=1&key=&#56566

would there be any cases where registrypermissions would be a problem running an msi with a transform file when the msi without a transform file would run fine?
the msi runs fine without the transform.
don't think I'll find anything about this vendor or application in any forums... it's not a big company and they are making updates and small adjustmenst as we go along ... :-(

if you care to see the logs from a successful installation (without transform) and a log from a non successful installation (with transform) I could send you the logs by e-mail. I don't want to post them on the web...

To respond to your above question, it totally depends on what your transform is doing. If the transform is trying to write to an area in the registry that the installing user has no write permissions for, then it's not going to work. Transforms can not only modify existing settings, but can add new settings and new files as well, so permissions can always become an issue to consider.

 The transform is quite simple. It just add 2-3 networkshares.
I know the application has .net components which probably get registered with Regasm.

Running the msi without transform runs fine. With transform - error.

I have few clues what to do next now...

Did you read my earlier posting about system v user context custom actions?
If your actions fail due to lack of access or privileges to the network shares then the error return will cause the entire install to fail. So perhaps this gives you a clue as to where to look first.

 Yes, beeing an amateur still, I can't say for sure, but I don't think that is the problem because I've tried to run the install as a domain admin with full adminstrative permissions on the local machines, and still the problem persists.
I guess the problem appears when it's trying to register components in registry because running the installation non-silent shows that it stops when the installation is at the stage "Writing system registry values".
I've asked the software supplier about certain .net dll's, and they say it's registering the files on the local server, so I guess the networksshares provided won't be an issue.
I've logged the msi installation with and without the transform, if you care to have a look I can send you both log files.
Maybe it will be easy for you to spot the problem whereas me not being an expert in application packaging can't find a solution.

also, in the release notes of the application it states that if I get the InstallUtilLib.dll error I need to install .net framework 3.5. But, running the msi without the transform works fine with .net framework 2.0, so I can't see why the transform should need .net framework 3.5 as long as the msi don't need it... 

I have a simple philosophy, based on experience. If the vendor information states something explicitly, then you need to try it, regardless of "logic", as you are going to kick yourself if it proves to be something as simple as ignoring vendor documentation that wastes hours of your time.
It doesn't take long to install 3.5 and find out if that is the issue, or not, especially if you are using a virtual machine for testing and can roll back to a previous snapshot when testing is finished.

 Installed .net framework 3.5, SP1 - same result :-(

OK, so at least you now know that its not a .NET framework issue.
Can you tell us why you need to map network drives from within your transform?
I have never seen anyone do this and I remain convinced that this is contributing to your problems.
How about mapping the network drives using a login script, for example, and then try the install with your transform edited to remove the network mapping scripts.
Since you know that something in your transform is causing the problem, now is the time to start editing the transform to test your changes one by one until the exact cause is identified.

 It's an workstation installation that has to be run on a terminal server. It connects to a seperate application and fileserver for the application.
The version prior to this worked fine except from the fact that I had to use wisecomcapture to add some files to registry.
I mapped the drives prior to starting the installation - alle from the same command window, so they should be available.
could it be wise adding rubbish to the transform?

 Yep - that's what I've done now - and it seem to works.
I'm glad that the changes that I needed in the transform was fairly easy, otherwise I don't think i would be able to make a transform this way.
Is there any good tips about using orca for beginners? And is there a way to find out when it is Wise itself causing the errors?

I always set the MSI to read only before creating transforms, as that avoids a lot of issues when using Wise, (which adds tables to the MSI otherwise).
I use ORCA for transforms as it is very clean and only adds what you tell it to. There is no user guide for it that I am aware of. However, if you need to make changes you are unsure of, you can always make them in Wise, then note the relevant table changes that Wise makes and choose the ones that you need to implement in ORCA.

 Thank you for useful input! I used the wise transform package as a "template" to fill in the necessary information in Orca. There is a lot of tables that I don't have any knowledge to, so very few changes were done in orca compared to Wise. I might have added to litle information in the orca transform, cause there are two minor errors: after testing the transform I tried an installation on a x64 server. First time I ran the msi and transform from a network drive. I got an errormessage that it couldn't find the transform even though the transform was in the same directory as the msi, so I put it on the local drive as I continued testing. Now it keeps on asking for the network path I used when I first time tried to install it. Where in orca do i correct this so it doesn't ask for the old path?

Some experiences from the problems that appeared in Wise: 
The transform made with installtailor made strange directory structures under program files, caused by custom actions removing some backslashes and this resulted in wrong names on the folders and the structure being wrong. All the "installTailorPropertyxxx" looked ok, but then later a lot of custom actions followed, destroying the directory structure. In the version pre, I had do delete serveral custom actions do get rid of this problem.
Any ideas of how this could have been done different if I were to use Wise more "correct"?

The installation paths are not stored in the MSI or MST but are recorded in the registry when the MSI is installed. To get rid of the call to the network location you need to uninstall the MSI completely then install again. If this does not work, track down a utility called MSIZAP on the net, and use this to clean up the install of your MSI and start again. Hopefully you have not used a production server as a test platform for an unproven MSI - that's what virtual machines are for.

Regarding your problems with InstallTailor, I cannot really offer you any suggestions as I don't totally understand what you have done, except that you appear to have had problems with faulty custom actions removing backslashes. The obvious advice is to write better custom actions that don't cause errors, as this is not really an issue with incorrect use of Wise, but incorrect coding of custom actions.

...so they must have come from elsewhere. I have used InstalTailor to create transforms for vendor MSIs since the dawn of time and have never had an issue of any significance with it.

Comments about the orca part:
I'm using virtual machines for this, and they get rolled back for every new testrun. As soon as I get an succesful installation I rewind to a "clean snapshot" before I try running the msi and the mst again, so there should be no letfovers.
I guess the problem has something to do with sourcelist and advertising, I will check it out

Comments about the Wise part:

The paths is not in cleartext in the msi or mst, but I find them in the log.

This is an extract from the log which shows an example:
(The numbers in the beginning is the line number)

"I always set the MSI to read only before creating transforms, as that avoids a lot of issues when using Wise, (which adds tables to the MSI otherwise)."

Maybe my msi file was already "polluted" when I started Install Tailor... ? (eventhough the same msi file works fine without the transform)

Have to mention that the msi file I use is extracted from an exe fil.

VBScab, who knows about these things, tells us that InstallTailor does not insert any custom actions.
Your log does appear to reference a DA custom action - is this one of yours?

However, if you have it running with ORCA, then it's probably not worth spending time pursuing this one, unless as an intellectual exercise. In that case, you need to source a clean copy of the MSI to ensure that the one you are working with is not "polluted" from a previous operation, and then monitor changes carefully.

 None of the custom actions are mine...  where do they come from..?!
It wouldn't be a problem if it wasn't for the thing that there is several applications from this supplier to come, (they need to be repackaged within limited time) and some of them needs more complicated transforms.
If using Wise cause me this kind of trouble repackaging the next applications I won't be able to make these transforms because it will be to complicated for me to do the needed changes in Orca ..

I will try to extract the msi file again, writeprotect it and see if I get less problems.

 tried extracting the file again, read only, made the transform. Same result.
Installs fine the first time, when trying to uninstall the original error message comes again.
Installtailor shows the there are a lot of catpured changes- eventhough I've hardly done any changes at all.
I will try to remove all actions except from the two changes I made ...

What is the operating system you are installing to?
A known issue with versions of windows installer prior to the version shipping with Windows 7, is that the locally cached MSI, when it is stripped of internal cabs (standard behaviour), also loses its code signing. This causes the uninstall to run into issues as the locally cached MSI handles the uninstall.

With V5 of windows installer, the entire MSI is cached to get around this problem.

Microsoft recommendations are to leave any installed system components in place when uninstalling.

 OS is windows xP/Server 2003 std. 
I did another try with Wise.
I had a look at the captured changes, and there was 98 of them.
I removed all of them except from the few changes I'd done, and it installed successfully.
When I ran the installation silent it  failed: Error 1606.Could not access network location Utils.
I will get an expert coming to help me out having a look at it in a couple of weeks and I'll update the forum when I got a solution
Thanks to you both for all the help so far!

No final solution, the consulent I hired didn't have any good explanation for the problems I experienced with this msi package.
Maybe it's a rotten msi package from the vendor.
Anyway, I will try Orca instead, and will open a new post for that - guess I will need some help...
thanx to all who have contributed

...tell me...[quote] the consulent I hired didn't have any good explanation for the problems I experienced with this msi package[/quote] you didn't pay this idiot, did you? If you did, can I come and consult for you? It sounds like easy money. In the meantime, let everyone know the name so we can advise clients to avoid him/his company.

AngelD - Sweden
VBScab - UK
EdT - UK

Still too far away I'm afraid  ..