Dbunload works fine, but only if you free space in database. In order to make this, you need to delete rows in BINARY_FILE and ALERTS tables.
You can mount the sem5 database using DBSERV9.EXE located in \ASA\WIN32 directory. At the same time, create a System DSN odbc using the Adaptive Server Anywhere Drivers. Only supply dba user credentials and ip address.
Then using DBISQLC.EXE util, you can delete them.
Execute querys:
delete from BINARY_FILE
AND
delete from ALERTS
it tooks over 3 hours, so be patient.
After running dbunload.exe you can reduce database size minor to 1 GB.
In my case, it reduced 35 GB to 700 MB database size. Then, the small database was migrated to SQL Express 2008.