The coresettings.config is an .xml file you can edit with notepad (although I strongly suggest using a TRUE .xml editor like notepad ++) . It is located in the Install Drive\Program Files\Altiris\Notification Server\Config directory.
The error you are seeing is that somehow the database server\database you are specify in setup cannot be determined, or has not been entered into this file, so post install configuration or setup cannot continue.
So a couple of things to check are 1) How did you install Altiris and SQL? - were you logged onto the server, or did you use terminal service session 0 (mstsc /admin)?
2)What OS (server) and database version are you using? 3) what authentication did you used (mixed mode)? 4) did the database actually get created during setup?