Ghost Solution Suite

Sometime last night, before our regular GhostCast maintenance cycle executed, the Ghost Configuration Server encountered an exception. It spit a backtrace at the screen and, ever since, every GhostCast I've tried (be it executing a task or just loading the GhostCast server) has absolutely refused to accept connections from the hosts. Ghost is running on a Windows Server 2003 Enterprise box with only ports 25, 6666, 6667, 6668 and 6669 blocked.

While I'm interested in finding the cause, my main concern at the moment is that we can no longer restore any images, so our maintenance is completely gone.

The backtrace is as follows:
Symantec Ghost Configuration Server
An exception has occurred of type c0000005
D:\Program Files\Symantec\Ghost\ngserver.exe 8.2.0.1117

e:\corporate\trunk\configureagent\server\database.cpp#1334:l_loadTask
unsigned int taskId = 0x48c638
unsigned int taskViewId = 0x47a878
Library::Gc::SHashMap * policyOverrides = 0xe9ff78
unsigned int compress = 0x1e46bc0
unsigned int stepId = 0
bool install = true
bool dos = false
MoveUserInfo * moveInfo = 0x14f055c
Library::Gc::SHashMap * policies = 0x405f01
unsigned int overwriteFlag = 11
unsigned int incrementalType = 0x4c4905e
ConfigInfo * mainConfig = 0x14f6651
bool useDefaultConfig = true
unsigned int dynamicCollectionId = 0x1e49e00
ViewItem * clientView = 0x4bf71327
Library::Gc::SHashMap * dynamicCollectionResults = 0x43f274
unsigned int sysprepRunMiniSetup = 0x4c2fc13
Library::Gc::SQueue * dosCommandSet = 0x14f12d8
int target = 0xe9fd18
Library::Gc::SQueue * winFileSet = 0x368
unsigned int local = 0x4c2ef9f
unsigned int putOverwrite = 0x7c821364
unsigned int wakeOption = 0x7ffdc000
unsigned int deployOption = 0x77e424a1
unsigned int removeFlag = 0x4c40b45
Library::Gc::SHashMap * configMap = 0x14f1201
Library::Gc::SQueue * installPackageSet = 0x14f21ec
Library::Gc::SQueue * winCommandSet = 0x45fc4b
bool casual = true
unsigned int partitionFlag = 0
unsigned int destPartition = 0x9024e8
CloneInfo * cloneInfo = 0x0
CloneInfo * imageInfo = 0x14f11b0
unsigned int configId = 0xe9fcd0
bool clientInventory = true
unsigned int sysprepSID = 0xe9fd00
bool getOrPut = true
unsigned int configCode = 0x1e46a1c
unsigned int localToClient = 8
unsigned int viewId = 0x1e46a50
unsigned int sourcePartition = 0x901f00
unsigned int backupRegimeMachineID = 0x1e49e18
bool sidChange = false
bool localPackage = false
Library::Gc::SQueue * removePackageSet = 0x14f6672
unsigned int putFlagInt = 0x1e46b2c
int udmMode = 0x1e46ad8
unsigned int sysprepRun = 1
Library::Gc::SQueue * incrementalSet = 0x49b480
unsigned int something = 0x77e6b700
Library::Gc::SQueue * dosFileSet = 0x14f12dd
bool fetchConfig = true
Library::Query::Database * db = 0x14f11f4
bool taskValid = true
unsigned int resourceViewId = 0xf6a44d64
unsigned int cancelOnWarning = 0
unsigned int instanceID = 0xe9fdf0
unsigned int throttle = 0x4c74dd0
unsigned int destDrive = 0
unsigned int putResourceViewId = 0x809146e4
unsigned int backupRegimeInstanceMachineID = 0x14f21ec
unsigned int brinstanceMachineID = 0xe9fef4
bool recoveryTask = true
bool mustExist = true
int preserveImage = 0x14f216c
int preserveDestination = 0x14f213c
bool useRecoveryImage = true
bool removeDefaultImage = true
bool useDefaultImage = true
int driveLetter = 0x14f6651
Library::Gc::SString * name = 0x0
ViewItem * clientView = 0x14f055c
bool isUNC = true
bool isHttp = true
bool isTargetPath = true
unsigned int machineInstanceID = 0
unsigned int userFilesOnly = 0xe9fea4
unsigned int wmiClassId = 22
Library::Gc::SString * sNamespace = 0x0
Library::Gc::SHashMap * classHash = 0x14f11f4
e:\corporate\trunk\configureagent\server\database.cpp#2487:l_launchTask
StartLpc * lpc = 0x16
Library::Gc::SHashMap * index = 0x16
Library::Gc::SObject * key = 0x0
unsigned int taskId = 0x14f11f4
unsigned int taskViewId = 0x14f21ec
unsigned int logId = 0x14f11d4
ActiveTask * active = 0x14f21ec
e:\corporate\trunk\configureagent\server\database.cpp#2487:l_launchTask
StartLpc * lpc = 0x1e2f530
Library::Gc::SHashMap * index = 0x13ef1d0
Library::Gc::SObject * key = 0xe9ff60
unsigned int taskId = 22
unsigned int taskViewId = 22
unsigned int logId = 202
ActiveTask * active = 0x16
e:\corporate\trunk\configureagent\server\database.cpp#2533:g_launchTaskView
StartLpc * lpc = 0x1e2f530
e:\corporate\trunk\configureagent\library\src\thread\worker.cpp#278:Library::Thread::QueueWorkerThread::dispatcher
@3 FunctionPointer * callback = 0x1e2f530
unsigned int param = 872
void * token = 0xc9ff44

Thanks for posting the backtrace; the fault trace indicates that some important initialization of the server process has been bypassed, and it's not immediately clear why - it appears that the configuration server process has been unable to read the client information from the database when it started, although I would have expected some error report to that effect to have been displayed at that time.

However, you also mention port 6666 being blocked: this is in fact one of the two primary UDP ports used by the GhostCast protocol. Port 6666 is the port number on which the GhostCast process listens for Ghost client connection requests, and I would expect that blocking this UDP port will stop both manual and automated (console-driven) GhostCasts from working.

The GhostCast and Ghost applications don't provide a way to change this port assignment; they have been fixed at that value since the Ghost multicast facility was first created. If you can, try unblocking that one port and see if that allows you to proceed with manual use of the GhostCast server. If there is some specific reason why unblocking that port is problematic for you, let us know what it is because it may indicate that we have to develop an alternative method for the clients and servers to contact each other.

My initial assesment of the stack backtrace is that the configuration server fault isn't directly related to the problem with manual GhostCast, but to explore that fault further we may need some additional diagnostic information. In the ghost program file directory, there should be a file named "NGERROR.DMP" which we may need to arrange to retrieve from you to diagnose this further.

Our head administrator found the port 6666 thing and created an exception in the firewall's rule for GhostSrv.exe. I'm a little confused as to how it ever worked to begin with, but I'm inclined to believe that the auto-update of the firewall (McAfee; it's what the university's IS department supports) updated that rule and removed our exception. Needless to say, we disabled the auto-update, so that shouldn't happen again. However, it seems (from my angle) that it's a separate issue from what caused the crash.

The same head admin had a look through the Windows Event Viewer and couldn't find anything immediately apparent. I found the NGERROR.DMP file from yesterday morning, but it's reported as 0 bytes, so we seem to be up the creek on that front. We discovered that Ghost is using a SQL database on the server--perhaps the database has been misconfigured?

Thanks for the update.

The database in question is a embedded runtime version of Sybase SQLAnywhere. In version 8.2 of Ghost (which is what you are running) it is possible for some firewalls to have an impact there too; the database engine is bound only to the local loopback address (127.0.0.1, so it isn't accessible remotely over the network) on port 2638.

Both the regular user-mode Ghost console application and the Configuration Server need access to this, and it can happen that one has been granted an exception rule and the other hasn't. Because the Configuration Server service starts at boot time, interactions with machine-level firewalls are tricky because they all approach boot time differently.

That's something you can at least have a look at; if this GP fault started occurring at the same time that the regular GhostCast stopped working, we should at least try and eliminate the possibility.

Assuming that doesn't pan out, we'll need to escalate the investigation a bit; I'll have to involve some extra people on this end for that. If nothing else, we may need to add this firewall software to our compatibility testing process.