Ghost Solution Suite

 View Only

  • 1.  ngtray cause other application hang

    Posted May 11, 2010 01:20 AM
    Hello All,

    We are encountering a hang problem here:

       In our application, there are two threads were trying to call OutputDebugString(...), and they both were waiting for a kernel level mutex
    which was hold by ngtray.exe (C:\program files\symantec\ghost\ngtray.exe).

       And after we kill ngtray.exe, our application recovered from hang mode.

    Our two waiting threads as:

    THREAD 88511da8  Cid 0460.045c  Teb: 7ffdf000 Win32Thread: e7154008 WAIT: (UserRequest) UserMode Non-Alertable
                889fdd68  Mutant - owning thread 889a5308
    and
            THREAD 88437020  Cid 0460.16f8  Teb: 7ff9b000 Win32Thread: e82424f8 WAIT: (UserRequest) UserMode Non-Alertable
                889fdd68  Mutant - owning thread 889a5308


     lkd> !process 0 2 ngtray.exe
    PROCESS 889a5788  SessionId: 0  Cid: 01c4    Peb: 7ffdd000  ParentCid: 072c
        DirBase: 0a4c0280  ObjectTable: e115c828  HandleCount:  54.
        Image: ngtray.exe
            THREAD 889a5308  Cid 01c4.01c8  Teb: 7ffdf000 Win32Thread: e1136ca0 WAIT: (Suspended) KernelMode Non-Alertable
    SuspendCount 1
                889a54a4  Semaphore Limit 0x2
            THREAD 8899eda8  Cid 01c4.01e0  Teb: 7ffde000 Win32Thread: e1337870 WAIT: (UserRequest) UserMode Non-Alertable
                889fdd68  Mutant - owning thread 889a5308


    Captured a dump file for ngtray.exe and get following call stack Msgs:

    0:000> ~* kb ffff
    .  0  Id: 1c4.1c8 Suspend: 0 Teb: 7ffdf000 Unfrozen
      Memory  ChildEBP RetAddr  Args to Child             
              0012fa88 7c90df5a 7c8025db 00000044 00000000 ntdll!KiFastSystemCallRet
            4 0012fa8c 7c8025db 00000044 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
           64 0012faf0 7c802542 00000044 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0x8b
           14 0012fb04 7c85ae03 00000044 ffffffff 00939620 kernel32!WaitForSingleObject+0x12
          260 0012fd64 7c85b440 00159148 00910804 014e014c kernel32!OutputDebugStringA+0xb7
           20 0012fd84 0041a207 0093b11c 00910804 00000001 kernel32!OutputDebugStringW+0x3b
              00000000 00000000 00000000 00000000 00000000 ngtray+0x1a207
       1  Id: 1c4.1e0 Suspend: 0 Teb: 7ffde000 Unfrozen
      Memory  ChildEBP RetAddr  Args to Child             
              00b0f850 7c90df5a 7c8025db 00000044 00000000 ntdll!KiFastSystemCallRet
            4 00b0f854 7c8025db 00000044 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
           64 00b0f8b8 7c802542 00000044 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0x8b
           14 00b0f8cc 7c85ae03 00000044 ffffffff 000006e7 kernel32!WaitForSingleObject+0x12
          260 00b0fb2c 004076cf 0093a42c 00000040 0091078c kernel32!OutputDebugStringA+0xb7
           2c 00b0fb58 00407196 00000040 0091078c 00b0fc28 ngtray+0x76cf
           1c 00b0fb74 00407997 00939d20 00b0fbb0 00000010 ngtray+0x7196
           20 00b0fb94 004079e4 004079ed 009105cc 00000005 ngtray+0x7997
            4 00b0fb98 004079ed 009105cc 00000005 00939ae0 ngtray+0x79e4
           10 00b0fba8 00407baa 00420240 00939cf0 004272a8 ngtray+0x79ed
           18 00b0fbc0 00407ed3 00407ee5 0091070c 00b0feb4 ngtray+0x7baa
            4 00b0fbc4 00407ee5 0091070c 00b0feb4 00912060 ngtray+0x7ed3
           20 00b0fbe4 004082b5 00000000 00939cf0 0091073c ngtray+0x7ee5
           18 00b0fbfc 00407738 00000040 00000040 0091078c ngtray+0x82b5
           18 00b0fc14 100011d3 100011ea 10003014 00406cf3 ngtray+0x7738
           70 00b0fc84 00411588 00000040 00910a64 00b0feb4 THREAD!suspendThreads+0x43
           18 00b0fc9c 00407b6a 0000000a 00407d79 0000000a ngtray+0x11588
            8 00b0fca4 00407d79 0000000a 00b0feb4 00911fe0 ngtray+0x7b6a
           20 00b0fcc4 004082b5 00911fe0 00939ae0 0091073c ngtray+0x7d79
           28 00b0fcec 004029d3 00910a64 00939aa0 00000000 ngtray+0x82b5
           68 00b0fd54 004079e4 004079ed 00405bb1 00000033 ngtray+0x29d3
            4 00b0fd58 004079ed 00405bb1 00000033 00b0fdb4 ngtray+0x79e4
            4 00b0fd5c 00405bb1 00000033 00b0fdb4 0040e47a ngtray+0x79ed
            c 00b0fd68 0040e47a 00911f10 0091078c 00b0fdbc ngtray+0x5bb1
           4c 00b0fdb4 0040250b 00b0feb4 00000000 00b0feb4 ngtray+0xe47a
           18 00b0fdcc 00402a9f 00910a64 00939a90 00000000 ngtray+0x250b
           c8 00b0fe94 00402d59 00b0feb4 00000000 009109fc ngtray+0x2a9f
           64 00b0fef8 004013f4 00910db4 00000001 00910df8 ngtray+0x2d59
           10 00b0ff08 004107d8 00910db4 009109fc 00000001 ngtray+0x13f4
           10 00b0ff18 004106fd 00410940 0093a3ec 00000000 ngtray+0x107d8
           10 00b0ff28 00410953 0093a3ec 00000004 0040390a ngtray+0x106fd
              00000000 00000000 00000000 00000000 00000000 ngtray+0x10953
    0:000> !handle 44 f
    Handle 44
      Type          Mutant
      Attributes    0
      GrantedAccess 0x1f0001:
             Delete,ReadControl,WriteDac,WriteOwner,Synch
             QueryState
      HandleCount   24
      PointerCount  33
      Name          \BaseNamedObjects\DBWinMutex



    And, what this is ngtray.exe used for? I found that ini our system not all PCs have this "ghost" folder(C:\program files\symantec\ghost), usually just has folder "Liveupdate" and several other files under "C:\program files\symantec". Whether our problem computers used the very old version of it?

    Anyone has any suggestion about how to avoid this issue or fix it?


    Thanks


  • 2.  RE: ngtray cause other application hang

    Posted May 11, 2010 04:59 AM
    Added some local kernel debug call stacks from ngtray.exe:

            THREAD 889a5308  Cid 01c4.01c8  Teb: 7ffdf000 Win32Thread: e1136ca0 WAIT: (Suspended) KernelMode Non-Alertable
    SuspendCount 1
                889a54a4  Semaphore Limit 0x2
            Not impersonating
            DeviceMap                 e5db5318
            Owning Process            889a5788       Image:         ngtray.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      16116694       Ticks: 131207 (0:00:34:10.109)
            Context Switch Count      801493                 LargeStack
            UserTime                  00:00:00.140
            KernelTime                00:00:00.500
            Win32 Start Address 0x0041c32a
            Start Address 0x7c810705
            Stack Init ba168000 Current ba167bf8 Base ba168000 Limit ba164000 Call 0
            Priority 10 BasePriority 8 PriorityDecrement 0 DecrementCount 16
            ChildEBP RetAddr 
            ba167c10 80503846 nt!KiSwapContext+0x2f (FPO: [Uses EBP] [0,0,4])
            ba167c1c 804fb078 nt!KiSwapThread+0x8a (FPO: [0,0,0])
            ba167c44 80502f82 nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
            ba167c5c 804ff8c4 nt!KiSuspendThread+0x18 (FPO: [3,0,0])
            ba167ca4 80545977 nt!KiDeliverApc+0x124 (FPO: [Non-Fpo])
            ba167cc4 804fb1b4 nt!KiUnlockDispatcherDatabase+0xcf (FPO: [Uses EBP] [0,0,4])
            ba167cec 805c075e nt!KeWaitForSingleObject+0x2fe (FPO: [Non-Fpo])
            ba167d50 8054162c nt!NtWaitForSingleObject+0x9a (FPO: [Non-Fpo])
            ba167d50 7c90e514 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ ba167d64)
    WARNING: Frame IP not in any known module. Following frames may be wrong.
            0012faf0 00610072 0x7c90e514
            0012faf4 0025006d 0x610072
            0012faf8 00300032 0x25006d
            0025006d 80020294 0x300032
            00250071 81020294 0x80020294
            00250075 82020294 0x81020294
            00250079 83020294 0x82020294
            0025007d 84020294 0x83020294
            00250081 85020294 0x84020294
            00250085 86020294 0x85020294
            00250089 87020294 0x86020294
            0025008d 88020294 0x87020294
            00250091 89020294 0x88020294
            00250095 8a020294 0x89020294
            00250099 8b020294 0x8a020294
            0025009d 8c020294 0x8b020294
            002500a1 8d020294 0x8c020294
            002500a5 8e020294 0x8d020294
            002500a9 8f020294 0x8e020294
            002500ad 90020294 0x8f020294
            002500b1 91020294 0x90020294
            002500b5 92020294 0x91020294
            002500b9 93020294 0x92020294
            002500bd 94020294 0x93020294
            002500c1 95020294 0x94020294
            002500c5 96020294 0x95020294
            002500c9 97020294 0x96020294
            002500cd 98020294 0x97020294
            002500d1 99020294 0x98020294
            002500d5 9a020294 0x99020294
            002500d9 9b020294 0x9a020294
            THREAD 8899eda8  Cid 01c4.01e0  Teb: 7ffde000 Win32Thread: e1337870 WAIT: (UserRequest) UserMode Non-Alertable
                889fdd68  Mutant - owning thread 889a5308
            Not impersonating
            DeviceMap                 e5db5318
            Owning Process            889a5788       Image:         ngtray.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      16116694       Ticks: 131219 (0:00:34:10.296)
            Context Switch Count      45733                 LargeStack
            UserTime                  00:00:00.031
            KernelTime                00:00:00.031
            Win32 Start Address 0x004036b0
            Start Address 0x7c8106f9
            Stack Init 9cc37000 Current 9cc36ca0 Base 9cc37000 Limit 9cc34000 Call 0
            Priority 11 BasePriority 9 PriorityDecrement 0 DecrementCount 16
            Kernel stack not resident.lkd> !process 889a5788
    PROCESS 889a5788  SessionId: 0  Cid: 01c4    Peb: 7ffdd000  ParentCid: 072c
        DirBase: 0a4c0280  ObjectTable: e115c828  HandleCount:  54.
        Image: ngtray.exe
        VadRoot 8899e828 Vads 52 Clone 0 Private 202. Modified 950526. Locked 0.
        DeviceMap e5db5318
        Token                             e1177c98
        ElapsedTime                       2 Days 22:30:50.495
        UserTime                          00:00:00.187
        KernelTime                        00:00:00.515
        QuotaPoolUsage[PagedPool]         54148
        QuotaPoolUsage[NonPagedPool]      2320
        Working Set Sizes (now,min,max)  (5216, 50, 345) (20864KB, 200KB, 1380KB)
        PeakWorkingSetSize                5216
        VirtualSize                       43 Mb
        PeakVirtualSize                   44 Mb
        PageFaultCount                    3591312
        MemoryPriority                    BACKGROUND
        BasePriority                      8
        CommitCharge                      247
        DebugPort                         89d9c160


  • 3.  RE: ngtray cause other application hang

    Posted May 11, 2010 05:04 AM
    > what this is ngtray.exe used for?

    It provides user interface - running in the logged-in user context - for the Symantec Ghost Solution Suite client service (ngserver.exe) running in the LocalSystem context.

    > Whether our problem computers used the very old version of it

    Yes, you have an old version; this (rare) deadlock was present in the initial 2006 release of Ghost Solution Suite 2.0; it was reported and resolved some three years ago and the fix is available in the LiveUpdates to that product. The LiveUpdate to receive the improved code should be run on the Ghost Solution Suite server, and after being updated (for the GSS 2.0 product, to version 2.0.2, which does not exhibit this problem) that server will in turn update the clients over time - you should communicate this with your network administrators so that they can update their servers for you.

    > Anyone has any suggestion about how to avoid this issue

    It requires a product update to fix; the deadlock is occurring due to a rare timing situation between two threads in the ngtray application interacting with each other (the timing window is extremely small, but any extremely low-probability event will occur given enough time).


  • 4.  RE: ngtray cause other application hang

    Posted May 11, 2010 09:18 PM

    Nigel, Thanks for your reply,

    I checked the file version for ngtray.exe 

    File version: 11.0.1.1533

    Internal Name: mctray

    Copyright: Copyright(C) 1998-2007 Symantec Corporation.

    If that means that it aready contains that fix?
     



  • 5.  RE: ngtray cause other application hang
    Best Answer

    Posted May 11, 2010 09:47 PM
    The version you have is from Ghost Solution Suite 2.0.1; as explained above, the version you require installed on the GSS server is 2.0.2, which is a simple LiveUpdate that can be run on the machine with the Ghost Solution Suite server the clients are connecting to. After that upgrade on the server, the server then can upgrade the client (as directed by your GSS server administrator), after which the file version of the ngtray.exe executable should become 11.0.2.1573

    Only the 2.0.0 and 2.0.1 releases of Ghost Solution Suite had this specific timing issue - in practice it basically never shows up except where additional third-party software is using the OutputDebugString APIs very heavily, which then causes the thread responsible for the deadlock to block right at the point where the deadlock can occur and thus opens up the timing window in which deadlocking is possible.