GSS2 Client Errors - Out of Proc

*Bump*

Hello Brian,
Just trying get a clearer picture on your situation, are you able to give more information on what happens?

When pushing 'items' do you mean files, images or both. And at what stage of the process do you get the 'Out of Proc' error. Also do you see this error in a log file (if so please post it) or does it appear in a separate window? Does it produce an error log in the Event Viewer?

Cheers,
Bruce

Not much more I can say about it other than the error that pops up on clients is an "Out of Process" error. This happens when nothing is even being done on the clients (client is idle).

When I say pushing items, I mean executing tasks (images, AI Packages, configuration, etc.).

Hmm. That's not an error message that I'm familiar with; is that the exact text? I just did a search of the GSS source code and the error message that seems to be closest to that is "Out-of-order producer completion", which comes from some new code in GSS2 and I have been trying to eliminate.

Normally, with pop-up message boxes you can hit Control-C to copy the text of the message to the clipboard, and that's an easier way to get the exact message into the forum given it doesn't support posting screenshots directly.

Hi Nigel,

Sorry for the delayed response. But it can take a while for end-users to report the actual errors.

First pop-up error:

Title bar: c:\program files\symantec\ghost\ngctw32.exe

aiobuf::sync()position error


Second pop-up error:

Title bar: c:\program files\symantec\ghost\ngctw32.exe

Out_of_order producer completion


Any ideas?

No problem; for this particular error, the root cause seems to be a threading bug in the client code, and it causes a range of different outcomes (and thus, visible errors) depending on the exact sequence two parts of the code are interleaved. I do have a development fix for this if you want to get in touch with me at nigel dot bree at gmail dot com

The first error message will indeed have been "out of order producer completion", that's another outcome of the same underlying bug.

Hi,

I just upgraded from GSS 1 to GSS 2 last week and experienced this error on 112 computers in a lab that prevented them from doing their automated shutdown for maintenance over the weekend. Is there a repair for this error or do I have to retrofit some 300 computers that I have upgraded to prevent this from happening again?

Thanks

You can get in touch with me for an update at nigel dot bree at gmail dot com; it may not be completely perfect just yet (even in the release version of GSS2 this was incredibly rare in our test labs, making it a difficult one to diagnose and trace through) but it certainly helps a lot and you're welcome to try it.

We're experiencing this very intermittently as well. Will there be any official patch or update soon? Any idea what causes it or if there are any workarounds?

Thanks!

As explained above, it's a threading bug in the code I wrote; there's unfortunately no way to mitigate this yourselves, you need a new executable.

I can't give you a date as to when a full product update would be made generally available for all customers.

Getting exactly the same pop ups.

Any plans for a service pack soon?

We are getting the same GSS2 Client Error - "Out-of-order producer completion" message randomly on clients on a control systems network.  The bad news is we can not do anything with the system once the error appears and it comes right back as soon as we acknowledge it.

After repeatedly hitting the only button option provided - "OK" - finally gave up and rebooted the system.  Alas, the first event was on THE configuration server for the entire server cluster...

I've already emailed Nigel for a copy of the "fix" so this post is just to further document the issue.

Pity this forum software unfortunately does not allow us to create "stickies". Anyway, the full LiveUpdate patch that resolves this issue (amongst others) has been released as announced here. I don't have a complete list of all the changes, but this is definitely intended to be among them.

I have experienced this error on one (and, so far at least, only one) client machine here, though that PC has had the Ghost client installed for a couple of months and the server has certainly been updated to 11.0.1.1533. Does LiveUpdate need to be run on the client somehow as well? I am trying a reinstallation of the client from the updated Console to see if this helps.

Thanks for that information, Ben. Basically, there must be one other mechanism for triggering this error in the code, but this far I've not found what it might be. We run all kinds of tests here pretty much continuously and in the months since 2.0.1 I have not heard of one single occurrence of this here in our labs - so any mechanism by which this still happens is fiendishly rare under normal circumstances.

We do have solid information that a small number of customers do have this happen a lot - for them, it's very frequent. Since it's happening about a million times more often for them than anyone else, there has to be a reason - a particular trigger of some kind like an interaction with another program and if we could just find out what it is, we'd be able to make it happen in our labs too and fix it in short order.

As it happens, we seem to have found a particular piece of software that is common to places where this is frequent and so something that program is doing might be triggering something in ours - we're pursuing this lead but as yet we don't have hard evidence one way or the other and until we do it's open season on this bug.

Could it be that there is an interaction with the Sophos, Bigfix, and Ghost agents?

It's certainly possible, but I don't have a reason to suspect those in particular - I do appreciate you taking the time to give us the data point though, every bit helps. I don't think an interaction is the "cause" as such, it's just one lead we're following since there seem to be some environments where it's hugely more common than others and one unusual third-party program has turned up in a couple of them.

The main thing we're after right now is still a way to trigger this in a controlled environment so we can diagnose the real root cause, not the final symptom - the error that's being reported in the pop-up is inconsistency in a really important piece of data, but the cause of that inconsistency isn't clear yet.

This is a ... complicated issue. Amongst the various work I've been doing, I think I have things so that the current builds don't do this - it's impossible to say with certainly whether I've really addressed the root cause since we have still been unable to reproduce it here, but I've got to a point where I can make it go away.

Now, we intend to release a 2.0.2 update at some point (which is as specific as I can be as to timing; I'm breaking the rules even by saying that I'm working on it). Unfortunately, however, the existing upgrade process itself seems to provoke an error too :-(. Upgrading from build 1533 to the current one would be worse for the majority of customers I can get the upgrade issues sorted out and that's the major focus of the next week or two. Fortunately that's a more tractable problem that this issue has been, but it's still going to take a little time to work through.

I do wish I had some more definitive and somewhat better news, but that's my understanding of where things stand with this issue.

Nigel,  I have been getting this message in a very specific lab, and if you're interested I can burn &ship you a copy of our image files so you can see what's installed and get you our hardware specs to see if that helps.  I first started to see this error after I modified the nightly inventory report to gather all installed software, rather than just the default inventory report. I am reachable by e-mail at adam dot zahn at fcps dot edu should you be interested in the image files.

Thanks for all the help, this one must be annoying, reading from the thread..

Thanks for the offer of help, Adam. If you can drop me a line at nigel.bree@gmail.com I'd like to hear more about the situation in your lab.

Since there are a few different threads going on, the current situation as of now is what I posted here. I do think I've got the problem mostly licked but any situation where a problem is occurring more frequently is still of real interest, because it may involve a different pathway for the problem to occur.