ProxySG & Advanced Secure Gateway

Hi Team,

Hi Team,

The ProxySG suddenly not work yesterday, it prompt login for users, then we re-join domain and it working now.

We found below logs from eventlogs.

Schannel (MPFA): Resetting Schannel due to error: 0xC0000001(-1073741823), DC: ADVMF1.mpfahk.org"  0 250042:1   lw_schannel.cpp:772
Schannel (MPFA): Connected to DC: ADVMF1.mpfahk.org"  0 250041:96   lw_schannel.cpp:1252

"Authentication failed with 40069 (0x00009C85) (symbol: 'The password is invalid'): user 'cliff_l' (domain mpfa) - user considered 'unknown'"  0 250017:96   lw_schannel.cpp:554

We found that the below two article which is related to this issue, but still am not sure that which causing issue.  Some other article they mentioned as BUG bug # 232071 but it resolves SGOS version 6.5.9.6 (or above). but customer already running 6.5.10.6 version.

Below are the article i have refered.

https://support.symantec.com/en_US/article.TECH244693.html

https://support.symantec.com/en_US/article.TECH245792.html.

Please advise,

Thanks,

Ram.

Hi Team,

Any update on this.

-Ram

Hi Team,

Any update or any suggestion on this

-Ram

Hi Ram,

               Better to get this checked via a case. An LSA debug taken at the time of issue could have helped. Raising a case will help in checking for the exiting logs and to plan for data collection if the issue repeats.

Hi Aravind,

The issue has been resolved after re-joined the domain.

Can i have a procedure to collect LSA debug logs if the issue repeats.

Thanks,

Ram.

Hi Ram,

           You will need to collect the information as requested in article https://support.symantec.com/en_US/article.TECH243203.html  . For taking packet captue, you may use a filter including all the ports mentioned in the article https://support.symantec.com/en_US/article.TECH243205.html

Hi Aravind,

The device still reset Schannel, we have already rejoin the domain but again the issue araises. 

Please find the attached eventlogs for reference. Please advise us to proceed further in order to fix this issue.

Thanks,

Ram.

Attachment(s)

events(9).zip   34 KB 1 version

Hi Team/Aravind,

Any luck/any suggestions on this?

-Ram.

Hi Ram,

              Not easy to track reason for Schannel resets. LSA debug and packet captures will be needed to confirm. These need to be collected when the issue is present.

Hi Aravind,

Thank you for the update.

We have requested client to collect those LSA logs.

Thanks,

Ram.

Hi Aravind,

I have shared you via private messagge the LSA logs for the Schannel reset issue.

Thanks,

Ram

Hi Ram,

            Add a packet capture when you are creating the case. The LSA debug is full of "No Schannel slots avaiable. Waiting for next available slot." . PCAP filter "port 53 or port 445 or port 389 or port 3268 or port 88"

Hi Aravind,

I have opened case with Symantec 14412445, yet to assign engineer.

Thanks,

Ram

Hi Team,

We have opened case with TAC. Please find the below recommentation from TAC engineer.

“From checking on information uploaded, below are the errors.

6282.263 Thread: 0x100311D No Schannel slots avaiable. Waiting for next available slot.
6282.187 Thread: 0x40311A No Schannel slots avaiable. Waiting for next available slot.
6281.630 Thread: 0x403116 No Schannel slots avaiable. Waiting for next available slot.

I suspected that the problem is caused by schannel congestion. You can try the following options.

1. If MaxconcurrentAPI registry setting is not increased to 10 on DC side, please do that and that is needed especially if more users are coming from the foreign domains.

https://support.microsoft.com/en-us/kb/2688798 (2003)

To change the MaxConcurrentApi setting, follow these steps:

1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters
3. On the Edit menu, point to New, and then click DWORD Value.
4. Type MaxConcurrentApi, and then press Enter.
5. On the Edit menu, click Modify.
6. Type the new MaxConcurrentApi setting in decimal, and then click OK.
7. At a command prompt, type the following command, and then press Enter:
net stop netlogon
8. Type the following command, and then press Enter:
net start netlogon

I would recommend to use 10 Schannels for now. This procedure has to be performed on:
- All DCs (because we don't know in advance which DC BCAAA will talk to)

2. Please make sure that there’s no network issues between SG and DC that would cause the delay.

3. We can try to switch the preferred and alternate DCs so that it will fix the problem if issue is related to current preferred DC.

4. Finally, the permanent and reliable solution here is to implement Kebreros auth so that schannel does not come into picture. Discuss this option with the customer and have them use Kerberos instead of NTLM.

Please try and let us know how it goes. If you still got issue please help capture a new set of information for us to check by follow steps below.

1. Create a custom snapshot for https://<IP_address>:8082/LSA/Stats (this URL is only available in newer SGOS 6.x versions) with 100 stored copies taken at a reasonable duration. Use 1 minute if the problem is happening at that time.

2. Create a similar snapshot for https://<IP_address>:8082/LSA/Debug with 100 stored copies taken at a reasonable duration. Use 1 minute if the problem is happening at that time.

3. Go to https://<IP_address>:8082/LSA/Debugmask and enable all debug masks apart from MODULE_SGOS_FDIO. Click Set Mask to apply the change.
4. Start a packet capture (use below filter) on the ProxySG appliance when reproducing the issue, stop the packet capture when finish.
- Go to Management Console > Maintenance > Service Information > Packet capture > make sure that the filter is "port 53 or port 389 or port 445 or port 139 or port 88 or port 3268" and start PCAP

Thanks,

Ram.

Hi Aravind,

One more customer also facing the authentication problem. Users not able access internet. After bypassing the authentication users can access.

We have taken LSA logs and PCAP. While analysing the LSA debug logs i found below logs.

7113.524 Thread: 0x6B801C20 No Schannel slots avaiable. Waiting for next available slot.
7113.520 Thread: 0x3C801CD8 No Schannel slots avaiable. Waiting for next available slot.
7113.414 Thread: 0xA3800A8A No Schannel slots avaiable. Waiting for next available slot.
7113.372 Deleting LW_Access_token
7113.372 User logged in via GSSAPI: hkwlee, schannel time 1, logon server SCBADS04.legco.hksar, user's domain LCSFNTDOMAIN1
7113.372 GSSAPI auth: User hkwlee is a member of group g_proxyuser
7113.372 Looked up user in access token: hkwlee

Shall we suggest same solution which the TAC engineer suggested for the earlier error.(To change the MaxConcurrentApi).

Please advicse.

Thanks,

Ram.

Hi Ram,

              Yes, if the issue is a genuine Schannel issue. By default the number of concurrent Schannel is set in proxy to 2. You can increase it to match what is configured on the AD side. Also as a best practice, try to move to "IP-Surrogate" mode of authentication to reduce the dependency on AD server much. Evaluate the customer enviornment before changing the policy (i.e. Shared machines, Citrix farms, NAT before hitting proxy etc).