I traced all the traffic from client via proxy to sepm server. Enabled all firewall/logging to see if it was trying to bypass proxy, but there was NOTHING to be found. All very very weird.
But i’ve made a strange discovery which has fixed the issue.
I hadnt made any changes at all to the squid proxy server - so the problem definitely doesn’t come from proxy itself.
The only thing I changed was the proxy settings in the local system account on the client machine. I had referenced the proxy by its name ‘proxy1’ in those settings, and added the IP for that to the Windows hosts file. In the real world this name would be resolved by DNS. At first I thought it was related to name lookup not working for content downloads, but I was wrong.
At the same time as this, I disabled the HTTP 1.1 through proxies setting. I used procmon to capture registry changes and eventually found this to be the magic fix (when using local system account for proxy settings):
[HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings]
"ProxyHttp1.1"=dword:00000000
You can also change this setting in Internet Explorer (providing you launch IE interactively as local system using psexec).
To me, it looks like whatever the specific component is in SEP which performs the content update download bit (an embedded LU-type thing?) doesn’t operate the same way as the management comms as far as HTTP is concerned. The reason I believe this to be the case, is all other proxied management traffic for HEAD/GET/POST to /secars/secars.dll including the hello secars test seems to work okay (lots of HTTP 200's in sylink and squid log for that), but the GET's to /content/{GUID}/xxxxxxxxx/ for the dax’s and full zip’s just don’t work!
Symantec must be using a different mechanism within the SEP application for content updates I would assume. Without HTTP 1.1 over proxies disabled, all management communication between SEP and SEPM works fine until SEP tries to perform a content download; then you get 997 errors (and others) in sylink log which misleadingly suggests a problem with the proxy server. In reality it seems the requests to HTTP GET content never even hit the proxy server!
i think in 12.1.3 the processing order for proxy servers was changed, however this doesn’t seem to impact much here as I couldn’t get this to work in 12.1.2 either so the problem is probably not proxy-order related.
In general, there is a certain flakiness about this anyway - IE proxy settings are written to the registry and those aren’t always changed or removed reliability unless you go into regedit and actually delete the registry entries in there.
Conclusion: I think this is a BUG. Please look into and fix this Symantec!