Working on a Task Server set of issues for a customer I have just learnt a number of interesting facts about the client task server and the agent.
First the Altiris Agent and Client Tasks Agent tries to connect to a task server, if assigned manually, and revert to the site management defined server when too many errors occur.
I'm not 100% sure on the process but we saw this happen just now. We had a subnet with no task server configured and needed to test if a CTA error was caused by communication issues or the CTA itself 
We assigned the client to a task server (not the NS). The client then tried to connect to the assigned TS but this did not work, the http request constantly returned error 500 (which is the main reasons for this, but we'll get to it after). So after a period of trial and error the CTA was switched back to the NS (returning to default behaviour -> in a subnet / site not containing a PS./TS).
Second then, we had the error 500 from the TaskServerManagement web-application on a Task Server. Checking on the server console we could see IIS running properly (both on the inetmgr.exe tool and via iexplore), the PS and asp pages were served without problem but the "http://localhost/altiris/taskservermanagement/register.aspx" returned error 500:
Compilation Error Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately. Compiler Error Message: CS0016: Could not write to output file c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\ Temporary ASP.NET Files\...
Thankfully Google had a lot to say about error CS0016 and MSDN Social contained the solution to this problem : the problem (has I had checked) did not related to the Network Service account right on the "%windir%\Microsoft.net" tree but rather an issue with the "%windir%\temp" folder, which the account had absolutely no access to.
Adding the account back and resetting IIS solved this issue.