Please see Altiris KB46279 - IE 8 Support.

...this usually occurs because the "User Agent" string for your browser is too long.

Check the registry at the following locations:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent\Post Platform
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent\Post Platform

See the following MS KB article also:
MSDN article

IE8 seems to work fine with NS6 console as long as you enable "Compatibility Mode".  You can enable this using the "broken page" icon next to the Address bar.  There are some small incompatibilities (I've noticed some scrolling issues when selecting a package in Software Delivery) but overall it seems to be OK.

Dear jharings, KSchroeder,

In my case, "Compatibility Mode" doesn't work, the same result comes out.

I wonder it is caused by recent Microsoft update patch.

Last month I can enter Altiris Console normally using IE8.
So I have uninstall all Microsoft update patchs installed since 2010/01, but syill doesn't work.

If it's a remote workstation, try removing the active-x controls and let them be reinstalled. So try deleteing the following from Windows\System32:

1.     AexMenuCtrLib.dll
2.     AeXNSConsoleUtilities.dll
3.     AeXNSPkgDLLib.dll
4.     AeXScheduleMgr.dl
5.     AeXTreeCtrlLib.dll

I haven't personally heard of any issues with IE8 (although it's not supported) or recent patches, and I've run both v6 and v7 consoles on it. Can you try another system if it isn't the server?

Again, please check your UserAgent string. This is the exact behavior we have experienced here with IE8 and the only fix was modifying the user agent string.

About User Agent, there is something strange, I cannot see any string about IE in below path

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent\Post Platform

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent\Post Platform

I have tried downgrade back to IE6 but it still shows "Altiris Console Minimum Requirements".
Then I upgrade to IE7, result is the same.

It is a remote station, and I have remove the following files from C:\Windows\System32

1.     AexMenuCtrLib.dll
2.     AeXNSConsoleUtilities.dll
3.     AeXNSPkgDLLib.dll
4.     AeXScheduleMgr.dl
5.     AeXTreeCtrlLib.dll

But result is the same, seems it did not reinstall these ActiveX Control files again.

1) Find a working system and export that key and import it back in to yours.

If that works, and the active-x controls don't reinstall (they can be a real bear to figure out)

2)Copy those files directly from a working system back into your Windows\System 32 directory.

...you may want to consider updating to the current SP3 rollup, R12, as there are many other improvements/fixes since R5.  Release Notes are available in AKB 46763.  You can view the changes from R5 to R11 in AKB 38996.