Yes, WinPE is a bit of a monster. I agree with the above comment... you COULD remove HTA/WMI functionality if you don't need it. You can also specifically include drivers that you need. That will help greatly.
Also, if you're running DS 6.x, make sure you have the "Compres WIM" option enabled, as that makes things a bit better as well.
Finally, consider replacing the "Altiris MTFTP Service" binary with the one from the free/Open Source "openTFTP" project. There is an article on here about getting that configured (or I can help if you need it). It speeds up the process by some 10x - which makes the WinPE size slightly less important.