This depends on what roles the site server will perform. You can use Windows XP for a site server functioning as a package server, but not a task server. (Task server cannot be installed on desktop operating systems.) And if your site server is only running a desktop OS like Windows XP, keep in mind connection limits -- 10, for TCP/IP.
Because you'll probably want to use both Package and Task services, you'll want an actual server.
Here's what the Altiris 7 Planning & Implementation Guide puts out (with a DISCLAIMER! -- not tested by Symantec Labs!):
Minimum requirements:
Pentium 4, 512MB RAM, any Windows OS, IIS 5, 1.5GB of HDD + 1.2x total size of packages, NTFS no RAID. Altiris Agent, .NET 1.1, and TCP ports 50120-50124 opened.
(Keep in mind this conflicts with what they've said elsewhere that a task server cannot be installed on Windows XP.)
Here's what they recommend:
P4 2GHz+, 1GB RAM, Windows 2003 Server Standard SP1+ (x86), IIS 6, 2GB HDD + 1.2x total size of packages, NTFS R1 if available. Altiris Agent, .NET 1.1, TCP ports 50120-50124 opened.
Hope this is helpful.