First, some background. We were setting up a new box running Server 2008 R2 Core on x64. In my department, we push out SEP to all our servers via Group Policy. On this new Server 2008 R2 Core on x64 box the install of SEP failed. We were puzzled until we ran into another application (in this case VMware Tools) that did not install properly either. We discovered that Server 2008 R2 Core does not have an "Installer" folder inside the Windows folder--this would usually be C:\Windows\Installer. Upon creating the folder, VMware Tools installed fine and on a reboot, SEP installed just fine as well.
Also, we were pushing out RU5, as that is necessary for full support on Server 2008 R2.
So, if you run into trouble installing SEP (or anything else for that matter) on Server 2008 R2 Core, make sure you have an "Installer" directory in your Windows folder.