I've always found the greatest speed bump to come from adding more RAM to the SQL Server so that it's able to keep more of the SEP DB in memory.
Depends on your environment and where the slowdown is coming in for you though. I'm afraid this is going to be quite subjective...