You can block applications using the MD5 hash, but you cannot force SEP to quarantine files. Only files that appear to be threats (from AV signatures) will be quarantined.
How to block applications in SEP using MD5 -
https://www-secure.symantec.com/connect/forums/how-block-applications-sep-using-md5
If you have suspected files that SEP is not catching, please submit these to Security Response for analysis.
http://www.symantec.com/security_response/submitsamples.jsp