I will suggest you to use firewall as without firewall sep is is not enough to protect you

I am not worried about the firewall part right now.
Can some tell if I withdraw the firewall policy from the SEPM does that mean the firewall on the client is unfunctional???

I know it will try to use the default firewall policy but if the client connects to a SEPM after the initial installation & the SEPM does not have firewall policy so will the client still use the default firewall policy??

Withdrawing the firewall policy via the SEPM has no affect on the clients. They will continue to use the same policy they have been using.

In the case of newly installed clients they will use whatever policy was built-in to the package they were installed from.

Hi,

      Please check the link given below which explains the configuration of the Application & Device Control Policy.

http://service1.symantec.com/SUPPORT/ent-security....

@kavin

Please go through this carefully...

Symantec Endpoint Protection installation features and properties

Feature tree ...
 

 
The feature tree shows four primary features as listed on the left. The Core feature must always be specified for installation. It contains thecore client communications functionality.
The other three features can be installed asstand-alone features. SAVMain installs antivirus and antispyware protection,

PTPMain installs TruScan proactive threat scanning technology, and ITPMaininstalls network threat protection.

COHMain and DCMain require two parents. COHMain is Proactive Threat
Scan and requires PTPMain and SAVMain. DCMain, which is Application and
Device Control, requires PTPMain and ITPMain

the feature does not work IF the parent feature is not installed...

 source: installation guide

Hence, as per your statement above "I am not worried about the firewall part right now" I'm asuming that you got a firewall already in your network  which is able to protect from "application layer/level attacks" I would say that just disable the F/W or create a balnk rule [Allow all ] would resolve the issue. 

However, as a personal recommendation I would still say that go with the symantec firewall ..coz most of the h/w appliance based f/w are not capable of protecting endpoints against app.layer attacks...Sym-firewall protects those pretty efficiently...as it's installed as a part of the package and at the endpoint [host] itself.

I've made this mistake once and relied on just one firewall[gateway level]..unaware about the fact that it's not capable of protecting application level attacks and my network got compromised.Since then I consider "layred security approach" and used symantec F/W which protects my network against app.layer attack.I' not worried how many F/W I got at the gateway level but I make sure I got one at the "endpoint/host" level.This's very imp in the current trend of malwares and attacks.Trust me !

If you're worried about the rules setup or configuration of the sym-firewall then consider some docs online...it's one of the most easiest F/W to configure and manage..which you can trust.