Yes, I have had some issues with loss of remote access to the computer, while removing NTP. Generally, it would first disconenct RDP session, and would not connect back. We would have to re-install Network drivers.
Having said that, this use to be on older versions of SEP and alsonot always.