Two solutions:

Solution1:

Create a firewall rule to Block all the WAN traffic for all other host.

Then

Create a IPS policy to allow the particlular set host.

Go to SEPM-->Policy tab-->IPS policy-->Settings-->excluded host provide the IP range inside your (LAN) computers. & One specific WAN IP address (remote administrator).

So The port should be closed to all other WAN access except the excluded host.

Or:

Solution 2:

First rule:

Create a firewall rule to allow all the WAN traffic for your (LAN) computers. & One specific WAN IP address (remote administrator)..

Second Rule:

Create a firewall rule to Block all the WAN traffic for all other host.

first rule in stack get high priority and then it process the second rule.So The port should be closed to all other WAN access except the excluded host.

Idea behind this document:

How to Restrict Users to Specific Web Sites by Creating Firewall Rules for Managed Clients

http://www.symantec.com/docs/TECH92097

Hope this helps..................................