And as I move along I discovered the location awareness feature so I don't have to duplicate groups. I combined the groups I split manually and used location awareness to set the management server list to the local SEPM at each site.
For others that wants to do what I'm trying:
1. Open SEPM
2. Click the Clients option on the left bar
3. You'll see the Add Location in the tasks below the group list.
Once you set that up you can turn group inheritance off for the Communications Settings then apply the site management server list you choose.