Solution for SMG 10.5
Protocols/Domains
Add the domain for wich you want TLS
In the Acceptance tab, Uncheck Local Domain and check Reject mail from this domain if not sent using TLS
In the Delivery tab, Check Optional delivery encryption and check Require TLS encryption and don't verify certificate (or Require TLS encryption and verify certificate)