Invalid Recipient rejection should not affect the reputation of your sending IP addresses. Enabling recipient rejecting in SBG will cause the message to be rejected during the converstation between the sender and your MTA, pre-data. You IP gets listed on the CBL by your MTA sending e-mails to one of the CBL's honeypot e-mail addresses.
We get 20 M e-mails per day and have Recipient rejection enabled (reject, not drop) and don't see any issues. Plus, you need to do this to participate in the probe network. About 98% of our mail is blocked by Symantec's reputation service, 1-2 % by invalid recipient.