Hello,
No Cost for setting up new SEPM (no new license required).
No cost for Sybase (Embedded) database.
Embedded database can support up to 5,000 users for Symantec Endpoint Protection 12.1
However, unfortunately in replication - "If you use an embedded database, you can only connect one Symantec Endpoint Protection Manager"
Check this Article:
How replication works
https://support.symantec.com/en_US/article.HOWTO81035.html
In other words, you would have to get SQL in your environment for adding more SEPM as replication partners which would be a cost to you.
Regards,