Endpoint Protection

If I have 1 Server with hMailServer (email) which has 40 mailboxes, how many licenses would I need to purchase to protect this server with endpoint protection?

Is it just one license for the server, or is it one for server + 1 for each mailbox?

It will be one license for Endpoint Protection, however you should be using Symante Mail Security for SMTP to scan the mail.

Yes, SEP should not be used to handle the email part. Use a product specifically designed to work with email systems. SEP should handle the OS and file system.
In fact, ideally then youd build exclusions so that SEP will leave the mailboxes alone and let the mail AV handle that.
But SEP is licensed per-machine..........

hMailServer can use external AV software to scan all email traffic before it is copied to recepient mailbox. If virus is found the message is deleted and recepient and sender are notified of this.

http://www.hmailserver.com/documentation/v5.0/?page=feature_antivirus_external

And also it is often recommended on hMailServer-forum, that it would be vise to exclude the email data directory from AV scan.

So what would Symantec Mail Security for SMTP do differently on this situation? What improvements would it offer?

In this situation if you want only to scan for viruses you can use DoScan.exe from SEP.
SMS will offer you spam filtering besides AV protection.

Edit:
SMS for SMTP is part of Symantec Multi-Tier Protection and Premium Anti-spam is also free with this package.