Good morning, all.
Got some time with the system in question to re-run the process and make sure I took appropriate notes.
Cycletech, we actually did try the uninstall the modem drivers/reinstalling after SEP. That was pretty much how we found out that the program in question is bypassing Windows completely. So, as far as security options... none in Windows as far as I can see, and nothing user-accessable in the program. Being a developer and yet having to support an antique, black-box software is a touch on the crazy-making side...
Jason, installed SEP, went to the control panel, ran the modem diagnostics, was able to get a modem query without any issues. I then went into the exceptions list, and added the folder which contains the files to the exception list. Should that be sufficient? The problem is that we have program A which calls on something like 10 or 15 different subprograms (in a subfolder - and yes, I did make sure I included subfolders when I put an exception on the main folder!) and I assumed that excluding the folder would be the same as excluding individual files. If that was a false assumption, I'll try it the other way.
Anyway, despite the folder being on the exception list and Windows being able to query the modem without a problem, no change; when the program tries to dial out, it seems to be trying to send data to the modem, but the modem isn't responding properly.
Prachand, I did confirm that LiveUpdate ran properly before I ran my tests, and confirmed that I was working with 11.0.4000.2263.