You would need to remove the package from the Install Packages tab.
Keep in mind that if you applied and the client check in, it already picked up the task to auto-upgrade so even if you remove it, for those clients that already check in and got the task, they will still upgrade.