That's an odd one. I've seen the numbers fail to match in the other way around (i.e. "All Users" has the higher number) because All Users includes external users as well, but not the way you have described.
Do you use AD sync for your internal users, and therefore for Group Membership? Can you check if there are any AD-Sync issues?