The component Remove Items From Collection uses Remove(..) method from System.Collections.ArrayList .NET class which is based on equality checking by calling Object.Equals(..).
The default implementation of Equals supports reference equality for reference types, and bitwise equality for value types. There is one exception - strings (Text data types) are treated as value type.
So, in your case, removing items from collection by pointing at elements from another collection will work only for collections of simple data types (Texts, Numbers etc.).
For complex data types you can solve this problem by using Configurable Collection Filter on base collection using as a filtering criteria elements from another collection and then pass filtering result to Remove component.