I have the same problem, but I think it's the way our brains are conditioned. Also, I don't know how useful it is to learn random intervals, around 80 % of the music around has a fixed key. The reason in my opinion why min3/maj3 confusion happens is because we always think or want a reference point ...