There is actually a good analytical result on how vector similarity can easily fail to recover relevant information https://arxiv.org/pdf/2403.05440

> For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization.

I am not strong in mathematics but if this paper claims run opposite to each other.