The spiritual baggage comes from the fact that in these traditions meditation is a tool to develop concentration so that the meditator can focus his/her full attention on some form of the question "who am I" - that is, "what is my true identity" - sometimes aided by focus on a koan (a puzzle meant to spur an insight into ones real identity)

Spoiler alert - you're supposed to discover by direct experience that your real self is neither a homunculus looking out through your eyes, nor is it coterminous with your body, and this direct experience that your true self is everything that is, is intended to lead to compassion for other beings.