How incorrect can my usage be if it's understood? :)

But seriously: Isn't considering material wealth one meaning of "materialism"?

Maybe the person you're replying to was thinking of the philosophical metaphysical concept of materialism, i.e. opposed to idealism. But your usage seems correct to me, it's that the word has multiple meanings.