I wouldn't blame people. 'Fruit' is a perfectly fine English word.

I would blame botanists for overloading everyday terms with their own specialised meanings.

Mathematicians are also prone to this, but I guess it's less likely someone will mix up a ring or field in the everyday sense and in the mathematical sense.