Different communities use different variants of the English language - different words, different grammar, and different meanings for the same words. The standard English on most English IQ tests is not going to be as accessible or understandable to somebody who speaks a different dialect and isn't as familiar with standard English.
It depends on what you use them for. If you want to compare the intelligence of people from different cultures, you need a culturally neutral test (whether such a thing exists is another question). For clinical diagnosis, when what you want to test is whether an individual is retarded or if there are suspicious patterns in their test results – like a large difference between their score at different subtests – I don't think it matters.
There are (or were) actually tests used by psychologists that include testing one's vocabulary. Which is knowledge, not intelligence.
From my experience of IQ tests administered in Germany the tests have multiple parts, among others one about reasoning about abstract shapes, one about spatial reasoning, and one about language (naming related words under time pressure and similar).
