If the biases are consistent on average then a change still matters. If I added or subtracted a random number between 0 and 100 to every reported IQ score in the world it would become virtually useless as a measure of each person, but the global average IQ would hardly change.
IQ is normalized so that the average is 100. Since the bias has to balance between the people who read high and the people who read low, population-level averages turn out OK so long as they are over a large enough population so that the high readers and low readers are both present.
