That's not it. The problem is that individuals have cognitive biases. People rarely change their minds. As a society, this is offset by a slow, regular replacement of its members.

If you end that cycle of renewal, the rate at which our society reacts to change and adopts new ideas would significantly slow.

That's a problem regardless of how well the society aligns with my ideals.