Yes, this is why anything like affirmative action was always a terrible idea.
If relevant to who gets hired, you are literally discriminating against candidates on the basis of race. If irrelevant, you still have the "maybe they're just an affirmative action hire" pall hanging over those people. Nobody wins.
So how do you address structural discrimination in American society, including the workplace? (If your answer is "it doesn't exist", please flag me and move on.)
If relevant to who gets hired, you are literally discriminating against candidates on the basis of race. If irrelevant, you still have the "maybe they're just an affirmative action hire" pall hanging over those people. Nobody wins.