If you ever intent to study law, become involved in a situation dealing with disparate impact, or are at the receiving end of disparate impact, knowing the legal definition may be helpful too. The DoJ spells[1] out the legal definition of disparate impact as so:
ELEMENTS TO ESTABLISH ADVERSE DISPARATE IMPACT UNDER TITLE VI
Identify the specific policy or practice at issue; see Section C.3.a.
Establish adversity/harm; see Section C.3.b.
Establish disparity; see Section C.3.c.
Establish causation; see Section C.3.d.
My point is that by the plain meaning of words you're right, disparate impact means any two groups impacted differently, regardless of anything else. In law, it means that an employment, housing, etc. policy has a disproportionately adverse impact on members of a protected class compared to non-members of that same class. It's much more specific and narrowly defined.