I wouldn't say CS is particularly special in this regard. Few people lacking prior mathematics background are going to succeed as math majors. The main difference is that math training is a mandatory part of the curriculum throughout elementary school and much of high school, in addition to being a pre-requisite for most university math programs.
If CS was mandatory from grades 1-12 then things might be a lot less skewed in those first year introductory CS courses. Of course you're still going to run into the issue of kids who take up programming as a hobby at a young age and arrive at university with a large lead in skills. On the other hand I've seen math students who have done the same thing.
Aside from maybe intro calculus, I'm not sure what math you learn in K-12 has a lot to do with being a math major. Of course, you need some math (arithmetic) foundation for almost everything.
I actually did well in math throughout high school but it wasn't a particularly strong skill of mine in university. And I could have no more majored in math (or physics) than I could have flapped my arms and flown.
The math you learn in K-12 is necessary but not sufficient.
My secondary point still stands. Just as there are kids out there who grew up programming in their bedrooms who come into first year CS and crush it, there are kids out there studying extra math and writing math contests etc who will come into first year with a big advantage.
If CS was mandatory from grades 1-12 then things might be a lot less skewed in those first year introductory CS courses. Of course you're still going to run into the issue of kids who take up programming as a hobby at a young age and arrive at university with a large lead in skills. On the other hand I've seen math students who have done the same thing.