It's not the results I find repugnant, it's the assumption that the results have any real world validity. The _something_ they're measuring is as likely to be demographic biases as it is technical or communication skills.
> it's the assumption that the results have any real world validity
Of course it has real world validity. It has real world validity regarding how well people are likely to do in tech interviews.
You can feel free to say that the way the tech industry does interviews is bad, or biased, or has all sorts of problems. But having vague measurements, such as "technical" or "communication" skills sounds very accurate to how tech interviews are actually done, in the real world.
All the moral outrage everyone is having about this, seems to have nothing to do with the accuracy of the report, which is about measuring tech interview performance. And it instead seems to be regarding the tech interview process in the industry, in general.
Over the last 22 years I've interviewed hundreds, hired and directly managed probably close to 100 individuals in widely varying environments, ranging from a 25k employee state university, to startups of various shapes and sizes, to the hottest SV IPO of 2020.
The reason for interview practices to be the way they are is to raise the floor for FAANG, who have high internal complexity and high salaries, leading to a very fat hiring funnel with a high risk of turning into dead weight in "rest and vest" mode once they get inside.
However humans are smart, and interviewers are lazy, so inevitably the people who optimize for this process start beating out more able engineers who don't have the time or inclination to jump through these hoops. In my experience, the proportion of really good software engineers is roughly equivalent across all companies with baseline competent technical leadership. FAANG does have a lot of the outliers on the high end, but they also have a lot of folks who can't tie their own shoes without the support of world-class tooling, infra support, and technical design guidance that those companies surround them with.
All I can say is that in my experience the average programmer at a company with a highly selective FAANG-style interview process is far sharper than the average programmer in the industry as a whole. Additionally, managers at more selective companies tend to be less parochial and less micro-managing.
The process isn't perfect, and it has some type I errors and a lot of type II errors, but it's a lot better than just throwing darts at a stack of resumes.
We're not in disagreement, note I explicitly did not say "the industry as a whole", and I added the qualifier "competent technical leadership". There's a lot packed into those three words, and without it you'll steadily bleed your best talent.
Micro-managing, non-technical leadership is the failure mode you're pointing out, and it's definitely the worst of all worlds, far worse than any failure mode at a FAANG. But on the other hand there is also parochial leadership who knows what they don't know and how to trust talent. Those environments can actually be fine for technical people. Granted, they won't necessarily get exposed to the exchange of ideas and mentorship from FAANG, but that's not a deal breaker in the modern internet age, and autodidactism has its place in furthering the state of the art by side-stepping social convergence to "best practices".
And on the flip side, I agree FAANG people are "sharper than average", but there are also headwinds to retaining the best talent. One is that you have to have a tolerance for moving slow, jumping through hoops, and generally dealing with a whole class of friction which many high performing engineers consider bullshit. Some will suck it up and deal with it to get the fat comp packages, but there is now an entire generation of <35 engineers who have had expectations set on comp levels based on a decade+ bull run of tech stocks which I suspect is unlikely to repeat over the next decade. There's also the appeal of working on classes of large problems that is only available at the biggest tech companies, but the actual interesting work is much fewer than the number of engineers. The majority are just dealing with incidental complexity and requirements of scale itself which can definitely occupy the mind, but may lead to an itch for more tangible impact.
Finally I will say there's a middle-ground between FAANG-style interviews and "throwing darts at a stack of resumes". If you are a small to mid-size company without the brand appeal and top-of-the-funnel recruiting volume of a FAANG, then you are absolutely shooting yourself in the foot by cargo-culting the FAANG approach. You know what the alternative is? Have qualified people do traditional interviews, going deep enough to get a gut feeling on their technical competence. Of course you'll get some Type I errors here, so then you have to actually pay attention to what they're doing once they start working. If they are not able to ramp and be productive in a reasonable amount of time, then you have to let them go (or at least pivot them into a position where they don't do damage). Big companies can't do that because there's enough chaos, lazy managers, and HR legal fears that Type II errors are a material risk. In summary, FAANG approach is solving for specific circumstances that most companies don't have, and it leaves a lot of talent on the table which is an arbitrage opportunity for companies willing to do the hard work to think about their recruiting strategy from first principles.
It doesn't necessarily say anything other than "where you work now has some predictive value for how well you'll be perceived by interviewers" in which case these aren't "top performers", they're "people with jobs" and it's just regurgitating a truism about how it's easier to get a job if you have a job.
I'm not buying this as a moral outrage question, I'm wondering if this is adding anything meaningful for us to look at or if it's just a surface-level puff piece masquerading as an analysis.
Except that interviewing.io interviews are specifically designed to be anonymous. This is even stated on their website. The interviewers do not see a resume or job history. As a candidate who's done a couple of interviews through them, the interviewers never asked anything about my background either. I don't recall even uploading a resume.
Yeah, but there's in-group jargon and technical approaches that FAANG and FAANG-adjacent engineers will pick up on. Just because you're anonymous doesn't mean you aren't unconsciously signaling your background to your interviewer.
Do you honestly believe this jargon is so unique to FAANG and "FAANG-adjacent" (that's a new one) SDEs that it can be used to pick them out, yet so secret it's not yet become industry-wide jargon?
Are you claiming they have no validity at all? Like if you built 2 teams: one team with candidates that all got 0% on the test and another team with candidates that all got 100%, you'd expect no difference at all in their real-world performance on a difficult problem?
If you're claiming something weaker than that, can you state it more precisely?
