I don't see that as terribly problematic. Unless a specific absolute scale is identified (as per another comment, e.g., the top of the scale meaning "I wrote the book on it"), I would assume the ratings were all relative to each other and the candidate's general skill/experience.

So 4 - 5 stars in an area just means "this is what I'm most comfortable working with and/or have the most experience with", not "if you rank yourself a 3 then I'm better than you at this" or "I'm a world-renowned expert at this". Otherwise, I'm not sure what you would expect from a fresh graduate who'd decided to use this format; 1 star for everything, or fractions of a star?

My lesson from that experience would be to throw away that useless 1-5 star rating scheme and skip to the interesting question immediately: "Tell us what you actually did with NodeJS."

To be honest the problem here is the question not the answer. It’s a bad question and shouldn’t be asked.

Why is it bad to ask about the actual experience someone had with something that I will be hiring them for? That makes no sense to me.

I'd say that the topic of "node.js" skill covers such a wide range of skills, experience and scenarios, that you'd need to be an expert already to judge how little you know. Someone who'd applied it to a single domain successfully can reasonably feel like they've got a good handle on it.

Your question is essentially a proxy for server-side architectures and applications, not nodejs itself. It invites misrepresentation.

I'm still not grasping how the question "what have you actually built with node?" is "a proxy for server-side architectures and applications, not nodejs itself." and "invites misrepresentation."

I feel like you guys think I asked them to rate themselves 1-5 on the subject. I did not, I got their resume, where they themselves rated them on the subject. With my question I was just checking whether that self-assessment was correct. I don't see how that is a bad question... Should I just believe their resume? Should I give them some programming assignments? Should I ask them to write a complex algo on a white board? I feel like just asking is a good way, personally, but maybe I'm wrong.

Maybe it’s like this: Given that I have no experience with Go, I’d still rate myself at least 3 when someone asks me how well I’d be able to work with it because the job they’re asking it for is ‘back-end development’.

Of course this falls apart the moment they ask me any Go question during the interview, but not when I’m actually asked to build a web application in it after being hired.

(the one time this happened people thought I’d been working with the new language for years, but no, it’s just a whole lot of transferrable skills)

The self-rating is useful because it gives you a measure of their perceived proficiency in skills relative to each other. You still have to ask a question like you did, but now that you know that "3/5 stars in Node" means advanced beginner level, you have a good idea what to expect when they rate their SQL skills 5/5 or their CSS 2/5.

Sorry: I misunderstood slightly, thinking it was something you put to them. In our tech interviewing I ignore self-assessed ratings, finding them useless for the reason you do, but I don't hold it against them if they're inaccurate just for the reasons I outlined.

Asking them about their actual experience is a fantastic question. Sounds like you got a good picture of their prior experience doing so.

Asking them to self rate themselves is useless however, as their answer showed.

They said the applicant had rated themselves on the resume. They weren't asked to rate themselves in the interview.

That's how I interpreted the comment and I'm pretty confident that's correct.

I ask candidates to rate themselves so I know where to go with the first few questions.

I ask for a 0 (never heard of it) to 10 (you made the thing or wrote a book on it) ranking and most people that have worked a couple years answer no higher than 5 or 6 but recent grads will say 8-9.

Pure anecdata but supports other comments about not knowing what you don’t know until you’ve been around different subjects for a bit.

Example: frontend design is self rated to a 6 but the person can not articulate the difference between raster and vector graphics (or doesn’t know what an SVG is or when to use one or not use one and vice versa).

I interviews interns from a school where they all ask their students to rates themselves like that.

At first I was put off, but now I interpret it as their own personal scale, with 5 being what they the best at, and not as "I know everything about this".

But even then, the best candidates are usually the ones who rated themselves the lowest.

That makes sense. The more I know about a topic, the more I know about how much I do not know. If I know a topic very well, I also know how very little I really do know, so I'd rate myself much lower.