Says who, you? Do you think phrases that are commonly used can't be offensive? What do idioms have to do with anything anyway?

You can pretend it's the PC police, or if you want to be the kind of person who isn't insensitive towards some subcultures you could try to avoid the phrase. There are plenty of alternatives.