Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It's not necessarily a lot of sites that block non-popular bots - but often it's big sites (i.e. content-centric sites such as Social Media). Think Yelp, Twitter, LinkedIn, Instagram, etc.

That can add up to a serious percentage of the web.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: