I disagree with this article in so many ways. I have a CS degree and I work with many people who don’t, and they are just as good as (or even better than) me at all the points raised by the article.
They meet deadlines, they are incredibly good at communication and collaboration and they have pretty good networking. Most of these traits come from the fact that they needed to develop them in order to succeed in learning by themselves.
It is a pretty limited view of the world to think that only college can bring you this. Immersing yourself in a coding bootcamp for some people means leaving the jobs they need to survive in order to have a better job in the future. I can’t imagine how being on college can teach more about meeting deadlines, teamwork, communication and perseverance than that.
I wish this article provided more facts to back its beliefs up.
Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.
I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.
Scrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address.
And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.
It’s a collection of snippets to explain behaviors that may be considered unexpected.