Hacker Newsnew | past | comments | ask | show | jobs | submit | spoondocz's commentslogin

Give me one reason I should use this when there is Apache Nutch and it's free?


The cost of crawling will be many times lower than Apache Nutch.

It's fully managed: nutch is NOTORIOUS for being hard to set up and maintain. It could take you months before you can launch a mid-scale crawl using nutch...


What's the largest crawl you can handle? meaning your upper limit...


We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: