Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yep. Built a crawler, an indexer/queryprocessor, and an engine responsible for merging/compacting indexes.

Crawling was tricky. Something like stackoverflow will stop returning pages when it detects that you're crawling, much sooner than you'd expect.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: