250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per ma... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		sethammons on June 15, 2023 \| parent \| context \| favorite \| on: Crawling a quarter billion webpages in 40 hours (2... 250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine. Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.

onion2k on June 15, 2023 [–]

You didn't write your Go stuff 11 years ago though...

sethammons on June 15, 2023 | [–]

We started with Go version 1.2 which was released over ten years ago. Pretty darn close to 11 years.

https://go.dev/doc/go1.2

haggy102 on June 18, 2023 | | [–]

Wow crazy how time flies. I did not realize Go has been in production for over 10 years now.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact