On a related note: When I indexed the whole English wikipedia last year, I was surprised, that it was possible to have a JSON version of it indexed[1] and searchable within half an hour on my laptop.
That being said, when it comes to indexing wikipedia, the indexing can be done well across multiple threads internally by elasticsearch. Multithreading the reading/parsing isn't a huge win. Doing decompression in a separate thread is however.
Yes, ES uses multiple threads nicely. But as you move to 32 or 64 cores - in my experience - a single threaded client won't keep ES/Lucene busy enough.
With SOLR, it's similar:
> Sometimes you need to index a bunch of documents really, really fast. [...] The solution is two-fold: batching and multi-threading
As you comment, Freebase is bigger than Wikidata. It is 22GB compressed (250GB uncompressed) while Wikidata is 5GB compressed (49GB uncompressed) [1].
Said that, I believe the process described in the blog post is not loading the whole Wikidata dump into memory and it would work the same to process Freebase or even larger data dumps with your laptop.
From the post:
How Akka Streams can be used to process the Wikidata dump in parallel and using constant memory with just your laptop.
Biased reply (I'm a data scientist there): Common Crawl[1]. We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone completely free.
From their video:
The presenter: "Why would you (the assistent lady) be interested in cars?"
The assistent: "I'm the perfect chick to be into Masserati."
It's a bit disturbing to see an employee presenting her personal life, kids, interests, and what not. Good job, IntentHQ!
I wouldn't say it was disturbing, but it was definitely cringe worthy. A lot of their blog feels that way. They've gone for an informal corporate approach like using puns[1] and memes[2] as headings. Even that video felt badly scripted; like it was meant to sound like an informal pub conversation but instead it came off awkward and unprofessional.
I'm sure their products are of the highest quality, but their blog isn't a great advert in my opinion.
I have to say I have mixed feelings about the video. On one hand I understand there's a whole world of people out there, and I don't mind openness and honesty. Big thumbs to her for being honest and cool. On the other hand, it goes the other way when you're bragging about your awesome product that analyzes people's life and sells that info to corporations.
The privacy thing didn't really bother me because it's either fake data or she's consented to publishing real data about her - either way it's a considered decision. My issue was just how awkward the presentation was delivered. Maybe that could have been resolved if they used a fictional character like Homer Simpson? But then that would have it's own issues.
Yes, the initial parts of the example could be accomplished with awk and xargs, but as the article goes on to demonstrate, even doing something like printing every nth element would be difficult.
I think the intent was for this to be more of a demonstrative example, and with a more complex, evolving, real-world processing pipeline, Akka streams could be really useful.
[1] Using parallel bulk indexer for ES: https://github.com/miku/esbulk