Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Isn't BigQuery real-time as well? For me, the wow factor is that you can host this yourself.


I'm unfamiliar with BQ, but from what I understand BQ doesn't have any concept of streams... with maybe an exception for change-streams (change tracking) and that's very different and less powerful than what Materialize is able to do.

After all, in BQ you're only able to query what has been already has been ingested or is being digested and that is somewhat by definition not 'real-time'.


GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam).

There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job.

https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...


Thank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.


BigQuery does have streaming ingest and can pickup new rows immediately but you still have to run a query to get the latest results.

This system is more like BigQuery with materialized views that are constantly updated as new data arrives.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: