Alex Monahan Robert Sahlin Mim Ideally you don't want your tasks to compete for the same resources airflow itself needs. It also allows for a more granular configuration of the resources. medium.com/bluecore-engin…
Rahul Jain Interesting, I find it awesome, but we stream data, even the data that is ingested via gcs. With BQ storage write api you can set up cdc as well. I have tested using CF and impressed by the performance.
Robert Sahlin I’m very curious about message based vs topic based architecture and details, sounds like an interesting idea.
Robert Sahlin There's legit use cases for ex. if you wish to load raw data where the schema could change.
Robert Sahlin Ah I see thanks. So when you deserialize, the metadata piece is all the same format, and the payload is separate and can only be deserialized once you determine the message type?
Mim Robert Sahlin Sudhir Hasbe Imagine that, it would just auto-magically change the slot pattern to save us money.
Robert Sahlin Ismael Ghalimi Mim Interesting! Never thought about this approach. Will try to test it out soon. I think it should be easy to have DuckDB queries running on Collab using BQ Storage API.
Mim JP Monteiro Robert Sahlin I have to agree. But the good news is that #Fabric is so much more than the AI stuff.
Mim Robert Sahlin Sarath Neelesh Salian 💻 I want to double like the last tweet.
This is the way.
Robert Sahlin Ismael Ghalimi Mim Couldn't resist trying out and was able to run a Collab with DuckDB querying BigQuery tables using the arrow interface!
If I'm correct, the Storage Read API is at least one order of magnitude cheaper than BQ, right?