Query a BIG query result


#1

Hello, new to redash and very interested so far. However, I’ve got a use case that I’m stumped with how to do. We have regular data queries that are run that we need to join the ID’s from that against a different data source to pull in a bunch of extra metadata. The data queries are in the 10k to 500k size, but the source that holds the metadata has 15-20M entries in couchbase. The query_result query_runner looks like it works by pulling the entire 15M entries and then trying to join that in a sqlite database. What we actually want is the 100k ID’s from the data query being passed to our couchbase query_runner to handle there. I think I can probably modify query_result to chunk somehow, but I didn’t know if there was a way to essentially pass a query result to a query runner instead? Hope that makes sense! Certainly a cross-platform joined query like we’re doing here is probably a bit of a corner case, but I’m happy to contribute back anything we do if there is a smart way to accomplish this.
Thanks!