Schema loader painfully slow with many tables


#1

We are wondering a good way to disable to schema viewer in the query window.

Our database has hundreds of tables, which causes ~ 5-10 second lag when opening any query URL :frowning:

Perhaps a way to disable the javascript for that section? Would be fine with a hacky solution for now.

Cheers.


#2

I think the easiest way for you will be to disable schema fetching in the server code as it’s easier to tinker with as you can change it “in place”.

There are two places where you can disable schema loading:

  1. The API handler: the code is very obvious. Just change it to return an empty array ([]). If you have multiple data sources in your instance and you want to disable it only for one of them, you can add some logic there to return empty array only for specific requests.
  2. In the query runner: each query runner has a get_schema or _get_tables method (depends on which class it inherits). You can change it to return an empty array. The benefit of doing this at the query runner level is that you won’t needlessly query your database for the schema.

If you don’t mind Redash querying for the schema in the background, then go for option #1.
If you go for option #2, then make sure to make the change in the correct query runner and delete the Redis cache key for the schema (data_source:schema:<data source id>).

We’re planning to improve the handling of large schemas by avoiding loading all the schema in the frontend, but this will be only in a release or two from now.


#3

Update: next version is going to have better handling of large schemas: https://github.com/getredash/redash/pull/1246.

I tested it with 1669 tables and 38K columns. Do you know how many you have?


#4

~3500 tables, probably around that many columns. :sob:


#5

I think it should be OK even with 3500 tables.