Schema browser is empty and error 500 on data sources page

Support Self Hosted Redash Support
1

Schema browser is empty for specific, newly created datasource.

When I switching to this datasource to make a query after a while I get message
"Schema refresh failed . Please try again later "

Query window shows and executes sql properly.

Redash 10.0.0-beta.b49597, deployed to AWS EKS cluster using https://getredash.github.io/contrib-helm-chart/
Database - PostgreSQL RDS instance in AWS

redash-server throws error (browser shows error 500 {“message”: “Internal Server Error”})

[2021-08-11 14:17:29,216[PID:11[ERROR[redash.app Exception on /api/jobs/e04f43a0-cf8d-4349-b7b4-c8d2f5e3ac5f [GET
Traceback (most recent call last):
File “/usr/local/lib/python3.7/site-packages/flask/app.py”, line 1949, in full_dispatch_request
rv = self.dispatch_request()
File “/usr/local/lib/python3.7/site-packages/flask/app.py”, line 1935, in dispatch_request
return self.view_functions[rule.endpoint
File “/usr/local/lib/python3.7/site-packages/flask_restful/init.py”, line 458, in wrapper
resp = resource(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/flask_login/utils.py”, line 261, in decorated_view
return func(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/flask/views.py”, line 89, in view
return self.dispatch_request(*args, **kwargs)
File “/app/redash/handlers/base.py”, line 33, in dispatch_request
return super(BaseResource, self).dispatch_request(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/flask_restful/init.py”, line 573, in dispatch_request
resp = meth(*args, **kwargs)
File “/app/redash/handlers/query_results.py”, line 462, in get
job = Job.fetch(job_id)
File “/usr/local/lib/python3.7/site-packages/rq/job.py”, line 299, in fetch
job.refresh()
File “/usr/local/lib/python3.7/site-packages/rq/job.py”, line 518, in refresh
raise NoSuchJobError(‘No such job: {0}’.format(self.key))
rq.exceptions.NoSuchJobError: No such job: b’rq:job:e04f43a0-cf8d-4349-b7b4-c8d2f5e3ac5f’
[2021-08-11 14:17:29,218[PID:11[INFO[metrics method=GET path=/api/jobs/e04f43a0-cf8d-4349-b7b4-c8d2f5e3ac5f endpoint=job status=500 content_type=application/json content_length=36 duration=2.26 query_count=2 query_duration=4.99

2

Possible duplicate of this one from last year: Redash can't connect any datasource

The problem is how Redash is connecting to Redis. Seems like you probably have two Redis servers running.

3

Initially I created separate standalone Redis instance and separate PostgreSQL instance to work with this Redash setup.

Migration from standalone docker deployment to kubernetes deployment was made via database dump.
So users, groups and databases were migrated looks like more-less properly…

The current installation is configured to use 2 redash servers for redundancy. Does it make sense to shrink redash-servers and keep only one?

4

Unless you have massive concurrency you don’t need two Redash servers. We have 10k+ active users off a single server, for reference…

[edit] Obviously you’ll need more workers. But the server component in Docker is very efficient (nginx does well at high volumes)

5

Jesse - is it possible to use Redash without Redis ?

6

No. Redis is required for the message queue. Why would you want to run without it?

7

Just restarted redash and redis cluster
Got logs from install container

…
This will retry connections until PostgreSQL/Redis is up, then perform database installation/migrations as needed.
Using external postgresql database
Using external redis database
Starting attempt 0 of 10
Return code: 124
Status:
Status command timed out after 10 seconds.
Waiting 10 seconds before retrying.
Starting attempt 1 of 10
Return code: 0
Status: {
“version”: “10.0.0-beta”,
“workers”: ,
“redis_used_memory”: 5637680,
“redis_used_memory_human”: “5.38M”,
“queries_count”: 1308,
“query_results_count”: 9843,
“unused_query_results_count”: 7502,
“dashboards_count”: 73,
“widgets_count”: 165,
“manager”: {
“queues”: {
“default”: {
“size”: 45
},
“schemas”: {
“size”: 8
},
“queries”: {
“size”: 0
},
“periodic”: {
“size”: 23
}
}
},
“database_metrics”: {
“metrics”: [
[
“Query Results Size”,
2181537792
],
[
“Redash DB Size”,
2508628847
]
]
}
}
Database appears to already be installed.
…

Looks good, database and redis are up and running but now ALL datasources generate message
"Schema refresh failed . Please try again later "

8

also in redash-server logs

[2021-08-11 19:18:04,791[PID:10[ERROR[redash.app Exception on /api/jobs/b7e816d8-f112-4518-bd1a-492d6a72cf6c [GET
Traceback (most recent call last):
File “/usr/local/lib/python3.7/site-packages/flask/app.py”, line 1949, in full_dispatch_request
rv = self.dispatch_request()
File “/usr/local/lib/python3.7/site-packages/flask/app.py”, line 1935, in dispatch_request
return self.view_functions[rule.endpoint
File “/usr/local/lib/python3.7/site-packages/flask_restful/init.py”, line 458, in wrapper
resp = resource(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/flask_login/utils.py”, line 261, in decorated_view
return func(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/flask/views.py”, line 89, in view
return self.dispatch_request(*args, **kwargs)
File “/app/redash/handlers/base.py”, line 33, in dispatch_request
return super(BaseResource, self).dispatch_request(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/flask_restful/init.py”, line 573, in dispatch_request
resp = meth(*args, **kwargs)
File “/app/redash/handlers/query_results.py”, line 462, in get
job = Job.fetch(job_id)
File “/usr/local/lib/python3.7/site-packages/rq/job.py”, line 299, in fetch
job.refresh()
File “/usr/local/lib/python3.7/site-packages/rq/job.py”, line 518, in refresh
raise NoSuchJobError(‘No such job: {0}’.format(self.key))
rq.exceptions.NoSuchJobError: No such job: b’rq:job:b7e816d8-f112-4518-bd1a-492d6a72cf6c’
[2021-08-11 19:18:04,792[PID:10[INFO[metrics method=GET path=/api/jobs/b7e816d8-f112-4518-bd1a-492d6a72cf6c endpoint=job status=500 content_type=application/json content_length=36 duration=2.02 query_count=2 query_duration=3.62

9

How many actual redis instances are you running though? The issue it seems is that you have more than one.

10

Yes ,we had 2 redis nodes in cluster, SSL enabled.
REDASH_REDIS_URL set as rediss://user:pass@redishost:6379/0 format
and deployed with
–set externalRedis=${REDASH_REDIS_URL}

I removed secondary node and left only primary node. Then rebooted this Redis node and Redash as well
Still getting the same behaviour

11

If you use multiple redis’s you will definitely have jobs get lost. I’m not a K8S user but your problem is almost certainly that Redash is looking for a specific job number in a queue that doesn’t contain it.

12

I reduced redis nodes to only one node in cluster which is hosted by AWS Elasticache

Mode|Shards|Nodes|Node Type|
Redis|1 |1 node|cache.t3.micro|

13

I even removed and created one datasource from scratch - still the same error for the schema displaying.
Dashboards and queries work fine

14

I switched my DEV setup to internal redis
…
Using Redis: redis://:******@redash-dev-redis-master:6379/0
…

Didn’t help and still getting the same error

15

I wonder if your setup didn’t create a schemas queue?

16

Also curious what happens if you try to manually kick-off a schema refresh with the API?

GET /api/data_sources/<data source id>/schema?refresh

17

http://localhost:5000/api/data_sources/58/schema?refresh
{“job”: {“id”: “f7cf776d-6104-4b00-aa11-5f6899c2fe9d”, “updated_at”: 0, “status”: 1, “error”: “”, “result”: null, “query_result_id”: null}}

18

So the job is enqueued properly.

What happens when you check the job status?

GET api/jobs/f7cf776d-6104-4b00-aa11-5f6899c2fe9d

19

http://localhost:5000/api/jobs/f7cf776d-6104-4b00-aa11-5f6899c2fe9d
{“message”: “Internal Server Error”}

20

If you post about an internal server error please post the log message :wink: