I have an issue with my Redash Setup. When I try to start Redash using the command
/opt/redash/current/bin/run gunicorn -b 127.0.0.1:5000 --name redash -w 4 --max-requests 1000 --log-level debug redash.wsgi:app
Sometimes it works and sometimes I randomly get errors showing “Worker Timeouts”:

[2018-04-11 11:33:11 +0000] [19694] [INFO] Starting gunicorn 19.7.1
[2018-04-11 11:33:11 +0000] [19694] [DEBUG] Arbiter booted
[2018-04-11 11:33:11 +0000] [19694] [INFO] Listening at: http://127.0.0.1:5000 (19694)
[2018-04-11 11:33:11 +0000] [19694] [INFO] Using worker: sync
[2018-04-11 11:33:11 +0000] [19702] [INFO] Booting worker with pid: 19702
[2018-04-11 11:33:11 +0000] [19707] [INFO] Booting worker with pid: 19707
[2018-04-11 11:33:11 +0000] [19712] [INFO] Booting worker with pid: 19712
[2018-04-11 11:33:11 +0000] [19717] [INFO] Booting worker with pid: 19717
[2018-04-11 11:33:11 +0000] [19694] [DEBUG] 4 workers
[2018-04-11 11:33:41 +0000] [19694] [CRITICAL] WORKER TIMEOUT (pid:19712)
[2018-04-11 11:33:41 +0000] [19694] [CRITICAL] WORKER TIMEOUT (pid:19707)
[2018-04-11 11:33:41 +0000] [19694] [CRITICAL] WORKER TIMEOUT (pid:19717)
[2018-04-11 11:33:41 +0000] [19694] [CRITICAL] WORKER TIMEOUT (pid:19702)
[2018-04-11 11:33:41 +0000] [19707] [INFO] Worker exiting (pid: 19707)
[2018-04-11 11:33:41 +0000] [19712] [INFO] Worker exiting (pid: 19712)
[2018-04-11 11:33:41 +0000] [19717] [INFO] Worker exiting (pid: 19717)
[2018-04-11 11:33:41 +0000] [19702] [INFO] Worker exiting (pid: 19702)
[2018-04-11 11:33:42 +0000] [19726] [INFO] Booting worker with pid: 19726
[2018-04-11 11:33:42 +0000] [19694] [DEBUG] 1 workers
[2018-04-11 11:33:42 +0000] [19727] [INFO] Booting worker with pid: 19727
[2018-04-11 11:33:42 +0000] [19736] [INFO] Booting worker with pid: 19736
[2018-04-11 11:33:42 +0000] [19737] [INFO] Booting worker with pid: 19737
[2018-04-11 11:33:42 +0000] [19694] [DEBUG] 4 workers

And so on…
The issue occurs in I’d say 90% of the cases when I try to start Redash but for some reason sometimes it works so I don’t know where to start looking for the issue. Any hints where I shall have a look to debug the issue?

Redash Version used: 3.0.0+b3134
Server: CentOS 7 vServer with 6GB or RAM and 2 Cores
Setup: Manually installed using the commands from https://github.com/oscasierra/redash-setup-centos

Thanks,
Nils

When running the commands from @oscasierra’s script, did you install supervisor?

I Installed but since it wasn’t working I stopped and disabled supervisor and manually issued the start command from the Supervisor script, just added the debug part

:+1: just wanted to make sure you don’t have two instances running.

I remember having this issue, but can’t recall what was causing it :man_facepalming: I think at the time it was due to not enough resources, but you seem to have a strong enough machine that it shouldn’t be a problem.

I would change the log level in your .env file to DEBUG as well to see if it shows anything meaningful.

Thanks for the hint with Debug setting. Maybe it is realted to this debug message?

[2018-04-11 14:14:11,659][PID:25686][DEBUG][redash.query_runner] BigQuery query runner enabled but not supported, not registering. Either disable or install missing dependencies.
[2018-04-11 14:14:11,666][PID:25686][INFO][requests.packages.urllib3.connectionpool] Starting new HTTP connection (1): metadata.google.internal

Full Debug Output:

[2018-04-11 14:14:11 +0000] [25678] [INFO] Starting gunicorn 19.7.1
[2018-04-11 14:14:11 +0000] [25678] [DEBUG] Arbiter booted
[2018-04-11 14:14:11 +0000] [25678] [INFO] Listening at: http://127.0.0.1:5000 (25678)
[2018-04-11 14:14:11 +0000] [25678] [INFO] Using worker: sync
[2018-04-11 14:14:11 +0000] [25686] [INFO] Booting worker with pid: 25686
[2018-04-11 14:14:11 +0000] [25689] [INFO] Booting worker with pid: 25689
[2018-04-11 14:14:11 +0000] [25696] [INFO] Booting worker with pid: 25696
[2018-04-11 14:14:11 +0000] [25701] [INFO] Booting worker with pid: 25701
[2018-04-11 14:14:11 +0000] [25678] [DEBUG] 4 workers
[2018-04-11 14:14:11,489][PID:25686][DEBUG][root] Creating Redis connection (redis://localhost:6379/0)
[2018-04-11 14:14:11,628][PID:25686][DEBUG][redash.query_runner] Registering Amazon Athena (athena) query runner.
[2018-04-11 14:14:11,659][PID:25689][DEBUG][root] Creating Redis connection (redis://localhost:6379/0)
[2018-04-11 14:14:11,659][PID:25686][DEBUG][redash.query_runner] BigQuery query runner enabled but not supported, not registering. Either disable or install missing dependencies.
[2018-04-11 14:14:11,666][PID:25686][INFO][requests.packages.urllib3.connectionpool] Starting new HTTP connection (1): metadata.google.internal
[2018-04-11 14:14:11,792][PID:25689][DEBUG][redash.query_runner] Registering Amazon Athena (athena) query runner.
[2018-04-11 14:14:11,823][PID:25689][DEBUG][redash.query_runner] BigQuery query runner enabled but not supported, not registering. Either disable or install missing dependencies.
[2018-04-11 14:14:11,830][PID:25689][INFO][requests.packages.urllib3.connectionpool] Starting new HTTP connection (1): metadata.google.internal
[2018-04-11 14:14:11,853][PID:25696][DEBUG][root] Creating Redis connection (redis://localhost:6379/0)
[2018-04-11 14:14:11,926][PID:25701][DEBUG][root] Creating Redis connection (redis://localhost:6379/0)
[2018-04-11 14:14:11,982][PID:25696][DEBUG][redash.query_runner] Registering Amazon Athena (athena) query runner.
[2018-04-11 14:14:12,014][PID:25696][DEBUG][redash.query_runner] BigQuery query runner enabled but not supported, not registering. Either disable or install missing dependencies.
[2018-04-11 14:14:12,021][PID:25696][INFO][requests.packages.urllib3.connectionpool] Starting new HTTP connection (1): metadata.google.internal
[2018-04-11 14:14:12,057][PID:25701][DEBUG][redash.query_runner] Registering Amazon Athena (athena) query runner.
[2018-04-11 14:14:12,090][PID:25701][DEBUG][redash.query_runner] BigQuery query runner enabled but not supported, not registering. Either disable or install missing dependencies.
[2018-04-11 14:14:12,096][PID:25701][INFO][requests.packages.urllib3.connectionpool] Starting new HTTP connection (1): metadata.google.internal
[2018-04-11 14:14:41 +0000] [25678] [CRITICAL] WORKER TIMEOUT (pid:25696)
[2018-04-11 14:14:41 +0000] [25678] [CRITICAL] WORKER TIMEOUT (pid:25689)
[2018-04-11 14:14:41 +0000] [25678] [CRITICAL] WORKER TIMEOUT (pid:25701)
[2018-04-11 14:14:41 +0000] [25678] [CRITICAL] WORKER TIMEOUT (pid:25686)
[2018-04-11 14:14:41 +0000] [25696] [INFO] Worker exiting (pid: 25696)
[2018-04-11 14:14:41 +0000] [25689] [INFO] Worker exiting (pid: 25689)
[2018-04-11 14:14:41 +0000] [25686] [INFO] Worker exiting (pid: 25686)
[2018-04-11 14:14:41 +0000] [25701] [INFO] Worker exiting (pid: 25701)

Try running curl http://metadata.google.internal – does it hang or take a long time?

Yes, when running it on the server it takes a long time.

bash-4.2$ curl http://metadata.google.internal
curl: (7) Failed connect to metadata.google.internal:80; Connection timed out

When running it on my local machine though I get a result immediately:

nils@enterprise ~ $ curl http://metadata.google.internal
curl: (6) Could not resolve host: metadata.google.internal

Probably network/firewall issues.

If you don’t plan on querying BigQuery, you can add to your .env file:

export REDASH_DISABLED_QUERY_RUNNERS=redash.query_runner.big_query

If you do want to use BigQuery, then you can change this method to always return False.

Thanks! I changed the method as suggested and don’t get the error anymore now. Strange that it sometimes worked, maybe some routing issue, not sure. But so far it works, thanks for your help!

1 Like