Worker time out, cannot start the redash server

Issue Summary

REDASH server always gives me worker timeout. Here is the debug logging info. It stuck at Registering Google Hangouts Chat (hangouts_chat) destinations If i send a new request to the server, it gives me a TIMEOUT and then Booting a new worker. Then it repeat everything until Registering Google Hangouts Chat.

2020-12-27T04:38:48.275353954Z [2020-12-27 04:38:48,275][PID:320][DEBUG][redash.destinations] Registering Google Hangouts Chat (hangouts_chat) destinations.
2020-12-27T04:39:16.068591825Z [2020-12-27 04:39:16 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:278)
2020-12-27T04:39:16.071430137Z [2020-12-27 04:39:16 +0000] [278] [INFO] Worker exiting (pid: 278)
2020-12-27T04:39:16.278121719Z [2020-12-27 04:39:16 +0000] [331] [INFO] Booting worker with pid: 331
2020-12-27T04:39:16.719180674Z [2020-12-27 04:39:16,718][PID:331][DEBUG][redash.query_runner] Registering Amazon Athena (athena) query runner.
2020-12-27T04:39:16.808714739Z [2020-12-27 04:39:16,808][PID:331][DEBUG][redash.query_runner] Registering Google Sheets (google_spreadsheets) query runner.
2020-12-27T04:39:16.808890037Z [2020-12-27 04:39:16,808][PID:331][DEBUG][redash.query_runner] Registering Graphite (graphite) query runner.
2020-12-27T04:39:16.870704703Z [2020-12-27 04:39:16,870][PID:331][DEBUG][redash.query_runner] Registering MongoDB (mongodb) query runner.
2020-12-27T04:39:16.871167331Z [2020-12-27 04:39:16,870][PID:331][DEBUG][redash.query_runner] Registering Couchbase (couchbase) query runner.
2020-12-27T04:39:16.877105418Z [2020-12-27 04:39:16,876][PID:331][DEBUG][redash.query_runner] Registering MySQL (mysql) query runner.
2020-12-27T04:39:16.877156548Z [2020-12-27 04:39:16,876][PID:331][DEBUG][redash.query_runner] Registering MySQL (Amazon RDS) (rds_mysql) query runner.
2020-12-27T04:39:16.891961023Z [2020-12-27 04:39:16,891][PID:331][DEBUG][redash.query_runner] Registering PostgreSQL (pg) query runner.
2020-12-27T04:39:16.892107049Z [2020-12-27 04:39:16,891][PID:331][DEBUG][redash.query_runner] Registering Redshift (redshift) query runner.
2020-12-27T04:39:16.892253993Z [2020-12-27 04:39:16,892][PID:331][DEBUG][redash.query_runner] Registering CockroachDB (cockroach) query runner.
2020-12-27T04:39:16.892810278Z [2020-12-27 04:39:16,892][PID:331][DEBUG][redash.query_runner] Registering Url (url) query runner.
2020-12-27T04:39:17.179383492Z [2020-12-27 04:39:17,178][PID:331][DEBUG][redash.query_runner] Registering InfluxDB (influxdb) query runner.
2020-12-27T04:39:17.179771804Z [2020-12-27 04:39:17,179][PID:331][DEBUG][redash.query_runner] Registering Kibana (kibana) query runner.
2020-12-27T04:39:17.179799410Z [2020-12-27 04:39:17,179][PID:331][DEBUG][redash.query_runner] Registering Elasticsearch (elasticsearch) query runner.
2020-12-27T04:39:17.180585556Z [2020-12-27 04:39:17,180][PID:331][DEBUG][redash.query_runner] Registering Amazon Elasticsearch Service (aws_es) query runner.
2020-12-27T04:39:17.183168360Z [2020-12-27 04:39:17,182][PID:331][DEBUG][redash.query_runner] Registering Presto (presto) query runner.
2020-12-27T04:39:17.194477285Z [2020-12-27 04:39:17,194][PID:331][DEBUG][redash.query_runner] Registering Hive (hive) query runner.
2020-12-27T04:39:17.194515318Z [2020-12-27 04:39:17,194][PID:331][DEBUG][redash.query_runner] Registering Hive (HTTP) (hive_http) query runner.
2020-12-27T04:39:17.194644369Z [2020-12-27 04:39:17,194][PID:331][DEBUG][redash.query_runner] Registering Databricks (databricks) query runner.
2020-12-27T04:39:17.213427750Z [2020-12-27 04:39:17,213][PID:331][DEBUG][redash.query_runner] Registering Impala (impala) query runner.
2020-12-27T04:39:17.225815372Z [2020-12-27 04:39:17,225][PID:331][DEBUG][redash.query_runner] Registering Vertica (vertica) query runner.
2020-12-27T04:39:17.226041577Z [2020-12-27 04:39:17,225][PID:331][DEBUG][redash.query_runner] Registering ClickHouse (clickhouse) query runner.
2020-12-27T04:39:17.251377229Z [2020-12-27 04:39:17,250][PID:331][DEBUG][redash.query_runner] Registering Yandex Metrica (yandex_metrika) query runner.
2020-12-27T04:39:17.251416030Z [2020-12-27 04:39:17,251][PID:331][DEBUG][redash.query_runner] Registering Yandex AppMetrica (yandex_appmetrika) query runner.
2020-12-27T04:39:17.251623069Z [2020-12-27 04:39:17,251][PID:331][DEBUG][redash.query_runner] Registering Rockset (rockset) query runner.
2020-12-27T04:39:17.259839007Z [2020-12-27 04:39:17,259][PID:331][DEBUG][redash.query_runner] Registering TreasureData (treasuredata) query runner.
2020-12-27T04:39:17.262252053Z [2020-12-27 04:39:17,262][PID:331][DEBUG][redash.query_runner] Registering Sqlite (sqlite) query runner.
2020-12-27T04:39:17.402840898Z [2020-12-27 04:39:17,402][PID:331][DEBUG][redash.query_runner] Registering DynamoDB (with DQL) (dynamodb_sql) query runner.
2020-12-27T04:39:17.405511703Z [2020-12-27 04:39:17,405][PID:331][DEBUG][redash.query_runner] Registering Microsoft SQL Server (mssql) query runner.
2020-12-27T04:39:17.406777941Z [2020-12-27 04:39:17,406][PID:331][DEBUG][redash.query_runner] MemSQL query runner enabled but not supported, not registering. Either disable or install missing dependencies.
2020-12-27T04:39:17.470037098Z [2020-12-27 04:39:17,469][PID:331][DEBUG][redash.query_runner] Registering Mapd (mapd) query runner.
2020-12-27T04:39:17.470297202Z [2020-12-27 04:39:17,470][PID:331][DEBUG][redash.query_runner] Registering JIRA (JQL) (jirajql) query runner.
2020-12-27T04:39:17.481050210Z [2020-12-27 04:39:17,480][PID:331][DEBUG][redash.query_runner] Registering Google Analytics (google_analytics) query runner.
2020-12-27T04:39:17.484789435Z [2020-12-27 04:39:17,484][PID:331][DEBUG][root] Checking for the appropriate ‘python-requests’ version…
2020-12-27T04:39:17.485359017Z [2020-12-27 04:39:17,484][PID:331][DEBUG][root] Detected version of ‘python-requests’ : 2.21.0. OK
2020-12-27T04:39:17.485473762Z [2020-12-27 04:39:17,485][PID:331][DEBUG][redash.query_runner] Registering Axibase Time Series Database (axibasetsd) query runner.
2020-12-27T04:39:17.514217833Z [2020-12-27 04:39:17,513][PID:331][DEBUG][redash.query_runner] Registering Salesforce (salesforce) query runner.
2020-12-27T04:39:17.543046829Z [2020-12-27 04:39:17,542][PID:331][DEBUG][passlib.utils.compat] loaded lazy attr ‘SafeConfigParser’: <class ConfigParser.SafeConfigParser at 0x7f019ba54600>
2020-12-27T04:39:17.543113712Z [2020-12-27 04:39:17,542][PID:331][DEBUG][passlib.utils.compat] loaded lazy attr ‘NativeStringIO’:
2020-12-27T04:39:17.543124083Z [2020-12-27 04:39:17,542][PID:331][DEBUG][passlib.utils.compat] loaded lazy attr ‘BytesIO’:
2020-12-27T04:39:18.061326412Z [2020-12-27 04:39:18,061][PID:331][DEBUG][redash.query_runner] Registering Query Results (results) query runner.
2020-12-27T04:39:18.061593274Z [2020-12-27 04:39:18,061][PID:331][DEBUG][redash.query_runner] Registering Prometheus (prometheus) query runner.
2020-12-27T04:39:18.078282414Z [2020-12-27 04:39:18,077][PID:331][DEBUG][redash.query_runner] Registering Qubole (qubole) query runner.
2020-12-27T04:39:18.096004484Z [2020-12-27 04:39:18,095][PID:331][DEBUG][redash.query_runner] Registering DB2 (db2) query runner.
2020-12-27T04:39:18.097806589Z [2020-12-27 04:39:18,097][PID:331][DEBUG][redash.query_runner] Registering Druid (druid) query runner.
2020-12-27T04:39:18.097966916Z [2020-12-27 04:39:18,097][PID:331][DEBUG][redash.query_runner] Registering Kylin (kylin) query runner.
2020-12-27T04:39:18.098295715Z [2020-12-27 04:39:18,098][PID:331][DEBUG][redash.query_runner] Registering Apache Drill (drill) query runner.
2020-12-27T04:39:18.112913586Z [2020-12-27 04:39:18,112][PID:331][DEBUG][redash.query_runner] Registering Uptycs (uptycs) query runner.
2020-12-27T04:39:18.123620699Z [2020-12-27 04:39:18,123][PID:331][DEBUG][snowflake.connector.ssl_wrap_socket] Injecting ssl_wrap_socket_with_ocsp
2020-12-27T04:39:18.125077598Z [2020-12-27 04:39:18,124][PID:331][DEBUG][snowflake.connector.auth] cache directory: /home/redash/.cache/snowflake
2020-12-27T04:39:18.388945042Z [2020-12-27 04:39:18,388][PID:331][DEBUG][redash.query_runner] Registering Snowflake (snowflake) query runner.
2020-12-27T04:39:18.407403505Z [2020-12-27 04:39:18,407][PID:331][DEBUG][redash.query_runner] Registering Phoenix (phoenix) query runner.
2020-12-27T04:39:18.407665752Z [2020-12-27 04:39:18,407][PID:331][DEBUG][redash.query_runner] Registering JSON (json) query runner.
2020-12-27T04:39:18.439246194Z [2020-12-27 04:39:18,438][PID:331][DEBUG][redash.query_runner] Registering Cassandra (Cassandra) query runner.
2020-12-27T04:39:18.439282941Z [2020-12-27 04:39:18,439][PID:331][DEBUG][redash.query_runner] Registering ScyllaDB (scylla) query runner.
2020-12-27T04:39:18.451436135Z [2020-12-27 04:39:18,451][PID:331][DEBUG][redash.query_runner] Registering Dgraph (dgraph) query runner.
2020-12-27T04:39:18.460179251Z [2020-12-27 04:39:18,459][PID:331][DEBUG][redash.query_runner] Registering Azure Data Explorer (Kusto) (azure_kusto) query runner.
2020-12-27T04:39:18.471247355Z [2020-12-27 04:39:18,470][PID:331][DEBUG][redash.query_runner] Registering Oracle (oracle) query runner.
2020-12-27T04:39:18.513928403Z [2020-12-27 04:39:18,513][PID:331][DEBUG][redash.query_runner] Registering Python (python) query runner.
2020-12-27T04:39:18.514057631Z [2020-12-27 04:39:18,513][PID:331][DEBUG][redash.destinations] Registering Email (email) destinations.
2020-12-27T04:39:18.514253196Z [2020-12-27 04:39:18,514][PID:331][DEBUG][redash.destinations] Registering Slack (slack) destinations.
2020-12-27T04:39:18.600599380Z [2020-12-27 04:39:18,600][PID:331][DEBUG][redash.destinations] Registering Webhook (webhook) destinations.
2020-12-27T04:39:18.600766707Z [2020-12-27 04:39:18,600][PID:331][DEBUG][redash.destinations] Registering HipChat (hipchat) destinations.
2020-12-27T04:39:18.600933696Z [2020-12-27 04:39:18,600][PID:331][DEBUG][redash.destinations] Registering Mattermost (mattermost) destinations.
2020-12-27T04:39:18.601059235Z [2020-12-27 04:39:18,600][PID:331][DEBUG][redash.destinations] Registering ChatWork (chatwork) destinations.
2020-12-27T04:39:18.604566869Z [2020-12-27 04:39:18,604][PID:331][DEBUG][redash.destinations] Registering PagerDuty (pagerduty) destinations.
2020-12-27T04:39:18.604642976Z [2020-12-27 04:39:18,604][PID:331][DEBUG][redash.destinations] Registering Google Hangouts Chat (hangouts_chat) destinations.

There is also no high cpu memory or io consumption.

Technical details:

• Redash Version: 8.0.0+b32245
• Browser/OS: Server is Centos 7, Chrome
• How did you install Redash: I installed with docker compose. It was running fine for month .

There have been issues in the past several weeks with integrations to Google API’s from Redash. Do you use the google hangouts chat alert destination? If not, you can comment out this line:

https://github.com/getredash/redash/blob/dcdff66e6260517e954785bb34da47e565402f1a/redash/settings/init.py#L403

And see if that works.

Is it possible to disable google api by changing the env file ? (I am using the docker deployment solution.) Like disable/enable query.runner. @k4s1m

What happens when you tried commenting out the line mentioned in my last post?

I don’t think you can remove query runners with environment variables.

I figure it out. It was a network issue. All containers were attached to my customized bridge network and each services were given a static ipv4 address. After I dropped the static ip addresses from my compose yaml file, everything works fine now. Though, It would be helpful if the log can show more details about the why the gunicorn worker was timeout.