Redash_worker_1 exited with code 1, Redis is throwing error for connection


#1

I am using redash (release/6.0.x), trying to run docker-compose up
Please see the below stacktrace:

Starting redash_redis_1 … done
Starting redash_postgres_1 … done
Starting redash_server_1 … done
Starting redash_worker_1 … done
Attaching to redash_redis_1, redash_postgres_1, redash_server_1, redash_worker_1
redis_1 | 1:C 25 Jan 05:21:17.412 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 25 Jan 05:21:17.412 # Redis version=4.0.12, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 25 Jan 05:21:17.412 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 25 Jan 05:21:17.413 * Running mode=standalone, port=6379.
redis_1 | 1:M 25 Jan 05:21:17.413 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 25 Jan 05:21:17.413 # Server initialized
redis_1 | 1:M 25 Jan 05:21:17.413 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 25 Jan 05:21:17.413 # Can’t handle RDB format version 9
redis_1 | 1:M 25 Jan 05:21:17.413 # Fatal error loading the DB: Invalid argument. Exiting.
postgres_1 | LOG: database system was shut down at 2019-01-25 05:20:15 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
redash_redis_1 exited with code 1
postgres_1 | LOG: autovacuum launcher started
worker_1 | Starting scheduler and 2 workers for queues: queries,scheduled_queries,celery…
server_1 | [2019-01-25 05:21:19,983][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
server_1 | [2019-01-25 05:21:20,027][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
worker_1 | [2019-01-25 05:21:20,924][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
worker_1 | [2019-01-25 05:21:20,964][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
server_1 | Traceback (most recent call last):
server_1 | File “/app/manage.py”, line 6, in
server_1 | from redash.cli import manager
server_1 | File “/app/redash/init.py”, line 75, in
server_1 | reset_new_version_status()
server_1 | File “/app/redash/version_check.py”, line 34, in reset_new_version_status
server_1 | latest_version = get_latest_version()
server_1 | File “/app/redash/version_check.py”, line 40, in get_latest_version
server_1 | return redis_connection.get(REDIS_KEY)
server_1 | File “/usr/local/lib/python2.7/dist-packages/redis/client.py”, line 880, in get
server_1 | return self.execute_command(‘GET’, name)
server_1 | File “/usr/local/lib/python2.7/dist-packages/redis/client.py”, line 578, in execute_command
server_1 | connection.send_command(*args)
server_1 | File “/usr/local/lib/python2.7/dist-packages/redis/connection.py”, line 563, in send_command
server_1 | self.send_packed_command(self.pack_command(*args))
server_1 | File “/usr/local/lib/python2.7/dist-packages/redis/connection.py”, line 538, in send_packed_command
server_1 | self.connect()
server_1 | File “/usr/local/lib/python2.7/dist-packages/redis/connection.py”, line 442, in connect
server_1 | raise ConnectionError(self._error_message(e))
server_1 | redis.exceptions.ConnectionError: Error -2 connecting to redis:6379. Name or service not known.
redash_server_1 exited with code 1
worker_1 | Traceback (most recent call last):
worker_1 | File “/usr/local/bin/celery”, line 11, in
worker_1 | sys.exit(main())
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/main.py”, line 16, in main
worker_1 | _main()
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py”, line 322, in main
worker_1 | cmd.execute_from_commandline(argv)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py”, line 496, in execute_from_commandline
worker_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 273, in execute_from_commandline
worker_1 | argv = self.setup_app_from_commandline(argv)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 479, in setup_app_from_commandline
worker_1 | self.app = self.find_app(app)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 501, in find_app
worker_1 | return find_app(app, symbol_by_name=self.symbol_by_name)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/app/utils.py”, line 359, in find_app
worker_1 | sym = symbol_by_name(app, imp=imp)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 504, in symbol_by_name
worker_1 | return imports.symbol_by_name(name, imp=imp)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py”, line 56, in symbol_by_name
worker_1 | module = imp(module_name, package=package, **kwargs)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py”, line 104, in import_from_cwd
worker_1 | return imp(module, package=package)
worker_1 | File “/usr/lib/python2.7/importlib/init.py”, line 37, in import_module
worker_1 | import(name)
worker_1 | File “/app/redash/init.py”, line 75, in
worker_1 | reset_new_version_status()
worker_1 | File “/app/redash/version_check.py”, line 34, in reset_new_version_status
worker_1 | latest_version = get_latest_version()
worker_1 | File “/app/redash/version_check.py”, line 40, in get_latest_version
worker_1 | return redis_connection.get(REDIS_KEY)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/redis/client.py”, line 880, in get
worker_1 | return self.execute_command(‘GET’, name)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/redis/client.py”, line 578, in execute_command
worker_1 | connection.send_command(*args)
worker_1 | File “/usr/local/lib/python2.7/dist-packages/redis/connection.py”, line 563, in send_command
worker_1 | self.send_packed_command(self.pack_command(*args))
worker_1 | File “/usr/local/lib/python2.7/dist-packages/redis/connection.py”, line 538, in send_packed_command
worker_1 | self.connect()
worker_1 | File “/usr/local/lib/python2.7/dist-packages/redis/connection.py”, line 442, in connect
worker_1 | raise ConnectionError(self._error_message(e))
worker_1 | redis.exceptions.ConnectionError: Error -2 connecting to redis:6379. Name or service not known.
redash_worker_1 exited with code 1


#2

Your Redis container fails to start. If it’s an existing environment you’re upgrading, try to recreate the Redis container.


#3

Yeah I recreated the Redis container, now it is failing to import json_loads with following stacktrace.

ubuntu@ip-172-31-1-59:~/redash$ git branch
master

  • release/5.0.x
    release/6.0.x
    ubuntu@ip-172-31-1-59:~/redash$ sudo docker-compose up
    Starting redash_redis_1 … done
    Starting redash_postgres_1 … done
    Starting redash_server_1 … done
    Starting redash_worker_1 … done
    Attaching to redash_postgres_1, redash_redis_1, redash_server_1, redash_worker_1
    postgres_1 | LOG: database system was shut down at 2019-01-25 06:29:38 UTC
    postgres_1 | LOG: MultiXact member wraparound protections are now enabled
    postgres_1 | LOG: database system is ready to accept connections
    postgres_1 | LOG: autovacuum launcher started
    redis_1 | 1:C 25 Jan 06:50:19.320 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
    redis_1 | 1:C 25 Jan 06:50:19.320 # Redis version=4.0.12, bits=64, commit=00000000, modified=0, pid=1, just started
    redis_1 | 1:C 25 Jan 06:50:19.320 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
    redis_1 | 1:M 25 Jan 06:50:19.322 * Running mode=standalone, port=6379.
    redis_1 | 1:M 25 Jan 06:50:19.322 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
    redis_1 | 1:M 25 Jan 06:50:19.322 # Server initialized
    redis_1 | 1:M 25 Jan 06:50:19.322 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
    redis_1 | 1:M 25 Jan 06:50:19.322 * DB loaded from disk: 0.000 seconds
    redis_1 | 1:M 25 Jan 06:50:19.322 * Ready to accept connections
    worker_1 | Starting scheduler and 2 workers for queues: queries,scheduled_queries,celery…
    server_1 | [2019-01-25 06:50:21,602][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
    server_1 | [2019-01-25 06:50:21,664][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
    server_1 | Traceback (most recent call last):
    server_1 | File “/app/manage.py”, line 6, in
    server_1 | from redash.cli import manager
    server_1 | File “/app/redash/init.py”, line 71, in
    server_1 | import_query_runners(settings.QUERY_RUNNERS)
    server_1 | File “/app/redash/query_runner/init.py”, line 175, in import_query_runners
    server_1 | import(runner_import)
    server_1 | File “/app/redash/query_runner/query_results.py”, line 9, in
    server_1 | from redash import models
    server_1 | File “/app/redash/models/init.py”, line 32, in
    server_1 | ImportError: cannot import name json_loads
    worker_1 | [2019-01-25 06:50:22,852][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
    worker_1 | [2019-01-25 06:50:22,894][PID:1][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
    redash_server_1 exited with code 1
    worker_1 | Traceback (most recent call last):
    worker_1 | File “/usr/local/bin/celery”, line 11, in
    worker_1 | sys.exit(main())
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/main.py”, line 16, in main
    worker_1 | _main()
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py”, line 322, in main
    worker_1 | cmd.execute_from_commandline(argv)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py”, line 496, in execute_from_commandline
    worker_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 273, in execute_from_commandline
    worker_1 | argv = self.setup_app_from_commandline(argv)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 479, in setup_app_from_commandline
    worker_1 | self.app = self.find_app(app)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 501, in find_app
    worker_1 | return find_app(app, symbol_by_name=self.symbol_by_name)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/app/utils.py”, line 359, in find_app
    worker_1 | sym = symbol_by_name(app, imp=imp)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/bin/base.py”, line 504, in symbol_by_name
    worker_1 | return imports.symbol_by_name(name, imp=imp)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py”, line 56, in symbol_by_name
    worker_1 | module = imp(module_name, package=package, **kwargs)
    worker_1 | File “/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py”, line 104, in import_from_cwd
    worker_1 | return imp(module, package=package)
    worker_1 | File “/usr/lib/python2.7/importlib/init.py”, line 37, in import_module
    worker_1 | import(name)
    worker_1 | File “/app/redash/init.py”, line 71, in
    worker_1 | import_query_runners(settings.QUERY_RUNNERS)
    worker_1 | File “/app/redash/query_runner/init.py”, line 175, in import_query_runners
    worker_1 | import(runner_import)
    worker_1 | File “/app/redash/query_runner/query_results.py”, line 9, in
    worker_1 | from redash import models
    worker_1 | File “/app/redash/models/init.py”, line 32, in
    worker_1 | ImportError: cannot import name json_loads
    redash_worker_1 exited with code 1

#4

I am not trying to upgrade the envirnoment. I am trying to start redash version 5.0 from github


#5

Just as a diagnostic step, if you try to start the redis image separately, does it work?

$ docker run --rm redis:3.0-alpine

For me, when I try it on my local development desktop, this is what’s shown:

$ docker run --rm redis:3.0-alpine
1:C 25 Jan 07:31:58.826 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.0.7 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

1:M 25 Jan 07:31:58.828 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 25 Jan 07:31:58.828 # Server started, Redis version 3.0.7
1:M 25 Jan 07:31:58.829 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 25 Jan 07:31:58.829 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 25 Jan 07:31:58.829 * The server is now ready to accept connections on port 6379

That last line “The server is now ready to accept connections on port 6379” is the important bit.

Note - to exit out of that, just hit Control+C. That’ll shut down and (in this case) automatically remove the left over container.


#6

Just noticed something that could be important.

Your redis version seems to be wrong. The docker-compose.yml in your GitHub branch clearly uses Redis version 3.

However, the version of Redis being shown in your startup logs shows it is using: Redis version=4.0.12.

Your docker-compose setup seems to be wonky. Probably need to re-create it, carefully using the correct docker-compose.yml file. :smile: