I’m having problems running the docker-compose self-hosted solution. The worker
component seems not be able to access the Postgres database. However, running docker-compose -f docker-compose.yml run server create_db
seems to work without error.
I get this error: worker_1 | [2017-02-08 05:53:49,111: ERROR/Beat] Removing corrupted schedule file 'celerybeat-schedule': DBAccessError(13, 'Permission denied')
This appears in the docker-compose output for the command docker-compose -f docker-compose.yml up
:
redash_redis_1 is up-to-date
redash_postgres_1 is up-to-date
Creating redash_server_1
Creating redash_worker_1
Attaching to redash_redis_1, redash_postgres_1, redash_server_1, redash_worker_1
redis_1 | [1] 08 Feb 05:53:34.536 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | _._
redis_1 | _.-``__ ''-._
redis_1 | _.-`` `. `_. ''-._ Redis 2.8.23 (00000000/0) 64 bit
redis_1 | .-`` .-```. ```\/ _.,_ ''-._
redis_1 | ( ' , .-` | `, ) Running in stand alone mode
redis_1 | |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
redis_1 | | `-._ `._ / _.-' | PID: 1
redis_1 | `-._ `-._ `-./ _.-' _.-'
redis_1 | |`-._`-._ `-.__.-' _.-'_.-'|
redis_1 | | `-._`-._ _.-'_.-' | http://redis.io
redis_1 | `-._ `-._`-.__.-'_.-' _.-'
redis_1 | |`-._`-._ `-.__.-' _.-'_.-'|
redis_1 | | `-._`-._ _.-'_.-' |
redis_1 | `-._ `-._`-.__.-'_.-' _.-'
redis_1 | `-._ `-.__.-' _.-'
redis_1 | `-._ _.-'
redis_1 | `-.__.-'
redis_1 |
redis_1 | [1] 08 Feb 05:53:34.549 # Server started, Redis version 2.8.23
redis_1 | [1] 08 Feb 05:53:34.549 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | [1] 08 Feb 05:53:34.549 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | [1] 08 Feb 05:53:34.549 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | [1] 08 Feb 05:53:34.549 * The server is now ready to accept connections on port 6379
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | creating configuration files ... ok
postgres_1 | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
postgres_1 | initializing pg_authid ... ok
postgres_1 | initializing dependencies ... ok
postgres_1 | creating system views ... ok
postgres_1 | loading system objects' descriptions ... ok
postgres_1 | creating collations ... ok
postgres_1 | creating conversions ... ok
postgres_1 | creating dictionaries ... ok
postgres_1 | setting privileges on built-in objects ... ok
postgres_1 | creating information schema ... ok
postgres_1 | loading PL/pgSQL server-side language ... ok
postgres_1 | vacuuming database template1 ... ok
postgres_1 | copying template1 to template0 ... ok
postgres_1 | copying template1 to postgres ... ok
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | ****************************************************
postgres_1 | WARNING: No password has been set for the database.
postgres_1 | This will allow anyone with access to the
postgres_1 | Postgres port to access your database. In
postgres_1 | Docker's default configuration, this is
postgres_1 | effectively any other container on the same
postgres_1 | system.
postgres_1 |
postgres_1 | Use "-e POSTGRES_PASSWORD=password" to set
postgres_1 | it in "docker run".
postgres_1 | ****************************************************
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | postgres -D /var/lib/postgresql/data
postgres_1 | or
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 | waiting for server to start....LOG: database system was shut down at 2017-02-08 05:53:36 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
postgres_1 | done
postgres_1 | server started
postgres_1 | ALTER ROLE
postgres_1 |
postgres_1 |
postgres_1 | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | LOG: received fast shutdown request
postgres_1 | LOG: aborting any active transactions
postgres_1 | LOG: autovacuum launcher shutting down
postgres_1 | LOG: shutting down
postgres_1 | waiting for server to shut down....LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | LOG: database system was shut down at 2017-02-08 05:53:37 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
worker_1 | Starting scheduler and 2 workers for queues: queries,scheduled_queries,celery...
worker_1 | [2017-02-08 05:53:46,515][PID:5][WARNING][redash.query_runner] BigQueryGCE query runner enabled but not supported, not registering. Either disable or install missing dependencies.
worker_1 | [2017-02-08 05:53:48,684: WARNING/MainProcess] /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161: CDeprecationWarning:
worker_1 | Starting from version 3.2 Celery will refuse to accept pickle by default.
worker_1 |
worker_1 | The pickle serializer is a security concern as it may give attackers
worker_1 | the ability to execute any command. It's important to secure
worker_1 | your broker from unauthorized access when using pickle, so we think
worker_1 | that enabling pickle should require a deliberate action and not be
worker_1 | the default choice.
worker_1 |
worker_1 | If you depend on pickle then you should set a setting to disable this
worker_1 | warning and to be sure that everything will continue working
worker_1 | when you upgrade to Celery 3.2::
worker_1 |
worker_1 | CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
worker_1 |
worker_1 | You must only enable the serializers that you will actually use.
worker_1 |
worker_1 |
worker_1 | warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
worker_1 |
worker_1 | -------------- celery@f7158f86b438 v3.1.23 (Cipater)
worker_1 | ---- **** -----
worker_1 | --- * *** * -- Linux-4.4.43-boot2docker-x86_64-with-Ubuntu-16.04-xenial
worker_1 | -- * - **** ---
worker_1 | - ** ---------- [config]
worker_1 | - ** ---------- .> app: redash:0x7f466e41f850
worker_1 | - ** ---------- .> transport: redis://redis:6379/0
worker_1 | - ** ---------- .> results: redis://redis:6379/0
worker_1 | - *** --- * --- .> concurrency: 2 (prefork)
worker_1 | -- ******* ----
worker_1 | --- ***** ----- [queues]
worker_1 | -------------- .> celery exchange=celery(direct) key=celery
worker_1 | .> queries exchange=queries(direct) key=queries
worker_1 | .> scheduled_queries exchange=scheduled_queries(direct) key=scheduled_queries
worker_1 |
worker_1 | [tasks]
worker_1 | . redash.tasks.check_alerts_for_query
worker_1 | . redash.tasks.cleanup_query_results
worker_1 | . redash.tasks.cleanup_tasks
worker_1 | . redash.tasks.execute_query
worker_1 | . redash.tasks.record_event
worker_1 | . redash.tasks.refresh_queries
worker_1 | . redash.tasks.refresh_schemas
worker_1 | . redash.tasks.send_mail
worker_1 | . redash.tasks.subscribe
worker_1 | . redash.tasks.version_check
worker_1 |
worker_1 | [2017-02-08 05:53:49,047: INFO/Beat] beat: Starting...
worker_1 | [2017-02-08 05:53:49,111: ERROR/Beat] Removing corrupted schedule file 'celerybeat-schedule': DBAccessError(13, 'Permission denied')
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 376, in setup_schedule
worker_1 | self._store = self._open_schedule()
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 366, in _open_schedule
worker_1 | return self.persistence.open(self.schedule_filename, writeback=True)
worker_1 | File "/usr/lib/python2.7/shelve.py", line 243, in open
worker_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
worker_1 | File "/usr/lib/python2.7/shelve.py", line 227, in __init__
worker_1 | Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
worker_1 | File "/usr/lib/python2.7/anydbm.py", line 85, in open
worker_1 | return mod.open(file, flag, mode)
worker_1 | File "/usr/lib/python2.7/dbhash.py", line 18, in open
worker_1 | return bsddb.hashopen(file, flag, mode)
worker_1 | File "/usr/lib/python2.7/bsddb/__init__.py", line 364, in hashopen
worker_1 | d.open(file, db.DB_HASH, flags, mode)
worker_1 | DBAccessError: (13, 'Permission denied')
worker_1 | [2017-02-08 05:53:49,127: INFO/MainProcess] Connected to redis://redis:6379/0
worker_1 | [2017-02-08 05:53:49,152: INFO/MainProcess] mingle: searching for neighbors
server_1 | [2017-02-08 05:53:46,291][PID:5][WARNING][redash.query_runner] BigQueryGCE query runner enabled but not supported, not registering. Either disable or install missing dependencies.
server_1 | [2017-02-08 05:53:49,638][PID:5][INFO][werkzeug] * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
worker_1 | [2017-02-08 05:53:49,188: ERROR/Beat] Process Beat
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/lib/python2.7/dist-packages/billiard/process.py", line 292, in _bootstrap
worker_1 | self.run()
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 553, in run
worker_1 | self.service.start(embedded_process=True)
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 470, in start
worker_1 | humanize_seconds(self.scheduler.max_interval))
worker_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 325, in __get__
worker_1 | value = obj.__dict__[self.__name__] = self.__get(obj)
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 512, in scheduler
worker_1 | return self.get_scheduler()
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 507, in get_scheduler
worker_1 | lazy=lazy)
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py", line 53, in instantiate
worker_1 | return symbol_by_name(name)(*args, **kwargs)
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 358, in __init__
worker_1 | Scheduler.__init__(self, *args, **kwargs)
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 185, in __init__
worker_1 | self.setup_schedule()
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 384, in setup_schedule
worker_1 | self._store = self._destroy_open_corrupted_schedule(exc)
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 372, in _destroy_open_corrupted_schedule
worker_1 | return self._open_schedule()
worker_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 366, in _open_schedule
worker_1 | return self.persistence.open(self.schedule_filename, writeback=True)
worker_1 | File "/usr/lib/python2.7/shelve.py", line 243, in open
worker_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
worker_1 | File "/usr/lib/python2.7/shelve.py", line 227, in __init__
worker_1 | Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
worker_1 | File "/usr/lib/python2.7/anydbm.py", line 85, in open
worker_1 | return mod.open(file, flag, mode)
worker_1 | File "/usr/lib/python2.7/dbhash.py", line 18, in open
worker_1 | return bsddb.hashopen(file, flag, mode)
worker_1 | File "/usr/lib/python2.7/bsddb/__init__.py", line 364, in hashopen
worker_1 | d.open(file, db.DB_HASH, flags, mode)
worker_1 | DBAccessError: (13, 'Permission denied')
worker_1 | [2017-02-08 05:53:50,179: INFO/MainProcess] mingle: all alone
server_1 | [2017-02-08 05:53:49,703][PID:5][INFO][werkzeug] * Restarting with stat
server_1 | [2017-02-08 05:53:51,470][PID:133][WARNING][redash.query_runner] BigQueryGCE query runner enabled but not supported, not registering. Either disable or install missing dependencies.
server_1 | [2017-02-08 05:53:52,806][PID:133][WARNING][werkzeug] * Debugger is active!
Has anyone experienced something similar or have suggestions for further diagnosis?