Celery process often grab CPU core, and eventually hosted machine get 100% CPU utils.
I will try to tune the parameters related celery worker.
- Docker Image: redash:6.0.0.b8537
- OS: Ubuntu 16.04 LTS
- 2core, 8GB
- current env variable at worker container
REDASH_REDIS_URL=redis://xxx PYTHONUNBUFFERED=0 QUEUES=queries,scheduled_queries,celery CELERY_WORKER_PREFETCH_MULTIPLIER=0 SHLVL=0 HOME=/home/redash WORKERS_COUNT=2 REDASH_DATABASE_URL=postgresql://xxx
I intend to tune worker_max_tasks_per_child, but there is default value.
In case of CPU hang at Celery, worker_max_tasks_per_child is useful ?
I’m recognizing worker_max_tasks_per_child as memory-bound. But it seems be impossible to tune worker_max_tasks_per_child in above the code.
First of all, it may be heavy query that is problem causing CPU hang.
And, I know the roadmap of Celery migration to RQ.
Now, I just want to know how tuning the parameters is possible.
waits for any advice.