Debugging high load celery worker

Issue Summary

The celery worker process is at 100% cpu load all the time. Might be related to Celery worker stuck with 100% CPU.

I am mainly interested in how to get more information about the problem rather than finding a fix.

I checked the syslog, but at least the current output does not show what the worker is occupied with. Does redash provide a utility for debugging/monitoring? I read up a bit on celery worker introspection, but the inspect recipe I found did not work out of the box (see below). Should I look more into celery/redis? Any help in getting more info about what is going on is appreciated.

Cheers,
Johannes

>>> from celery.app.control import Inspect
>>> i = Inspect()
>>> i.active()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/celery/app/control.py", line 108, in active
    return self._request('active')
  File "/usr/local/lib/python2.7/dist-packages/celery/app/control.py", line 88, in _request
    return self._prepare(self.app.control.broadcast(
AttributeError: 'NoneType' object has no attribute 'control'

Technical details:

  • Redash Version: 7
  • Browser/OS: Ubuntu 18.04
  • How did you install Redash: WIth celery, and redis.
1 Like

We ran into similar issues with Celery on our production server which is why we ultimately replaced Celery with RQ.

Related thread on the forum:

And a patch was developed for similar issues on Celery here: