We’ve been noticing recently that there are several queries that are stuck “saving results”. It seems it’s isolated to adhoc queries (unsaved) but I don’t have more evidence.
Has anyone seen this before? I don’t see much documentation on the states of the query either.
Redash v3.0.0+b3134 running on EC2, Each celery worker is limited to either 100MB or 1GB depending on queue but this happens on both.
Edit: More details
It appears that UI shows all the results but these still show up as executing. I’ve done the following:
- stopped all processes using supervisorctl
- killed all celery workers
- redis-cli flushall
- start all processes
After this the above queries that are executing return in the queue but are not actually running. the Logs for redis, celery and api are not showing any errors. Disk is 2% used memory is under 30% and CPU is in under 10%.
This seems to be causing workers to get stuck and not be recycled which causes new queries to be stuck in queue