1

Issue Summary

We recently successfully spin up the self-hosted redash through ECS fargate, but the scheduler seems not stable all the time. We have a couple of scheduled queries and none of them get executed ever since yesterday, and some of them are marked as “outdated query” in the system status dashboard. I couldn’t find any documentation explainning what is outdated query, besides, I walked through all the logs and nothing really helpful, not even a single error/exception found in logs. Does anyone have similiar issue?

Screenshots

  1. Queues

    Screen Shot 2021-02-09 at 9.03.57 AM
    Screen Shot 2021-02-09 at 9.03.57 AM1845×592 26.9 KB

  2. Workers

    Screen Shot 2021-02-09 at 9.03.43 AM
    Screen Shot 2021-02-09 at 9.03.43 AM1847×593 71.1 KB

  3. Queries

    Screen Shot 2021-02-09 at 9.02.10 AM
    Screen Shot 2021-02-09 at 9.02.10 AM1835×589 37.8 KB

  4. Outdated Query

    Screen Shot 2021-02-09 at 9.02.34 AM
    Screen Shot 2021-02-09 at 9.02.34 AM1843×636 85.1 KB

Technical details:

  • Redash Version: 9
  • Browser/OS: Chrome
  • How did you install Redash: self-hosted ECS fargate
2 Likes
2

I have the same issue for adhoc queries. Trying to figure out the issue.

3

Same on my side (self install on Kubernetes V9 preview)

the preriodic are not updated; The schemas seem impacted also as not available (is that linked to Redash V9 cant refresh schema - #6 by xavier-d ?)

image

4

What happens if you manually run the query?

5

@k4s1m The query can be executed normally if I manually run it. Do you have any insights? : )

6

Try restarting the scheduler container to see if that fixes it.

7

I actually first flushed the redis and restart the worker and scheduler, then it started to work.
But I think restarting the scheduler did the trick as you said.

I’m wondering how did you come up with this idea? Why thoses scheduled queries never return results and remain outdated? Is that something related to time limit?

Thanks!

8

Might be duplicated: Worker shutdowns in v9 resulting in stuck queries

9

Also faced this issue when running the docker image - redash/redash:9.0.0-beta.b42121 (this image was built in June, 2020). Also restared scheduler but it works for a day and then same problem again.

After trying to find if discussion on this problem, ran into the below github issues and to this ghost locks.
The issues are,
#4926 Query doesn’t not get queued because redash thinks it is already queued
#5087 Add: periodic job to remove ghost locks.
Also noticed that the ghost locks was fixed in August, 2020 via the last issue above (issue 5087).

Also,
@Albert mentions this issue might be duplicate of Worker shutdowns in v9 resulting in stuck queries.
But a pull request was already merged in Nov. 2020 it seems.
So if you are running the latest version then it may not be any of the above problems… I guess…

Btw, I just noticed docker image redash/redash:preview was pushed 10days ago - which will have the latest.
Now, I want to try running the preview docker image, but, @xavier-d already said he is running preview image since more than 10days ago.
So I am kind of confused about the preview tag…
Maybe, is the preview image being overwritten by pushing new image every few days… :thinking: ?
(also this image I checked has rq==1.5.0 maybe it won’t have the stuck query issues…)

I want to check the redash version that is more closer to the upcoming release of v9.
So, I need to be checking the redash/redash:preview right? not the old image redash/redash:9.0.0-beta.b42121 tag…

Sorry about my confusions and doubts… new to redash, please execuse

Thanks so much!

10

Indeed the image preview is overwritten with the lastest build; The one you are seeing right now (10 days ago) has for digest 0fca9eeb. I’m using it since it has been pushed (I’ve used the previous ones also).
I don’t have any issue with it on my Kubernetes infra for the refresh of the schema (that was my main concern)

My first installs were failing due to a pull request I did not fetched on my Helm file (see here: Feedback on Kubernetes self hosted Install of preview version - #3 by xavier-d)

But as you, the auto refresh of the queries seems not working. I did not had time to dig for now

image
image933×34 3.47 KB

But I’m not facing same issue as you; The outdated page does not report anything

image

1 Like