Error running queries after PostgreSQL update

Issue Summary

I’m working on setting up a test environment for redash to get a feel for how to set it up in a fault-tolerant way on AWS when we’re ready to deploy it.

I had it mostly working with the version of PostgreSQL the CentOS repo comes with (9.2). The only problem I had was I was unable to manage users due to Redash using PostgreSQL features only available in newer versions.

So I loaded PostgreSQL 11 onto my DB server and re-setup Redash, at which point I was able to access user maintenance as expected. I’m also able to create data sources and the “test connection” indicates a success.

However, when I run queries, I get an HTTP 500 error. Checking the logs for the redash container yields the following. I can’t for the life of me figure out what’s going on here, nor am I seeing any reason this would suddenly be happening just because I upgraded PostgreSQL

[2019-06-19 15:24:20,304][PID:14][ERROR][redash] Exception on /api/queries/1/results [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 477, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/app/redash/handlers/base.py", line 31, in dispatch_request
    return super(BaseResource, self).dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 587, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/app/redash/permissions.py", line 48, in decorated
    return fn(*args, **kwargs)
  File "/app/redash/handlers/query_results.py", line 208, in post
    return run_query(query.parameterized, parameters, query.data_source, query_id, max_age)
  File "/app/redash/handlers/query_results.py", line 94, in run_query
    "Query ID": query_id
  File "/app/redash/tasks/queries.py", line 115, in enqueue_query
    if job.ready():
  File "/app/redash/tasks/queries.py", line 91, in ready
    return self._async_result.ready()
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 311, in ready
    return self.state in self.backend.READY_STATES
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 680, in _get_task_meta_for
    return self.decode_result(meta)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 284, in decode_result
    return self.meta_from_decoded(self.decode(payload))
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 280, in meta_from_decoded
    meta['result'] = self.exception_to_python(meta['result'])
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 260, in exception_to_python
    exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
TypeError: __init__() takes at least 4 arguments (2 given)

Technical details:

• Redash Version: 7.0.0
• Browser/OS: CentOS 7.4
• How did you install Redash: Used setup.sh script, but tweaked it so PostgreSQL and Redis are not part of Docker (I’m going to be using Amazon RDS and ElastiCache for them).

Seems to be related to Redis somehow, but I’m not sure why.

I started with a fresh Redash install (using the native docker stuff), and got that working. Then I updated that install to point to my existing (11.0) PostgreSQL database, and it still worked.

Then I tried to move to the standalone Redis install, and I started getting the same errors I was getting on the old Redash docker host.

Interesting. Any idea which version of Redis that was with?

I just used what CentOS offered, which looks like 3.2.12.

Cool. Was just checking. Nothing sounds obviously wrong with your setup.

It sounds like an experiment/testing instance, so no real data in it you want to keep (for now).

If that’s right, then I’d probably go with wiping the PostgreSQL database and recreating it, just to see if that make a difference. The “Create Database” step here gives the cut-n-paste-able instructions:

https://redash.io/help/open-source/dev-guide/docker

If nothing changed when that’s done, then it’s not a problem with the PostgreSQL database contents themselves, so other things will need checking (/opt/redash/env variables maybe?).

I started from scratch again and got it working the way I wanted. However, I did run into the issue once or twice, but it seemed to go away and now I can’t duplicate it.

Seemed to be an issue with a specific query I was trying to run, but I’m not sure why there would make a difference. If I’m able to reproduce it again I’ll include the query in question as well as the result set.

Maybe related to this issue?: https://github.com/getredash/redash/issues/3280

I had a similar issue with a query that just would not run with the same errors in the log, something seemed to be wrong with internal metadata. I “fixed” it by making an inconsequential change to the query text (e.g. alter the name of an alias) - this seems to be enough to generate a different query hash internally so Redash treats it as a new query and ignores the existing bad metadata that it’s cached.

That sounds about right, and that explains why it resolved itself when I changed an alias. I was thinking it might be related to a single quote I had in a column alias, so I changed it and the issue resolved. Just for the heck of it I changed it back, and to my surprise it was still working, so I wasn’t sure what to make of it.