1

Issue Summary

We’re using self hosted redash legacy setup. We upgraded it to version 7 couple of months back.

Right now, I’m trying to move the backend to dedicated rds.

I stopped all the services, took a pgdump. Then I loaded the pgdump to rds pgsql instance.

Then I created a new redash instance (for POC) using docker based ami. Updated the env details to point to the new RDS, started the services. I am able to login to the redash but can’t see the data sources.

Upon checking the docker logs, I see that ‘InvalidToken’ error appears for accessing the data sources.

[2019-07-16 12:40:51,896][PID:14][ERROR][redash] Exception on /api/data_sources [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 477, in wrapper
resp = resource(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
  File "/app/redash/handlers/base.py", line 31, in dispatch_request
return super(BaseResource, self).dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 587, in dispatch_request
resp = meth(*args, **kwargs)
  File "/app/redash/permissions.py", line 48, in decorated
return fn(*args, **kwargs)
  File "/app/redash/handlers/data_sources.py", line 88, in get
for ds in data_sources:
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 98, in instances
util.raise_from_cause(err)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 79, in instances
rows = [proc(row) for row in fetch]
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 511, in _instance
loaded_instance, populate_existing, populators)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 611, in _populate_full
dict_[key] = getter(row)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/type_api.py", line 1226, in process
return process_value(impl_processor(value), dialect)
  File "/app/redash/models/types.py", line 28, in process_result_value
return ConfigurationContainer.from_json(super(EncryptedConfiguration, self).process_result_value(value, dialect))
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 409, in process_result_value
decrypted_value = self.engine.decrypt(value)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 216, in decrypt
decrypted = self.fernet.decrypt(value)
  File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 103, in decrypt
raise InvalidToken
InvalidToken

I went through the forum, github issues and understood that this is due to data sources configuration encryption in the redash version 7.

REDASH_COOKIE_SECRET is already configured in the existing instance. So I tried using the same
key details in the new instance, restarted the container, but still getting the same invalid token error.

I’ve spent time in redash forums from morning to an extent that I’ve been promoted to ‘trust level’ :smiley:

Most of the relevant issue topics are pre 7x version. The closest topic was Upgrading to V7, Invalid token error and checked the Unable to connect to data sources after upgrading to version v7.0.0.b17489 · Issue #3617 · getredash/redash · GitHub, but we already set the COOKIE_SECRET in the production instance and tried with the same in POC instance, still no luck. Looking for your inputs.

Further posts I referred for this migration:

Technical details:

  • Redash Version: 7
  • Browser/OS: firefox
  • How did you install Redash: docker ami
2

Just to make sure:

  1. Are you sure the previous instance didn’t use the dedicated setting? (REDASH_SECRET_KEY)
  2. Is the previous instance currently working?
  3. How did you restart?
3

Hi Arik,

Thank you for your prompt response.
1. Are you sure the previous instance didn’t use the dedicated setting? (REDASH_SECRET_KEY)
Yes, REDASH_SECRET_KEY is not used. The setup refers to .env file under /opt/redash/.env and only REDASH_COOKIE_SECRET is configured. Even the .env file under /opt/redash/redash.7.0.0.b17535 is symlinked to /opt/redash/.env, so I’m sure REDASH_SECRET_KEY is not used. Is there any other configs do I have to look into or any way to find whether the variable is used?

2. Is the previous instance currently working?
Yes. I didn’t make any changes in the instance. I stopped the services using ‘supervisorctl stop all’, took pgdump, then started the services. It is working as expected.

3. How did you restart?
After env file update, I restarted using ‘sudo docker-compose up -d’ .
Also I tried various options like ‘sudo docker-compose restart -t 1’, ‘sudo docker-compose down’ followed by ‘sudo docker-compose up -d’.

4

Also to add:
In the docker-compose.yml, the env_file is passed at the top,

version: '2'
x-redash-service: &redash-service
  image: redash/redash:latest
  depends_on:
          #    - postgres
    - redis
  env_file: /opt/redash/env
  restart: always
services:

Seeing your comment in one of the github issues, I just added REDASH_COOKIE_SECRET: “xxxxx”’ under environment section of each services, also added env_file: /opt/redash/env under redis section. Still no luck.

5

It seems like everything you’re doing is correct. Are you sure it uses the database you restored?

The old instances used a dedicated DB named redash, while the Docker ones just use postgres. Maybe the db was restored into redash while the code looks at postgres?

Nevermind: just noticed that you mentioned you’re using RDS :thinking:

6

Hi Arik,

If I understand correctly, the data sources configuration are encrypted and not able to view it due to the invalid token error. This encryption will impact only the existing data sources. So wondering can I manually recreate the data sources exist in the production redash, here? If I do that, it should work seamlessly in POC instance right? I mean, will the invalid token affect anything else?

7

Ok, I created a new data source. But that data source also not visible after creation, because redash tries to access the data sources page again and getting internal server error.

[2019-07-17 07:06:52,121][PID:14][ERROR][redash] Exception on /api/data_sources [GET]
..
..
 File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 103, in decrypt
    raise InvalidToken
8

You will need to delete the old data sources for things to work. But weren’t they used for anything already?