How to change REDASH_SECRET_KEY?

shinsuke-nara · September 10, 2019, 9:13am

Hi.

Is there a feature to re-encrypt data source options with new REDASH_SECRET_KEY?

If REDASH_SECRET_KEY is leaked, I want to change REDASH_SECRET_KEY and encrypt data source options with the new REDASH_SECRET_KEY.

I checked documents and source code but I can’t find about it.

Could you help me?
Thanks.

jesse · September 23, 2019, 2:33am

Welcome to the forum. I think one of these could work. Definitely back up your Redash metadata before attempting either of these.

Use the API

Decrypt and extract the data source info from the current database. Also pull info on which queries use those sources. Drop the data sources. Kill Redash.
Change the REDASH_SECRET_KEY located in redash.settings.DATA_SOURCE_SECRET_KEY and restart Redash.
Create new data sources with the API and the decrypted connection information
Cycle through the queries from step 1 and update each data_source_id.

Alternatively:

We use sqlalchemy_utils’s EncryptedType to implement the encryption.

From their documentation:

The key parameter accepts a callable to allow for the key to change per-row instead of being fixed for the whole table.

This suggests you can use more than one SECRET_KEY on the same table, as long as SQLAlchemy has access to both secret keys. It follows that you could write a database migration to re-encrypt the data source connection information in place.

But in my opinion, the API pattern would be easier.

shinsuke-nara · September 25, 2019, 10:01am

Thank you for your reply.

I wrote a script to reencrypt data source options with sqlalchemy_utils.
The script is https://github.com/shinsuke-nara/redash/blob/change-secret-script/change-secret.py
I think this will help other redash users.

I’m glad to make PR if you can accept this.
How do you think about my suggestion?

arikfr · September 25, 2019, 11:29am

Thanks, @shinsuke-nara! This is a clever use of SQLAlchemy You’re welcome to make a PR and add this as a new CLI command.

There is one more way, which has some benefits as it allows doing these changes without downtime. The downside is that it requires a bit more effort, but here’s the details in case someone wants to take a stab:

We’re using sqlalchemy-utils's FernetEngine, which in turn uses Fernet from cryptography.
cryptography has another implementation: MultiFernet. It works the same way as Fernet, except that it can take multiple keys (and try to decrypt with all of them) along with a rotate method which will re-encrypt with the new key.
FernetEngine is quite simple and we can create a version of it that uses MultiFernet instead and accepts multiple keys. Just need to figure out how to expose the rotate method.

shinsuke-nara · September 27, 2019, 5:20am

Thank you for your reply.

I made a PR for this: https://github.com/getredash/redash/pull/4190

Your suggestion which is runtime reencryption is interesting.
I want to make an issue in github if you can allow it.

arnav0024 · October 27, 2021, 6:34pm

Just wanted to understand How shall someone get REDASH_SECRET_KEY not able to grep from enviornment variables of redash containers. I need REDASH_SECRET_KEY since within the script it requires both new and old secret keys

jesse · October 27, 2021, 6:54pm

How did you deploy Redash? These won’t be set by the containers but by the host (the machine that’s running the containers)

arnav0024 · October 27, 2021, 7:03pm

So basically we have recently decoupled our redash architecture under ECS flow just gone through the code and understood by default we use REDASH_COOKIE_SECRET as the secret key. So I ran the following command exec /app/manage.py database reencrypt <oldkey> <newkey>. And currently I am getting following error. Not sure what exactly wrong I am doing here.

  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 481, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/manage.py", line 9, in <module>
    manager()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/app/redash/cli/database.py", line 124, in reencrypt
    _reencrypt_for_table("data_sources", "DataSource")
  File "/app/redash/cli/database.py", line 114, in _reencrypt_for_table
    selected_items = db.session.execute(select([table_for_select]))
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1268, in execute
    return self._connection_for_bind(bind, close_with_result=True).execute(
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1130, in _connection_for_bind
    engine, execution_options
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 431, in _connection_for_bind
    conn = bind._contextual_connect()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2239, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1544, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 481, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

(Background on this error at: http://sqlalche.me/e/e3q8)```

jesse · October 27, 2021, 7:29pm

In what context did you execute this code? Within a container?

arnav0024 · October 27, 2021, 7:33pm

Context : We want to re-encrypt our postgres data due to some security concerns.
By within a container here I mean by running following command within redash server as well as worker container which already has connection with postgres and redis. Which is failing in both the cases.

jesse · October 28, 2021, 11:32am

There’s your problem. This command needs to be run in the context of the entire docker-compose “cluster” (not sure that’s the right word for the collection of containers). You only need to run it once.

The way you invoke it is like this on the machine that is running docker-compose:

$ docker-compose run --rm server manage database reencrypt old_secret new_secret

Here are the components of the command:

docker-compose run --rm server means to use server service as the execution context.
manage means to execute the manage command defined in the docker entrypoint
reencrypt *args is passed directly to the CLI which now has appropriate context to make the changes.