Problems with DB migration when upgrading to 7.0

#1

I’m having problems updating Redash 6.0.0.b8537 to 7.0.0.b18042.

I’m hosting it on Kubernetes (Google Kubernetes Engine) and using Google Cloud SQL as a backend. Due to Kubernetes, the normal update instructions won’t work because docker-compose is not used.

Updating Redash itself works, but I can’t get db migrate to work.

I tried to recreate update steps in the following way:

  1. Snapshot Cloud SQL for backup
  2. Delete Redash deployment (I chose to delete whole deployment because trying to kill pods/containers/processes inside turned out very tricky/impossible). At this point, Redash services are practically turned off and DB is still running.
  3. Run Kubernetes job to migrate DB. This will launch Redash container (version 7.0) and only run python manage.py db upgrade -command and exits after that.

Migrate script gives following output before completing:

[2019-04-02 13:33:29,826][PID:6][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-04-02 13:33:29,912][PID:6][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2019-04-02 13:33:41,363][PID:6][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2019-04-02 13:33:41,365][PID:6][INFO][alembic.runtime.migration] Will assume transactional DDL.
  1. Re-deploy Redash services using the 7.0 version

Trying to access web interface after upgrade results in Internal Server Error.

Logs are referring to error in DB schema:

[2019-04-02 13:36:31,510][PID:9][ERROR][redash] Exception on /favicon.ico [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1636, in full_dispatch_request
    request_started.send(self)
  File "/usr/local/lib/python2.7/dist-packages/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/app/redash/models/users.py", line 54, in update_user_active_at
    if current_user.is_authenticated and not current_user.is_api_user():
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 343, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 302, in _get_current_object
    return self.__local()
  File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 26, in <lambda>
    current_user = LocalProxy(lambda: _get_user())
  File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 302, in _get_user
    current_app.login_manager._load_user()
  File "/usr/local/lib/python2.7/dist-packages/flask_login/login_manager.py", line 317, in _load_user
    return self.reload_user()
  File "/usr/local/lib/python2.7/dist-packages/flask_login/login_manager.py", line 279, in reload_user
    user = self.user_callback(user_id)
  File "/app/redash/authentication/__init__.py", line 63, in load_user
    user = models.User.get_by_id_and_org(user_id, org)
  File "/app/redash/models/mixins.py", line 28, in get_by_id_and_org
    return query.one()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2954, in one
    ret = self.one_or_none()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2924, in one_or_none
    ret = list(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2995, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
ProgrammingError: (psycopg2.ProgrammingError) column users.details does not exist
LINE 1: ..._api_key, users.disabled_at AS users_disabled_at, users.deta...
                                                             ^
HINT:  Perhaps you meant to reference the column "users.email".
 [SQL: 'SELECT users.profile_image_url AS users_profile_image_url, users.groups AS users_groups, users.updated_at AS users_updated_at, users.created_at AS users_created_at, users.id AS users_id, users.org_id AS users_org_id, users.name AS users_name, users.email AS users_email, users.password_hash AS users_password_hash, users.api_key AS users_api_key, users.disabled_at AS users_disabled_at, users.details AS users_details \nFROM users \nWHERE users.id = %(id_1)s AND %(param_1)s = users.org_id'] [parameters: {'id_1': u'1', 'param_1': 1}] (Background on this error at: http://sqlalche.me/e/f405)

To me it looks like DB migrate doesn’t run as it should. I’ve also tried to run it directly from deployed Redash server without shutting down the deployment, but it still gives the same output.

0 Likes

#2

Having the same issue here. Were you able to find a solution?

0 Likes

#3

Not yet. I’ve seen some other people having problems with migration script too when using Kubernetes.

0 Likes

#4

Based on the outputs you shared, it looks like no migration were applied when you ran db upgrade. Please make sure that:

  1. You executed the migrations step with the correct image.
  2. You executed the migrations step with the correct ENV vars (pointing to the same DB).
0 Likes

#5

Weird, I got it working now, but last time I tried it I got following error:

[2019-04-02 13:32:14,557][PID:7][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-04-02 13:32:14,603][PID:7][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2019-04-02 13:32:21,102][PID:7][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2019-04-02 13:32:21,103][PID:7][INFO][alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    manager()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 345, in main
    return AppGroup.main(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 229, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_migrate/cli.py", line 132, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python2.7/dist-packages/flask_migrate/__init__.py", line 239, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 254, in upgrade
    script.run_env()
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 427, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/local/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python2.7/dist-packages/alembic/util/compat.py", line 198, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "migrations/env.py", line 87, in <module>
    run_migrations_online()
  File "migrations/env.py", line 80, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python2.7/dist-packages/alembic/runtime/environment.py", line 836, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python2.7/dist-packages/alembic/runtime/migration.py", line 321, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 243, in upgrade
    return script._upgrade_revs(revision, rev)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 340, in _upgrade_revs
    for script in reversed(list(revs))
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 174, in _catch_revision_errors
    compat.raise_from_cause(util.CommandError(resolution))
  File "/usr/local/lib/python2.7/dist-packages/alembic/util/compat.py", line 262, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 143, in _catch_revision_errors
    yield
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 336, in _upgrade_revs
    revs = list(revs)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/revision.py", line 645, in _iterate_revisions
    requested_lowers = self.get_revisions(lower)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/revision.py", line 299, in get_revisions
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/revision.py", line 304, in get_revisions
    for rev_id in resolved_id)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/revision.py", line 304, in <genexpr>
    for rev_id in resolved_id)
  File "/usr/local/lib/python2.7/dist-packages/alembic/script/revision.py", line 362, in _revision_for_ident
    resolved_id)
alembic.util.exc.CommandError: Can't locate revision identified by 'e5c7a4e2df4d'

Nothing really was different this time :thinking: perhaps there was some kind of conflict/error somewhere in deployment with version tags that was accidentally fixed.

Anyway, it works now. Here is a recap how version tags were configured to get it working if somebody else faces the same problem.

  • In step 2, deleted Redash deployment was version 6.0. I only deleted Kubernetes deployment for Redash. I left all other object (secrets, volumes, DB-proxy, Redis…) intact.
  • In step 3, Kubernetes job to upgrade DB used version 7.0 image.
  • In step 4, Redash deployment was updated to use version 7.0.
0 Likes