[Temporarily Fixed] Upgrade to Redash 9 failed with migrations and Alert Destinations are throwing internal server error

m_abdelfattah · February 3, 2021, 3:38pm

Issue Summary

Scroll to the end of this post to see the temp fix
I just upgraded to Redash 9.0 beta (redash:9.0.0-beta.b42121) and redash itself is working. But when I try to go to “Alert Destinations” I get " Internal Server Error". When I checked the logs, I found that the migration did not create notification_destinations, I tried to run managed db migrate and no errros (the log is available below)

Technical details:

Redash Version: 9.0.0-beta
Browser/OS:
How did you install Redash: self-hosted (docker-compose)

Here is the docker-compose logs server

server_1            | [2021-02-03 15:23:54,396][PID:17][ERROR][redash.app] Exception on /api/destinations [GET]
server_1            | Traceback (most recent call last):
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
server_1            |     cursor, statement, parameters, context
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
server_1            |     cursor.execute(statement, parameters)
server_1            | psycopg2.errors.UndefinedColumn: column notification_destinations.options does not exist
server_1            | LINE 1: ...tinations.type AS notification_destinations_type, notificati...
server_1            |                                                              ^
server_1            |
server_1            |
server_1            | The above exception was the direct cause of the following exception:
server_1            |
server_1            | Traceback (most recent call last):
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
server_1            |     rv = self.dispatch_request()
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
server_1            |     return self.view_functions[rule.endpoint](**req.view_args)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 458, in wrapper
server_1            |     resp = resource(*args, **kwargs)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 261, in decorated_view
server_1            |     return func(*args, **kwargs)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask/views.py", line 89, in view
server_1            |     return self.dispatch_request(*args, **kwargs)
server_1            |   File "/app/redash/handlers/base.py", line 33, in dispatch_request
server_1            |     return super(BaseResource, self).dispatch_request(*args, **kwargs)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
server_1            |     resp = meth(*args, **kwargs)
server_1            |   File "/app/redash/handlers/destinations.py", line 93, in get
server_1            |     for ds in destinations:
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3342, in __iter__
server_1            |     return self._execute_and_instances(context)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3367, in _execute_and_instances
server_1            |     result = conn.execute(querycontext.statement, self._params)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
server_1            |     return meth(self, multiparams, params)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
server_1            |     return connection._execute_clauseelement(self, multiparams, params)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
server_1            |     distilled_params,
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
server_1            |     e, statement, parameters, cursor, context
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
server_1            |     util.raise_from_cause(sqlalchemy_exception, exc_info)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
server_1            |     reraise(type(exception), exception, tb=exc_tb, cause=cause)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
server_1            |     raise value.with_traceback(tb)
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
server_1            |     cursor, statement, parameters, context
server_1            |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
server_1            |     cursor.execute(statement, parameters)
server_1            | sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column notification_destinations.options does not exist
server_1            | LINE 1: ...tinations.type AS notification_destinations_type, notificati...
server_1            |                                                              ^
server_1            |
server_1            | [SQL: SELECT notification_destinations.id AS notification_destinations_id, notification_destinations.org_id AS notification_destinations_org_id, notification_destinations.user_id AS notification_destinations_user_id, notification_destinations.name AS notification_destinations_name, notification_destinations.type AS notification_destinations_type, notification_destinations.options AS notification_destinations_options, notification_destinations.created_at AS notification_destinations_created_at
server_1            | FROM notification_destinations
server_1            | WHERE %(param_1)s = notification_destinations.org_id ORDER BY notification_destinations.id ASC]
server_1            | [parameters: {'param_1': 1}]
server_1            | (Background on this error at: http://sqlalche.me/e/f405)
server_1            | [2021-02-03 15:23:54,398][PID:17][INFO][metrics] method=GET path=/api/destinations endpoint=destinations status=500 content_type=application/json content_length=36 duration=5.14 query_count=2 query_duration=3.30

And here is the output for docker-compose run --rm server manage db migrate

/opt/redash# docker-compose run --rm server manage db migrate
Starting redash_redis_1 ... done
Starting redash_postgres_1 ... done
[2021-02-03 15:35:49,117][PID:1][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2021-02-03 15:35:49,122][PID:1][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2021-02-03 15:35:49,194][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'queries_id_seq' as owned by integer column 'queries(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,335][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'query_snippets_id_seq' as owned by integer column 'query_snippets(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,359][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'alert_subscriptions_id_seq' as owned by integer column 'alert_subscriptions(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,417][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'events_id_seq' as owned by integer column 'events(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,439][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'data_source_groups_id_seq' as owned by integer column 'data_source_groups(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,479][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'widgets_id_seq' as owned by integer column 'widgets(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,542][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'changes_id_seq' as owned by integer column 'changes(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,564][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'access_permissions_id_seq' as owned by integer column 'access_permissions(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,586][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'favorites_id_seq' as owned by integer column 'favorites(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,610][PID:1][INFO][alembic.ddl.postgresql] Detected sequence named 'api_keys_id_seq' as owned by integer column 'api_keys(id)', assuming SERIAL and omitting
[2021-02-03 15:35:49,669][PID:1][INFO][alembic.autogenerate.compare] Detected removed column 'dashboards.options'
[2021-02-03 15:35:49,741][PID:1][INFO][alembic.autogenerate.compare] Detected added column 'notification_destinations.options'
[2021-02-03 15:35:49,758][PID:1][INFO][alembic.autogenerate.compare] Detected removed column 'notification_destinations.encrypted_options'
  Generating /app/migrations/versions/29dc288308f8_.py ...  done

And here is the output of manage db upgrade

redash@71308b676653:/app$ ./manage.py db upgrade
[2021-02-03 17:10:27,161][PID:340][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2021-02-03 17:10:27,161][PID:340][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2021-02-03 17:10:27,173][PID:340][INFO][alembic.runtime.migration] Running upgrade e5c7a4e2df4d -> 8ac7524f6193, empty message
Traceback (most recent call last):
  File "./manage.py", line 9, in <module>
    manager()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_migrate/cli.py", line 134, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.7/site-packages/flask_migrate/__init__.py", line 95, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_migrate/__init__.py", line 280, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "migrations/env.py", line 87, in <module>
    run_migrations_online()
  File "migrations/env.py", line 80, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 520, in run_migrations
    step.migration_fn(**kw)
  File "/app/migrations/versions/8ac7524f6193_.py", line 22, in upgrade
    op.add_column('notification_destinations', sa.Column('options', redash.models.types.Configuration(), nullable=False))
NameError: name 'redash' is not defined

Temp Fix

I did edit the migration file that has the issue (/app/migrations/versions/8ac7524f6193_.py), and added import redash to it.

Then, tried to re-run the manage db upgrade, and got the following error ```

sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) column "options" contains null values 
[SQL: ALTER TABLE notification_destinations ADD COLUMN options TEXT NOT NULL]

I connected to redash’s postgresql and ran the SQL command manually without the NOT NULL (because I already had some rows in notification_destinations
```
ALTER TABLE notification_destinations ADD COLUMN options TEXT
```
Now, I can access my “Alert Destinations” and edit the destinations. But unfortunately, I had to re-add all the configurations for the destinations. As, the previous configurations were in encrypted_options column, and redash now is using options column.

▼Categories

▼Tags

[Temporarily Fixed] Upgrade to Redash 9 failed with migrations and Alert Destinations are throwing internal server error

Issue Summary

Technical details:

Temp Fix