Upgrade from 3.0.0+b3134 to v5


#1

Hello, trying to upgrade with sudo bin/upgrade but get error on “Running migrations (if needed)”

Running migrations (if needed)…
Failed running: sudo -u redash bin/run ./manage.py db upgrade
Exit status: 1
Output:
[2018-09-24 21:15:34,076][PID:4370][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2018-09-24 21:15:34,076][PID:4370][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2018-09-24 21:15:34,093][PID:4370][INFO][alembic.runtime.migration] Running upgrade d1eae8b9893e -> 7671dca4e604, empty message
[2018-09-24 21:15:34,119][PID:4370][INFO][alembic.runtime.migration] Running upgrade 7671dca4e604 -> 5ec5c84ba61e, Add Query.search_vector field for full text search.
[2018-09-24 21:15:34,374][PID:4370][INFO][alembic.runtime.migration] Running upgrade 5ec5c84ba61e -> 6b5be7e0a0ef, Re-index Query.search_vector with existing queries.
[2018-09-24 21:15:34,600][PID:4370][INFO][alembic.runtime.migration] Running upgrade 6b5be7e0a0ef -> 969126bd800f, Update widget’s position data based on dashboard layout.
Updating dashboards position data:
Traceback (most recent call last):
File “/opt/redash/redash.5.0.0.b4755/manage.py”, line 9, in
manager()
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)


#2
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column dashboards.tags does not exist
LINE 1: ...ived, dashboards.is_draft AS dashboards_is_draft, dashboards...
                                                             ^
 [SQL: 'SELECT dashboards.updated_at AS dashboards_updated_at, dashboards.created_at AS dashboards_created_at, dashboards.id AS dashboards_id, dashboards.version AS dashboards_version, dashboards.org_id AS dashboards_org_id, dashboards.slug AS dashboards_slug, dashboards.name AS dashboards_name, dashboards.user_id AS dashboards_user_id, dashboards.layout AS dashboards_layout, dashboards.dashboard_filters_enabled AS dashboards_dashboard_filters_enabled, dashboards.is_archived AS dashboards_is_archived, dashboards.is_draft AS dashboards_is_draft, dashboards.tags AS dashboards_tags \nFROM dashboards'] (Background on this error at: http://sqlalche.me/e/f405)

the table does not contain dashboards.tags column at all

#3

This happens because the migration code uses the dashboard model code (redash.models.Dashboard) which assumes some new column exists (that gets created in another migration). I don’t think I’ll have the time to fix this migration :disappointed: You can manually alter it not use redash.models.Dashboard but rather do the update using SQL.

You can see an example in the way we update tags in dashboards in another migration.


#4

I added tags column to db with

sudo -u redash psql -c “alter table dashboards add column tags varchar;” redash

then tried to upgrade again and following migrations were successful
upgrade d1eae8b9893e -> 7671dca4e604 empty message
upgrade 7671dca4e604 -> 5ec5c84ba61e Add Query.search_vector field for full text search.
upgrade 5ec5c84ba61e -> 6b5be7e0a0ef Re-index Query.search_vector with existing queries.
upgrade 6b5be7e0a0ef -> 969126bd800f Update widget’s position data based on dashboard layout.
upgrade 969126bd800f -> 1daa601d3ae5 add columns for disabled users
upgrade 1daa601d3ae5 -> d4c798575877 empty message
upgrade d4c798575877 -> e7004224f284 add_org_id_to_favorites

but this one
upgrade e7004224f284 -> a92d92aa678e inline_tags

created an error
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column “tags” of relation “dashboards” already exists
[SQL: ‘ALTER TABLE dashboards ADD COLUMN tags VARCHAR[]’] (Background on this error at: http://sqlalche.me/e/f405)

and these migrations are still left to do
0f740a081d20_inline_tags.py Change migration to be safe to run along with other migrations
65fc9ede4746_add_is_draft_status_to_queries_and_.py Modernize Python 2 code to get ready for Python 3
71477dadd6ef_favorites_unique_constraint.py Handle duplicate favorite calls.

any hints how to overcome this situation


#5

Of course it will create an error: it tries to create the same column you already created. I think that if you comment out the line that creates the tags column on dashboards, it will actually work.


#6

just to be extra sure,

  1. I will comment out that line in 0f740a081d20_inline_tags.py
  2. run db upgrade with manage.py in folder redash.5.0.0.b4755
  3. change current -> /opt/redash/redash.5.0.0.b4755

and we are there


#7

managed to get migrations done and new version is running

the file that needed modifications was a92d92aa678e_inline_tags.py


#8

What is the long term fix for this? Should we modify a92d92aa678e_inline_tags.py?

I can try to help with a Pull Request once I understand what to do.

This also seems related to https://discuss.redash.io/t/solved-how-to-upgrade-from-3-0-x-to-5-0-x/2426 .


#9

Thanks!

We need to modify the update widget positions migration (969126bd800f) to not use models, but rather run SQL queries directly.


#10

I feel like I am not very proficient on sqlalchemy/alembic. I can try a bit more. Is there some example of using SQL queries directly that I can use? I was looking at models.py, which have some db.session.query(), but db seems to be a instance of flask_sqlalchemy.SQLAlchemy, can we do that in migrations?


#11

If anyone else run across this issue: my solution was to upgrade to v4 before upgrading to v5.
Since there is no easy way to do this, you need to modify bin/upgrade script, at line 97 change:

        release = response.json()[0]

to:

    release = response.json()[3] 

At this moment, latest v4 is at [3], but it might be a higher number in the future.
After v4 is installed, just run bin/upgrade again and everything is up and running with latest v5.


#12

I managed to upgrade using the instructions above, thanks!

  1. sudo -u redash psql -c “alter table dashboards add column tags varchar;” redash
  2. comment out first line that creates column “tags”, in method upgrade() in /opt/redash/redash.5.0.2.b5485/migrations/versions/a92d92aa678e_inline_tags.py