Updated migration script

arikfr · July 15, 2021, 5:08am

For a long while we had a script to migrate content between Redash instances. It worked, but wasn’t very much maintained. About a year ago @jesse started the effort of refreshing this script as part of the redash-toolbelt project, it didn’t go far but we’re restarting the effort now.

The goal is to create a maintained version of the script built on top of the toolbelt that will support latest version of Redash along with all the objects (previous versions of the script didn’t migrate alerts or favorites).

The PR is here: Add migration script to examples by susodapop · Pull Request #23 · getredash/redash-toolbelt · GitHub.

If anyone interested in helping with testing, please let us know. We will update on this thread with the progress of this effort.

pruce · July 22, 2021, 9:38am

Do we have an ETA on when the migration script will be usable?

jesse · July 23, 2021, 4:46pm

Late July or early August. You can follow along on the PR Arik linked.

prerna · August 9, 2021, 11:44am

in which database I have to store my dashboards which is available in your database after migrating

jesse · August 9, 2021, 3:03pm

Redash uses Postgres.

prerna · August 10, 2021, 4:15am

ok after migrating where will my dashboards which are in your database will be stored, which database I have to use those dashboards which you have stored ,do I have to use Postgres ,and how do I have to store that can you tell me the steps

prerna · August 10, 2021, 5:09am

hi
can you tell me the dashboards which are stored in your Postgres then how will I have to migrate those dashboards in my Postgres account.
Thanks
Prerna

jesse · August 10, 2021, 3:16pm

All this will be handled by the migration script.

prerna · August 27, 2021, 4:25am

hi how to take backup of the data

jesse · September 1, 2021, 6:54pm

Hey everyone, last call for feedback on the migration script before it’s deployed out to Pypi:

github.com/getredash/redash-toolbelt

[RT-5] Add migration script to examples

getredash:master ← getredash:issue-5

opened 01:39PM - 17 Apr 20 UTC

susodapop

+851 -18

## ⚠️ Help Requested: Beta Testers ⚠️ The migration script in this PR is now… ready for wider testing. Please follow these steps to test it: ### Setup 1. Clone this repository and check out the `issue-5` branch 2. Make a fresh Python virtual environment with Python 3.6 or newer: `python3 -m virtualenv <env name>` 3. Activate your virtual environment: `. <env name>/bin/activate` (macos) or `<env name>\scripts\activate` (windows) 4. `pip install poetry` 5. `poetry build` 6. `pip install dist/redash_toolbelt-0.1.5.tar.gz` ### Usage **Warning**: this script is provided as-is. Make sure you back up your Redash data before attempting to migrate. You can now run the script from within your virtual environment with the command `redash-migrate`. The first time you run `redash-migrate` a file named `meta.json` will be created in your working directory. Before you can proceed, you must provide the address and admin API key for both the origin and destination Redash instances in this file. You will also need to enter a mapping of origin data source id -> destination data source id in `meta.data_sources`. If you already created some users in the destination instance, be sure to enter them into the `meta.users` mapping as well. Throughout `meta.json`, the keys in each top-level object represent the origin id for that object. The JSON spec does not allow the keys of an object to be integers. So you should wrap these id's in quotation marks. The migration script will convert them to integers before proceeding. `redash-migrate` includes several sub-commands: - `check_data_sources` - `users` - `queries` - `visualizations` - `dashboards` - `alerts` - `favorites` - `disable_users` They should be run in the order presented above. As you run each step of the migration, the script will print debug messages to the screen. If you encounter issues with the script, please open an issue against this repository. ### Next Steps Once testing is complete, we will push these changes to Pypi. Typical usage will not require cloning the repo. Users will need only to run `pip install redash-toolbelt`. ## Type of PR - [x] Add a script ## Description This effort will add a revised version of [this gist](https://gist.github.com/arikfr/2c7d09f6837b256c58a3d3ef6a97f61a) that uses the `redash_toolbelt` client, supports recent versions of Redash and migrates all of the content (the gist was skipping alerts and favorites). ## Steps - [x] Convert to Python3 with `2to3` - [x] Refactor to use `client.py` - [x] Add support for migrating alerts - [x] Add support for migrating favorites - [x] Add a "dry run" / "read only" option - [x] Add script entry point to `pyproject.toml` - [ ] Publish documentation / examples ## Related Tickets & Documents #5

electric-al · September 24, 2021, 12:01pm

Hi there all.

We are planning the work required to migrate to our own hosted Redash in Q4.

Is there a guide on how to do it end to end?

jesse · September 24, 2021, 12:02pm

Once OSS V10 and the migration script are released we’ll make docs for this

electric-al · September 29, 2021, 1:00pm

Ok great. Just raising that this leaves about 2 months to migrate, and for larger companies who have trusted hosted redash and paid them, this does not leave a lot of time to do the migration.

Is there an ETA on this?

Arron · September 30, 2021, 2:40pm

Agreed that this doesn’t leave a lot of time to do the migration and test/validate everything.

We will need some contingency time for issues or implementing alternative solutions if it doesn’t work as expected, so we ideally would want to be running the migration script around now.

Is there any talk of offering an extension for those that wish to move to self hosted?

jesse · October 2, 2021, 2:39am

The V10 release is now pending (just waiting for our images to build). The migration script will be published next week. The 30 November deadline is firm. Migration, even for large accounts, should require only a few minutes. We’ve tested the migration script on our internal instance (many thousands of queries and dashboards and alerts) and it takes around 6 six minutes. Most instances will be much quicker than that.

wtfzambo · October 5, 2021, 9:38am

Just a quick question:

will the migration script also be able to export current users, or shall they re-create a new account?

pavlova-marina · October 5, 2021, 1:42pm

Hello!

Will the IDs of queries, widgets, visualizations, and dashboard slugs change after migration? Does the migration script cope with the following:

queries to the source ‘Query Results’ where we use Redash’s own query ids inside the query like ‘select … from cached_query_xxx’;
query based dropdowns?

Will there be available the matching of the old and the new query ids after the migration? (for user information purposes)?

electric-al · October 7, 2021, 4:08pm

Just wondering whats the ETA on the script?

jesse · October 7, 2021, 4:27pm

This week

Also, we as a community, need a better way to communicate upcoming timelines. I think it’s frustrating for everyone.

jesse · October 7, 2021, 4:28pm

This is possible since meta.json contains a map of $originQueryId to $destinationQueryId. It will not be part of the release this week. But we will aim to add it within the next few weeks (or would merge a contribution that implements it sooner).

[Edit] I added an issue to redash-toolbelt to track this: Migration Script: convert origin QRDS queries to destination · Issue #64 · getredash/redash-toolbelt · GitHub

▼Categories

▼Tags

Updated migration script