Bring back bootstrap.sh


#1

First let me say that I love Redash. It’s brilliant.

I was a little disappointed that the new version 5 seems to no longer have an easy way to install redash on your own machine without using a docker (https://redash.io/help/open-source/setup). Im sure there are other who dont want to use dockers for some or other reason. The new Setup Script creates a docker installation.

Lucky for me I still had a copy of the old bootstrap.sh file used in v4. I used this and then did a upgrade from v4 to v5.

Could you please give us an updated script to install redash 5 without using dockers.

Alternatively give us some clearer instructions than the ones in https://redash.io/help/open-source/dev-guide/setup. When I read it I had too many questions (think step by step installation guide for noobs). Again I was glad for the old bootstrap.sh and it will probably be easier to bring an updated version of this back?

Thanks again for a great project.


#2

:heart_eyes:

Any reason you prefer not to use Docker?

We deprecated the old setup in favor of the new one for the following reasons:

  1. It’s more robust: now whenever we ship an upgrade it comes with all the needed dependencies. So no more cases where you upgrade your instance and some query runners no longer work. Also, it will allow us to do bigger changes (like switch to Python 3 from Legacy Python) with ease, as it’s transparent to the user.
  2. We’re a small team and maintaining two deployment options is not something we can do.

Having said that if there is a good use case for the non Docker deployment we can see how we can make it a community supported effort or improve the documentation. I just want to understand the use case better first.

Thanks!


#3

Dont dockers use more resources and slow down performance? It’s ok for testing out on a vm but why would I want a docker on a production machine?

When I was looking for an open source dashboard solution I tested out redash and metabase. Metabase came in a docker and it was a pain (it also had other limitations that redash did not have). So my docker experience was painful. It felt like I needed to learn a whole new way of doing things.

I get the small team thing (you are doing a great job) so I understand why you wound not be able to commit the resource to two deployment options. I’ll look at the https://redash.io/help/open-source/dev-guide/setup and see if I can do it on a fresh vm - and make notes as I go.


#4

Hi there Arik. I too opted-out of docker installs. I’m using lxd to manage my linux apps. These work as separated linux boxes and not as containers like docker. This way we can work with usual tool set and not mess with endless docker* tools and configs.

It is a bit worrying that you now force users to go docker or cloud to get redash. That is a huge constraint and had I known it before I would not have selected redash while evaluating existing options.

In the scenario where you uprade dependencies I would create a new clean lxd ubunut machine install redash vX and upgrade the instance database… Can’t see the issue there.

Just letting you know my use case.


#5

On non “native” operating systems like MacOS or Windows Docker is used via a VM, and hence has some overhead. But on a Linux machine for an application like Redash the overhead of using Docker is minimal (if existing). I do know that running databases might introduce some challenges with Docker and certain scale, because of the filesystem use. But we recommend splitting the database to a separate instance anyway for serious use :slight_smile:

I would really not be worried about Docker’s overhead for Redash’s own processes (that is everything except Postgres and Redis). The benefits it provides in terms of ease of deployment definitely worth it.

You can also use the old bootstrap script as reference – I doubt a lot changes since then.

But I encourage you to give the Docker based instances a try to see if the experience changed since the previous time you gave Docker a chance.


#6

We don’t force anyone to do anything :slight_smile: Redash is still open source and I’m sure that an experienced operator will figure out how to deploy it whatever way they need it.

It just that the default deployment we support is using Docker. This makes it easier both for us and the end user.

I will check about lxd, but Docker has the benefit of being more widely adopted and the image more portable – can be used by those who use Docker Compose, Kubernetes, ECS, etc.


#7

So, Im close but stumped.

I edited the old bootstrap and tried it out on a new clean Ubuntu 18.04.1 LTS VM. It all seems to install correctly except for when it is doing the “pip install -r $REDASH_BASE_PATH/current/requirements_all_ds.txt.”

At this point boto3 1.9.26, influxdb 2.7.1 and memsql 2.16.0 dont seem to install because there are new versions of the files they are dependent on (eg six). The rest of the script works. I had an idea to edit the requirements_all_ds.txt file and add a > sign to those line items but didnt want to break it more.

the rest of the instalation seems to work. But when I try to access the site after the instalation compleates it does not look right:

If I enter details in and click setup I get a “Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.”

This is how i edited the bootstrap.sh file:

#!/bin/bash
#
# This script setups Redash along with supervisor, nginx, PostgreSQL and Redis. It was written to be used on 
# Ubuntu 18.04. Technically it can work with other Ubuntu versions, but you might get non compatible versions
# of PostgreSQL, Redis and maybe some other dependencies.
#
# This script is not idempotent and if it stops in the middle, you can't just run it again. You should either 
# understand what parts of it to exclude or just start over on a new VM (assuming you're using a VM).

set -eu

REDASH_BASE_PATH=/opt/redash
REDASH_BRANCH="${REDASH_BRANCH:-master}" # Default branch/version to master if not specified in REDASH_BRANCH env var
REDASH_VERSION=${REDASH_VERSION-5.0.1} # Install latest version if not specified in REDASH_VERSION env var
LATEST_URL="https://github.com/getredash/redash.git"
VERSION_DIR="$REDASH_BASE_PATH/redash.${REDASH_VERSION}"
REDASH_TARBALL=/tmp/redash.tar.gz
FILES_BASE_URL=https://raw.githubusercontent.com/getredash/redash/release/4.0.x/setup/ubuntu/files

cd /tmp/

verify_root() {
    # Verify running as root:
    if [ "$(id -u)" != "0" ]; then
        if [ $# -ne 0 ]; then
            echo "Failed running with sudo. Exiting." 1>&2
            exit 1
        fi
        echo "This script must be run as root. Trying to run with sudo."
        sudo bash "$0" --with-sudo
        exit 0
    fi
}

create_redash_user() {
    adduser --system --no-create-home --disabled-login --gecos "" redash
}

install_system_packages() {
    apt-get -y install software-properties-common
    apt-add-repository universe
    apt-get -y update
    # Base packages
    apt install -y python-pip python-dev nginx curl build-essential pwgen
    # Data sources dependencies:
    apt install -y libffi-dev libssl-dev libmysqlclient-dev libpq-dev freetds-dev libsasl2-dev
    # SAML dependency
    apt install -y xmlsec1
    # Storage servers
    apt install -y postgresql redis-server
    apt install -y supervisor
}

create_directories() {
    mkdir -p $REDASH_BASE_PATH
    chown redash $REDASH_BASE_PATH
    
    # Default config file
    if [ ! -f "$REDASH_BASE_PATH/.env" ]; then
        sudo -u redash wget "$FILES_BASE_URL/env" -O $REDASH_BASE_PATH/.env
    fi

    COOKIE_SECRET=$(pwgen -1s 32)
    echo "export REDASH_COOKIE_SECRET=$COOKIE_SECRET" >> $REDASH_BASE_PATH/.env
}

extract_redash_sources() {
#    sudo -u redash wget "$LATEST_URL" -O "$REDASH_TARBALL"
#    sudo -u redash mkdir "$VERSION_DIR"
    sudo -u redash git clone "$LATEST_URL" "$VERSION_DIR"
#    sudo -u redash tar -C "$VERSION_DIR" -xvf "$REDASH_TARBALL"
    ln -nfs "$VERSION_DIR" $REDASH_BASE_PATH/current
    ln -nfs $REDASH_BASE_PATH/.env $REDASH_BASE_PATH/current/.env
}

install_python_packages() {
    sudo pip install --upgrade pip
    # TODO: venv?
    pip install setproctitle # setproctitle is used by Celery for "pretty" process titles
    pip install -r $REDASH_BASE_PATH/current/requirements.txt
    pip install -r $REDASH_BASE_PATH/current/requirements_all_ds.txt
}

create_database() {
    # Create user and database
    sudo -u postgres createuser redash --no-superuser --no-createdb --no-createrole
    sudo -u postgres createdb redash --owner=redash

    cd $REDASH_BASE_PATH/current
    sudo -u redash bin/run ./manage.py database create_tables
}

setup_supervisor() {
    wget -O /etc/supervisor/conf.d/redash.conf "$FILES_BASE_URL/supervisord.conf"
    service supervisor restart
}

setup_nginx() {
    rm /etc/nginx/sites-enabled/default
    wget -O /etc/nginx/sites-available/redash "$FILES_BASE_URL/nginx_redash_site"
    ln -nfs /etc/nginx/sites-available/redash /etc/nginx/sites-enabled/redash
    service nginx restart
}

verify_root
install_system_packages
create_redash_user
create_directories
extract_redash_sources
install_python_packages
create_database
setup_supervisor
setup_nginx

#8

It didn;t like how i pasted the scrip in above… the BOLD parts are actually # out


#9

You should turn it into a code block by wrapping with “```”.

Based on how it looks, it seems that the compiled static assets are missing.Did you use one of the release tarballs we provide or downloaded codebase from GitHub?


#10

Ok I see - I was installing the codebase. Where are the tarballs - Ive been looking for half an hour and cant find the location. If I knew where they were then I could fix it.


#11

You should’ve followed the original script more closely, as it’s there :wink:

You call version.redash.io/api/releases, and use the download_url property of the release you want.


#12

You are my hero.

I did try with the original file but for some reason it would not find the file - I must have typed something incorrectly - so I tried making a git clone. I now realise you cant just do that. Anyway It was a good learning experience.

The script below works (for others who want to install the latest redash version easily without using dockers) on Ubuntu 18.04.1 LTS.

#!/bin/bash
#
# This script setups Redash along with supervisor, nginx, PostgreSQL and Redis. It was written to be used on 
# Ubuntu 18.04. Technically it can work with other Ubuntu versions, but you might get non compatible versions
# of PostgreSQL, Redis and maybe some other dependencies.
#
# This script is not idempotent and if it stops in the middle, you can't just run it again. You should either 
# understand what parts of it to exclude or just start over on a new VM (assuming you're using a VM).

set -eu

REDASH_BASE_PATH=/opt/redash
REDASH_BRANCH="${REDASH_BRANCH:-master}" # Default branch/version to master if not specified in REDASH_BRANCH env var
REDASH_VERSION=${REDASH_VERSION-5.0.2.b5485} # Install latest version if not specified in REDASH_VERSION env var
LATEST_URL="https://s3.amazonaws.com/redash-releases/redash.${REDASH_VERSION}.tar.gz"
VERSION_DIR="$REDASH_BASE_PATH/redash.${REDASH_VERSION}"
REDASH_TARBALL=/tmp/redash.tar.gz
FILES_BASE_URL=https://raw.githubusercontent.com/getredash/redash/release/4.0.x/setup/ubuntu/files

cd /tmp/

verify_root() {
    # Verify running as root:
    if [ "$(id -u)" != "0" ]; then
        if [ $# -ne 0 ]; then
            echo "Failed running with sudo. Exiting." 1>&2
            exit 1
        fi
        echo "This script must be run as root. Trying to run with sudo."
        sudo bash "$0" --with-sudo
        exit 0
    fi
}

create_redash_user() {
    adduser --system --no-create-home --disabled-login --gecos "" redash
}

install_system_packages() {
    # these 2 lines are needed to get pip installed
    apt-get -y install software-properties-common
    apt-add-repository universe
    apt-get -y update
    # Base packages
    apt install -y python-pip python-dev nginx curl build-essential pwgen
    # Data sources dependencies:
    apt install -y libffi-dev libssl-dev libmysqlclient-dev libpq-dev freetds-dev libsasl2-dev
    # SAML dependency
    apt install -y xmlsec1
    # Storage servers
    apt install -y postgresql redis-server
    apt install -y supervisor
}

create_directories() {
    mkdir -p $REDASH_BASE_PATH
    chown redash $REDASH_BASE_PATH
    
    # Default config file
    if [ ! -f "$REDASH_BASE_PATH/.env" ]; then
        sudo -u redash wget "$FILES_BASE_URL/env" -O $REDASH_BASE_PATH/.env
    fi

    COOKIE_SECRET=$(pwgen -1s 32)
    echo "export REDASH_COOKIE_SECRET=$COOKIE_SECRET" >> $REDASH_BASE_PATH/.env
}

extract_redash_sources() {
    sudo -u redash wget "$LATEST_URL" -O "$REDASH_TARBALL"
    sudo -u redash mkdir "$VERSION_DIR"
    sudo -u redash tar -C "$VERSION_DIR" -xvf "$REDASH_TARBALL"
    ln -nfs "$VERSION_DIR" $REDASH_BASE_PATH/current
    ln -nfs $REDASH_BASE_PATH/.env $REDASH_BASE_PATH/current/.env
}

install_python_packages() {
    sudo pip install --upgrade pip
    pip install setproctitle # setproctitle is used by Celery for "pretty" process titles
    pip install -r $REDASH_BASE_PATH/current/requirements.txt
    pip install -r $REDASH_BASE_PATH/current/requirements_all_ds.txt
}

create_database() {
    # Create user and database
    sudo -u postgres createuser redash --no-superuser --no-createdb --no-createrole
    sudo -u postgres createdb redash --owner=redash

    cd $REDASH_BASE_PATH/current
    sudo -u redash bin/run ./manage.py database create_tables
}

setup_supervisor() {
    wget -O /etc/supervisor/conf.d/redash.conf "$FILES_BASE_URL/supervisord.conf"
    service supervisor restart
}

setup_nginx() {
    rm /etc/nginx/sites-enabled/default
    wget -O /etc/nginx/sites-available/redash "$FILES_BASE_URL/nginx_redash_site"
    ln -nfs /etc/nginx/sites-available/redash /etc/nginx/sites-enabled/redash
    service nginx restart
}

verify_root
install_system_packages
create_redash_user
create_directories
extract_redash_sources
install_python_packages
create_database
setup_supervisor
setup_nginx