Rebooted EC2 instance with Redash AMI - all processes stopped working

Support Self Hosted Redash Support
1

Issue Summary

Hi Team,

I set up Redash on EC2 using the AWS AMI. I rebooted the instance, and all the processes stopped working. I tried to look for supervisorctl, but that too isn’t installed in the AMI. Kindly help! We are using Redash in production.

Technical details:

  • Redash Version: redash-8.0.0
  • Browser/OS:Chrome/Ubuntu 18.04.3 LTS
  • How did you install Redash: Through AWS AMI (redash-8.0.0-b32245-1-us-east-1
    ami-0d915a031cabac0e0)
2

Welcome to the forum and sorry you’ve in pinch here :confused:

The AMI’s are configured to automatically launch docker-compose after reboot. The images are intentionally stripped quite thin so that only the necessary services are installed, which is why you don’t see supervisorctl.

What do you see when you run docker-compose ps?

3

Hi @jesse

When I run docker-compose ps, here’s what I see:

docker-compose ps
ERROR: Couldn’t connect to Docker daemon at http+docker://localhost - is it running?

If it’s at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

4

Try prepending sudo to the command.

5

sudo docker-compose ps
ERROR:
Can’t find a suitable configuration file in this directory or any
parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

This is what I see.

6

There you see that’s progress. The error message is pretty clear. You need to be in the /opt/redash directory.

7

Thanks @jesse

Am I good to go? Do I need to manually restart any containers? Here’s the output of docker-compose ps:

sudo docker-compose ps
          Name                         Command                 State                 Ports           
-----------------------------------------------------------------------------------------------------
redash_adhoc_worker_1       /app/bin/docker-entrypoint ...   Exit 1                                  
redash_nginx_1              nginx -g daemon off;             Up           443/tcp, 0.0.0.0:80->80/tcp
redash_postgres_1           docker-entrypoint.sh postgres    Up           5432/tcp                   
redash_redis_1              docker-entrypoint.sh redis ...   Up           6379/tcp                   
redash_scheduled_worker_1   /app/bin/docker-entrypoint ...   Exit 1                                  
redash_scheduler_1          /app/bin/docker-entrypoint ...   Restarting                              
redash_server_1             /app/bin/docker-entrypoint ...   Restarting
8

Look at the output. Both your workers have crashed and two other services are stuck restarting. You need to figure out why.

So run sudo docker-compose logs scheduled_worker and see why that container crashed. Do the same for the other containers.