1

I’ve been using the AWS AMI (Redash 8.0.0 ami-0527e82bae7c51958) in Singapore region on a t2.small EC2 instance successfully for almost a year.

Yesterday I got an Internal Server Error message when I tried to access the Redash front-end.

In the past when I’ve had any issues I’ve been able to reboot the instance and it would start working again. This time however it’s not working. I already tried upgrading to larger EC2 instances (t2.medium and t2.large).

I also can’t connect using the EC2 Instance Connect (browser-based SSH connection) using the recommended ubuntu user name specified in the Redash AWS setup documentation (I also tried the default root user name and that also failed. The error message I’m seeing is:

There was a problem setting up the instance connection
Log in failed. If this instance has just started up, try again in a minute or two.

I’m guessing since the original instance was t2.small I must have hit a resource limit that’s causing it to crash. But since I can’t login to the server to check I’m not sure how to fix it.

Has anyone had this issue before and can advise on:

  • (1) how to login to the instance or
  • (2) how to fix it some other way

Thanks. I’ve been a happy Redash user for many years.

2

Update:
When I check the system log I see
OSError: [Errno 28] No space left on device

I tried doubling the EBS volume from 8to 16 GB and I still get this error.

3

Solved it. I’ll leave the solution here in case it helps anyone else in the future:

Basically what happened is the disk was 100% full. To avoid this catching you by surprise you should set an alarm to alert you before it gets too full.

The way to fix this in AWS is to find the EBS (Elastic Block Store) Volume and select Modify Volume and increase the size, e.g. 8GB --> 16GB.

This won’t automatically fix it the next time you try to start your instance, you’ll still get the No space left on device error, which you can see in the Instances > Instance Settings > Get System Log.

Because of this error you won’t be able to ssh or connect to this box in any way to fix it, so you’ll need to detach the EBS and attach it to another EC2 that you can ssh into.

Once you do this and can access the volume you need to give it access to the additional disk and verify that it’s no longer at 100% used capacity.

Then you can detach it again and restart your Redash instance with this fixed EBS volume.

4 Likes
4

Hey there, I am facing this same issue.
I was able to solve it by increasing disk in the EC2 instance but few hours later ended up on the same problem :frowning: … Were you able to find the root cause of those overlay2 massive folders piling up in size?