1

Issue Summary

I am experience slaggish performance on self-hosted redash on Kubernetes cluster

Technical details:

  • Redash Version: 8
  • Browser/OS: Chrome
  • How did you install Redash: using kubernetes charts from redash contrib repository

Running on AWS EKS cluster. Postress DB is a clustered Aurora Postgress instance.
For example, if I am getting list of data source, I see in the pod logs:
method=GET path=/api/data_sources endpoint=data_sources status=200 content_type=application/json content_length=2147 duration=65.92 query_count=22 query_duration=41.33
or
method=GET path=/api/data_sources endpoint=data_sources status=200 content_type=application/json content_length=2147 duration=335.57 query_count=22 query_duration=52.11

Why so big difference between duration and query-duration? Is there any way to optimize?

My resources for service are following:
resources:
limits:
cpu: 1000m
memory: 4Gi
requests:
cpu: 1000m
memory: 2Gi

I do not see any errors in logs. Version 6 (docker install) was performing just file with even smaller Database)

1 Like
2

This has come up before with regard to Kubernetes:

It seems like your nginx is taking awhile to serve the request which causes the discrepancy between request duration and query duration.

1 Like
3

I figured this out. I use AWS ALB to terminate HTTPS connection and use http to talk to redash.
My k8s annotations were like those:

service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0,
external-dns.alpha.kubernetes.io/hostname: mydns
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:760781926556:certificate/3154bb0",
# Note that the backend talks over HTTP.
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http,
# Only run SSL on the port named "https" below.
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

this resulted in a lot of 302 status code on the backend. After rolling back to plain http performance increased. That said, I should have to use nginx to enable https instead of relying on ALB.

2 Likes