Export of large CSV via API

Support Self Hosted Redash Support
1

Hi there,
is there any way how I can export the results of the query for a quite large dataset (~200MB)?
I have tried to download via UI and API - in both cases, it fails with error 502.
I saw that there is the API client (redash-toolbelt), is it possible to set up export there in chunks?

Regards,
izdryk

2

Interesting that export in JSON format works fine. Only export to CSV fails

3

Hey @iho01 thanks for your question. The code doesn’t explicitly limit the size of exports. The 502 error means something went wrong during the export process. It may have run out of memory.

Can you check your docker logs for a stack trace?

4

hi @jesse
I see only this error in the log:

image
image1808×85 24.9 KB

5

Try increasing your redash debug level using the REDASH_LOG_LEVEL=DEBUG environment variable and try again.

My best guess is this is an nginx thing…

6

hi @jesse
here is the debug log:
[2021-12-07 09:37:52,192][PID:37][DEBUG][metrics] table=users query=select duration=2.55
[2021-12-07 09:37:52,195][PID:37][DEBUG][metrics] table=api_keys query=select duration=1.81
[2021-12-07 09:37:52,201][PID:37][DEBUG][metrics] table=queries query=select duration=4.14
[2021-12-07 09:37:52,205][PID:37][DEBUG][metrics] table=data_sources query=select duration=1.94
[2021-12-07 09:37:52,209][PID:37][DEBUG][metrics] table=data_source_groups query=select duration=1.61
[2021-12-07 09:37:52,213][PID:37][DEBUG][metrics] table=queries query=select duration=1.71
[2021-12-07 09:37:53,304][PID:37][DEBUG][metrics] table=query_results query=select duration=1089.27
[2021-12-07 09:37:54,156][PID:37][DEBUG][metrics] table=data_sources query=select duration=1.29
[2021-12-07 09:37:54,158][PID:37][DEBUG][metrics] table=data_source_groups query=select duration=0.84
[2021-12-07 09:37:54,160][PID:37][DEBUG][metrics] table=data_source_groups query=select duration=0.77
[2021-12-07 09:38:22 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:37)
[2021-12-07 09:38:22,051][PID:37][INFO][metrics] method=GET path=/api/queries/167/results.csv endpoint=query_result status=500 content_type=? content_length=-1 duration=29841.68 query_count=11 query_duration=1108.74
[2021-12-07 09:38:22 +0000] [37] [INFO] Worker exiting (pid: 37)
[2021-12-07 09:38:22 +0000] [44] [INFO] Booting worker with pid: 44

7

How did you obtain those logs? It shows the 500 error, but you should be able to see the actual stack trace of the error.

To show all the logs:

  1. change the REDASH_LOG_LEVEL environment variable to DEBUG
  2. Restart Redash
  3. Run docker-compose logs -f --tail
  4. In your browser, attempt the download