No authentication for CSV in Version 10

er-sarjan · November 1, 2021, 11:32am

Issue Summary

I am migrating hosted redash from app.redash.io to self hosted version. While migrating to latest version of redash, it has the CSV option as a data source but it does not have Authentication parameters.

Self Hosted Version

Current Hosted Version

There is also a difference in Data Source for CSV.
Self Hosted version has CSV source but hosted version has CSV from URL.

How to make csv url work that has authentication?

Technical details:

Redash Version: Version: 10.0.0 (9c928bd1)
Browser/OS: CentOs with Docker Image of Redash 10
How did you install Redash:
Manual Installation

er-sarjan · November 8, 2021, 7:18am

Any update on this? Any workaround?

jesse · November 8, 2021, 3:39pm

Sorry this slipped past my radar. Thanks for bumping the topic.

It’s not possible with the csv query runner in Redash V10. We should add this.

Meanwhile, we did open source the old CSV query runner here. You can add it to your instance by duplicating the file in the redash/query_runner directory and adding it to the default_query_runners setting in redash/settings/__init__.py. If you need help with this please send me a message.

er-sarjan · November 9, 2021, 8:22am

Hi jesse,

Thanks for the prompt reply.

I made the changes that you suggested in docker image directly. After changes, I restarted the docker container and I can see the CSV from Url data source in list.

After setting up a new data source with Authentication parameter, it does not work for testing a connection or making a query.

When I run a query in query console, it fails without any success or error.

Am I missing something? Is there any other changes needs to add to make it work?

er-sarjan · November 9, 2021, 12:16pm

This is little off topic but related to CSV as I’m trying to make csv queries work with some work-around.

I setup a proxy API that works without requiring authentication but now, the Redash CSV query is not working for an API endpoint. It works fine on when I hit it from browser or Postman.

The query_results API is returning Forbidden response.

Part of response

{
“data”: {
“columns”: [
{
“name”: “{“message”:“Forbidden”}”,
“friendly_name”: “{“message”:“Forbidden”}”,
“type”: “string”
}
],
“rows”: [ ]
}
}

Is there any restriction for hitting the API host? CSV from github raw file is working fine.

github.com

getredash/redash/blob/master/redash/query_runner/csv.py#L56

    
      
          path = ""
          ua = ""
          args = {}
          try:
              args = yaml.safe_load(query)
              path = args['url']
              args.pop('url', None)
              ua = args['user-agent']
              args.pop('user-agent', None)
          
          
    if is_private_address(path) and settings.ENFORCE_PRIVATE_ADDRESS_BLOCK:
                  raise Exception("Can't query private addresses.")
          except:
              pass
          
          
try:
              response = requests.get(url=path, headers={"User-agent": ua})
              workbook = pd.read_csv(io.BytesIO(response.content),sep=",", **args)
          
          
    df = workbook.copy()
              data = {'columns': [], 'rows': []}

I saw this but the error message is different in response so I’m not sure what is causing the problem.

My API endpoint is not getting hit by Redash.

Any thoughts?

justinclift · November 9, 2021, 1:55pm

Just to be super clear (!), the source code for the old CSV query runner is released under the same license as the main Redash code base yeah?

So, in theory it could be added back to the Redash source code (with appropriate naming)?

jesse · November 9, 2021, 6:03pm

Yes same license. I updated the gist to reflect that.

We’re not going to have two CSV query runners on master. But anyone is free to add this if they want.

The new one just needs to support basic auth

er-sarjan · November 10, 2021, 7:40am

The problem was related to missing user-agent.

For CSV data source, use following query syntax:

url: "CSV_ENDPOINT"
user-agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"

jesse · November 10, 2021, 8:05pm

I will update our documentation to include this as a debugging step.