How works Excel and CSV datasources?

Issue Summary

Hello (again) !
According to 9207, I’m looking to query an Excel file. Unfortunately something wrong append and I don’t really understand why as I follow the tutoriel.

Here’s some captures about my problem :

Any idea to solve it ? How works Excel data-sources ? May I have to install the requirements_all_ds.txt dependencies ?

Thank you again !

William

Technical details:

• Redash Version: v.10.0.1
• Browser/OS: Firefox Browser v.99.0.1
• How did you install Redash: Docker

Unfortunately, we’ve some troubles to install all the dependencies from requirements_all_ds.txt. A lot of conflits with python 3.8 and others applications working with it.

How are you installing these requirements? I’ve not seen any reports of conflicts during installation…

Here’s what we did :

git clone https://github.com/getredash/redash.git
cd redash/
touch .env
nano .env /*inscription de : REDASH_COOKIE_SECRET = X*/

nano docker-compose.yml /*change port 5000:5000 to 5050:5000*/
pip install -r requirements.txt -r requirements_dev.txt
npm install --global yarn@1.22.10
sudo npm install --global yarn@1.22.10

yarn --frozen-lockfile
yarn build
docker compose up -d

docker compose run --rm server create_db

At this point, everything was fine. We just had to reencrypt datasources and it worked.
But then, we try this :

pip install -r requirements_all_ds.txt

And impossible to follow the installation due to errors :

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

There is some troubles with pymssql :

ERROR: Could not find a version that satisfies the requirement pymssql==2.1.4 (from versions: 

But also with others depencies I had to install :

apt-get install libmysqlclient-dev
pip install pymssql-linux
pip install Cython

LAST ERROR :

Collecting pymssql==2.1.4
  Using cached pymssql-2.1.4.tar.gz (691 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      /usr/local/lib/python3.8/dist-packages/setuptools/dist.py:51: DistDeprecationWarning: Do not call this function
        warnings.warn("Do not call this function", DistDeprecationWarning)
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-uzk7tq3b/pymssql_9ce41034d18347d0aa036a72c65d8d0c/setup.py", line 160, in <module>
          print("setup.py: platform.linux_distribution() => %r" % (platform.linux_distribution(),))
      AttributeError: module 'platform' has no attribute 'linux_distribution'
      setup.py: platform.system() => 'Linux'
      setup.py: platform.architecture() => ('64bit', 'ELF')
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: Discarding https://files.pythonhosted.org/packages/2e/81/99562b93d75f3fc5956fa65decfb35b38a4ee97cf93c1d0d3cb799fffb99/pymssql-2.1.4.tar.gz#sha256=3201eb1b1263ad55b555d727ed8bed0b12b7b9de3ce5e529206e36d4be1a7afb (from https://pypi.org/simple/pymssql/) due to build failure: metadata generation failed
ERROR: Could not find a version that satisfies the requirement pymssql==2.1.4 (from versions: 1.0.0, 1.0.1, 1.0.2, 1.0.3, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4rc1, 2.1.4, 2.1.5, 2.2.0, 2.2.1.dev0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5)
ERROR: No matching distribution found for pymssql==2.1.4

Something is wrong with our configuration. Any idea ?

Our configuration :

node -v /*return : 14.19.1*/
yarn -v /*return : 1.22.10*/
python3 --version /*return : Python 3.8.10*/
pip --version /*return : pip 20.0.2*/
npm version /*return : 6.14.16*/

Thanks for your help!

Thanks for providing your detailed setup steps. I think that’s your problem.

You’re trying to blend a local python development setup with a Docker development setup. All of the Python dependencies will be installed when you run docker-compose up -d. You don’t need to install these on your local machine. If you do install them locally, there’s a bunch of additional setup steps you’ll need to configure your postgres and redis instances.

Likely the installation problem is because you’re trying to install Python deps on your system python installation. It would be best to use a virtualenv instead. I suggest you abandon this path until you’re more familiar with how Redash works. Stick to the Docker Development setup. It’s much simpler.

For good measure: the excel and csv data sources aren’t working because your installation is broken. Once you install Redash properly they’ll work just fine.

Hi Jesse ! Thanks for your help ! It helped us a lot !
We reinstalled everything under Docker (according to this). Now we’re close but there are still some mistakes.

CONFIGURATION
Docker : 20.10.14
Node : 14.19.1
Yarn : 1.22.10

ERRORS
FIRST STEP : docker-compose up -d
C1

C2

C3

C4

C5

I think nothing was very wrong because docker start well at this step !
C6

SECOND STEP : yarn --frozen-lockfile
C7

C8

C9

At this step, after creating the database with

docker-compose run --rm server create_db

we had this :

C10

THIRD (AND LAST) STEP : yarn build
Unfortunately, we can’t build the frontend part with yarn build and yarn start due to this error :
C11

Any idea ? How can we solve this issue ?
Thanks again for everything

About the fsevents errors: these aren’t an issue. fsevents only works on MacOS but it’s not required for linux.

That is a strange error. I wonder: did you completely wipe this system before starting over on the Docker development setup? I wonder if your previous attempts created some conflicting dependency installation on the machine. Otherwise your commands look fine.

Also, what operating system are you using?

I think so !
We wipe the system as we’re working on a Virtual Machine with Ubuntu 20.04. We used an old snapshot to do it.
For you, the problem only comes from the build of frontend with yarn build ? We can try to update manualy all depencies like typescript. We’re not the only ones (#10011).

Thanks again, We’ll try again. If you have others ideas about what we can do… feel free to share, we’re lost

We only have 2GB of memory on our VM. Do you think it could be the problem ?

We have this error during yarn build :

error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Thanks.

Yeah, this is specifically mentioned in the Docker setup instructions:

If you hit an errno 137 or errno 134 particularly at RUN yarn build , make sure you give your Docker VM enough memory (4GB or more).

This is a fair point. Very strange that it doesn’t reproduce for almost anyone. i.e. nobody on the core team is able to reproduce this error…

Thanks for your answer. There is no way to install Redash on a 2G memory VM ?

Well, thanksful to your advices, we have now an instance of Redash working fine
Thanks a lot Jesse !

We still have two questions :

1. Is it possible to install Redash on a 2GB memory VM ?
2. Is it possible to specify a sheet inside an Excel document as Spreadsheets with ‘|0’ or ‘|1’?

About Redash ft. Nextcloud (XLS or XLSX) :

1. create an Excel datasource (you’ll use the same datasource for all your xls/xlsx documents !).
   

2. Inside Nextcloud, share and copy the link
   

   

   Capture11906×449 33 KB

3. Open a new tab with the link and right click on “Download” to copy the targetlink
   

   

   Capture21898×689 85.8 KB

4. Go back to redash, create a query with your excel datasource and paste the link preffixed with “url:”
   

   

   capture31920×880 60.6 KB

5. Refresh and enjoy

Thanks again for your help !

UPDATE
1. Is it possible to install Redash on a 2GB memory VM ?
→ yes, I wrote a trick here : #10426

2. Is it possible to specify a sheet inside an Excel document as Spreadsheets with ‘|0’ or ‘|1’?
→ yes : #9207