Install and Run Redash on Apple M1 Pro Chip

SteveJ · June 14, 2022, 6:46am

Issue Summary

Hi Redash community,

I was trying to install and run Redash using docker on my macbook with M1 Pro chip. I have ran into a number of errors when I followed the Docker Based Developer Installation Guide.

The first few errors happened during docker-compose when installing python packages. I managed to bypass those errors by adding pre steps in dockerfile and commenting a few packages in requirements_all_ds.txt file. They are:

add one step before install python packages in dockerfile:
RUN pip install Cython;
this is because pymssql will need Cython as a dependency, and it somehow has to be installed first.
comment pymapd and ibm-db as pyarrow and another dependency won’t work with M1 Chip

After fixing those errors, the containers were up and running, and localhost:5000/ping returns pong.

However I was later stuck at installing dependency for the front-end. I followed the instruction yarn --frozen-lockfile. It returned errors of
error An unexpected error occurred: "https://registry.yarnpkg.com/domhandler/-/domhandler-2.4.2.tgz: read ECONNRESET".
and finally stopped at the stage of fetching packages. I managed to find another post in this forum talking about M1 chip issue and replaced the command with npm install && npm run build. Unfortunately, it returned the same error - error code EINTEGRITY.

As I don’t have much experience with frontend development and really desperate to find a solution to those errors, I just wonder if someone here has met the same problems and can shed a bit of light on this?

By the way, I tried another mac with Intel chip and under the same network, it works well with all those steps in the installation guide. So it can’t be the network issue.

The below are full logs for yarn --frozen--lockfile command

[14/06/22 4:07:44] (base)  ✘ jize@G536RR24R3  ~/Documents/GitHub/redash   master ±  yarn --frozen-lockfile
yarn install v1.22.19
$ cd viz-lib && yarn link --link-folder ../.yarn
yarn link v1.22.19
warning There's already a package called "@redash/viz" registered. This command has had no effect. If this command was run in another folder with the same name, the other folder is still linked. Please run yarn unlink in the other folder if you want to register this folder.
✨  Done in 0.03s.
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
info There appears to be trouble with your network connection. Retrying...
info No lockfile found.
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] 🔍  Resolving packages...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
warning babel-cli > chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning babel-cli > babel-register > core-js@2.6.12: core-js@<3.4 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
warning babel-cli > babel-polyfill > core-js@2.6.12: core-js@<3.4 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
warning babel-cli > babel-runtime > core-js@2.6.12: core-js@<3.4 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
warning babel-cli > chokidar > fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
info There appears to be trouble with your network connection. Retrying...
warning babel-cli > chokidar > readdirp > micromatch > snapdragon > source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
info There appears to be trouble with your network connection. Retrying...
warning babel-cli > chokidar > readdirp > micromatch > snapdragon > source-map-resolve > resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
warning babel-cli > chokidar > readdirp > micromatch > snapdragon > source-map-resolve > urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
warning babel-cli > chokidar > readdirp > micromatch > snapdragon > source-map-resolve > source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
warning babel-eslint@7.2.3: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "https://registry.yarnpkg.com/domhandler/-/domhandler-2.4.2.tgz: read ECONNRESET".
info If you think this is a bug, please open a bug report with the information provided in "/Users/jize/Documents/GitHub/redash/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
warning babel-preset-es2015@6.24.1: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read https://babeljs.io/env to update!
info There appears to be trouble with your network connection. Retrying...
warning eslint > file-entry-cache > flat-cache > circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor.
info There appears to be trouble with your network connection. Retrying...
warning jest > jest-cli > sane@1.4.1: some dependency vulnerabilities fixed, support for node < 10 dropped, and newer ECMAScript syntax/features added
warning jest > jest-cli > jest-haste-map > sane@1.4.1: some dependency vulnerabilities fixed, support for node < 10 dropped, and newer ECMAScript syntax/features added
warning jest > jest-cli > jest-config > istanbul@0.4.5: This module is no longer maintained, try this instead:
  npm i nyc
Visit https://istanbul.js.org/integrations for other alternatives.
warning jest > jest-cli > jest-environment-jsdom > jsdom > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
info There appears to be trouble with your network connection. Retrying...
warning jest > jest-cli > jest-environment-jsdom > jsdom > request > har-validator@5.1.5: this library is no longer supported
warning jest > jest-cli > jest-environment-jsdom > jsdom > request > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
info There appears to be trouble with your network connection. Retrying...
warning webpack > watchpack > chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning webpack > node-libs-browser > url > querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
[2/4] 🚚  Fetching packages...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...

Technical details:

Redash Version: v10.1.0
Browser/OS: macOS Monterey 12.4, MacBook Pro with Apple M1 Pro
How did you install Redash: Following Docker Based developer installation guide.

codeofyu · June 14, 2022, 2:22pm

Hey, I also encountered this song problem can not be solved!!!

jesse · June 14, 2022, 5:14pm

Related context: Error: Unable to run docker-compose on Apple M1 Silicon - #16 by ajgreen

SteveJ · June 14, 2022, 11:48pm

Hi @jesse ,

thanks for your reply. I checked that page before I made new post. It was one error I met before. I solved it by pointing node version to 14 in homebrew and after that, node and yarn commands can be triggered.
My error is when installing dependencies for the frontend, some packages look like not compatible with M1 chip. I’m not a front end developer, so it’s just my gut feeling. The error message was error An unexpected error occurred: "https://registry.yarnpkg.com/domhandler/-/domhandler-2.4.2.tgz: read ECONNRESET".. Would you mind shedding some lights on?

Thanks,
Steve

jesse · June 21, 2022, 4:40pm

Hmmm, I’ll give this a try on an M1 system when I have a moment.

This shouldn’t be a problem given that M1 supports Rosetta 2.0, no?

SteveJ · June 28, 2022, 12:25pm

Hi @jesse ,

Thanks for your reply. yes, it supports Rosetta. I haven’t touched that bit for now. Will explore it and post if I have any updates.

jesse · July 2, 2022, 4:24pm

I have a working version of Redash v11-dev on my M1 machine now.

Hardware: M1 Max Macbook Pro (2021)
OS: MacOS Monterey 12.4
CPU: M1 Max

Backend (Docker)

I too had to follow @SteveJ 's guidance to:

Add RUN pip install Cython to my Dockerfile
Comment out requirements_all_ds.txt dependencies that won’t build on M1

I actually just built my containers with skip_ds_deps enabled, since I didn’t need them for testing. Just run docker-compose build --build-arg skip_ds_deps=1 before you run docker-compose up -d.

Also, MacOS Monterey uses port 5000 for system services (Airplay discovery, I think). So I modified the EXPOSE command in my Dockerfile to use port 5001 instead.

Frontend (yarn)

I had a few hiccups:

I needed to install node 14.18.2 with nvm because my version of node was set to 17.x.x… Before I did this, there were all manner of strange errors around incompatibility.
I modified my webpack.config.js to set redashBackend to port 5001

Complete patch:

You can use this patch to make my changes on your local version.

From ce29bb8fa9291c690965359feb99272e4477a301 Mon Sep 17 00:00:00 2001
From: Jesse Whitehouse <jesse@whitehouse.dev>
Date: Sat, 2 Jul 2022 11:16:08 -0500
Subject: [PATCH] Make compatible with M1 Macs

---
 Dockerfile         | 1 +
 docker-compose.yml | 2 +-
 webpack.config.js  | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 68f3cf974..272d6cb9e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -88,6 +88,7 @@ ENV PIP_NO_CACHE_DIR=1
 
 # rollback pip version to avoid legacy resolver problem
 RUN pip install pip==20.2.4;
+RUN python3.7 -m pip install cython
 
 # We first copy only the requirements file, to avoid rebuilding on every file change.
 COPY requirements_all_ds.txt ./
diff --git a/docker-compose.yml b/docker-compose.yml
index fdfcdd1a5..1a63eb14b 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -27,7 +27,7 @@ services:
       - postgres
       - redis
     ports:
-      - "5000:5000"
+      - "5001:5000"
       - "5678:5678"
     environment:
       <<: *redash-environment
diff --git a/webpack.config.js b/webpack.config.js
index 76465291d..5aa2a49af 100644
--- a/webpack.config.js
+++ b/webpack.config.js
@@ -34,7 +34,7 @@ const isDevelopment = !isProduction;
 const isHotReloadingEnabled =
   isDevelopment && process.env.HOT_RELOAD === "true";
 
-const redashBackend = process.env.REDASH_BACKEND || "http://localhost:5000";
+const redashBackend = process.env.REDASH_BACKEND || "http://localhost:5001";
 const baseHref = CONFIG.baseHref || "/";
 const staticPath = CONFIG.staticPath || "/static/";
 const htmlTitle = CONFIG.title || "Redash";
-- 
2.32.1 (Apple Git-133)

[edit] On my personal fork I have a dev branch that applies these changes. The README includes basic setup steps: GitHub - susodapop/redash: Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

jesse · July 5, 2022, 8:58pm

Following up, I think pyarrow will work on M1 if we make a few changes. The problem is there aren’t wheels for pyarrow that support the M1 architecture, so pip tries to build one.

jesse · July 9, 2022, 2:09am

I’ve opened a PR with a potential fix for this: install requirements_all_ds.txt line-by-line so that the build will succeed even when some of the data source requirements cannot be installed.

github.com/getredash/redash

Install python dependencies one-by-one. Don't exit on failures.

getredash:master ← getredash:fix-m1-build

opened 02:08AM - 09 Jul 22 UTC

susodapop

+2 -1

## What type of PR is this? - [x] Refactor ## Description `docke…r-compose build` fails while installing `requirements_all_ds.txt` on Apple Silicon because some data source dependencies are pinned to versions incompatible with the ARM architecture. This blocks users from running / developing Redash on our fancy new Macs unless we selectively remove the incompatible lines of `requirements_all_ds.txt` (`pymapd` for example). Long-term we should ensure that all dependencies are compatible across platforms. This PR implements an effective, hands-free solution until then. #### Benefits This change allows the build to continue even if a data source dependency cannot be installed. It works by using `xargs` to feed `requirements_all_ds.txt` to `pip install` line-by-line. The final command `|| true` suppresses the exit code from `xargs` (which will be code `123` if any of the calls to `pip install` returns a non-zero code) so that Docker will continue building. Since query_runners wrap their imports in `try...` blocks, any query runner that depends on a package that wasn't installed will fail gracefully without crashing the app. #### Risks Calling `pip install` line-by-line slows down `docker-compose build`. Because we catch all non-zero exit codes, a user will need to run `docker-compose build --progress=plain` to see whether or not all dependencies were installed. ## How is this tested? - [ ] Unit tests (pytest, jest) - [ ] E2E Tests (Cypress) - [x] Manually - [ ] N/A ## Related Tickets & Documents ## Mobile & Desktop Screenshots/Recordings (if there are UI changes) N/A

jesse · July 13, 2022, 1:58am

Redash is a community driven project. The discussion and pull request in this thread exist to move us towards full Apple Silicon compatibility. But it happens at the rate of volunteers

jesse · July 13, 2022, 2:02am

As a follow-up, it’s unlikely that we’ll build images for Apple M1 specifically since it’s unlikely for Redash to be deployed on these platforms. The purpose of this discussion is for development / demonstration instances of Redash. In a production environment we want to support arm64 platforms, but probably not M1 specifically – unless Apple return to the enterprise server market