Running celery worker on separate machine

Hello,

Is it possible to setup celery on separate machine than redash server. We are trying to install redash in a decouple manner to make it scaleable and for that we were thinking if we can redirect different celery tasks to different machines.

so for example I have two data sources
ds1 – queue = ‘ds1_queue’
ds2 – queue = ‘ds2_queue’

Is it possible to set ds1_queue and ds2_queue on separate machine.

current architecture uses same machine as redash server for all the processing.

Thanks

You can certain do this. Our Docker-compose configuration effectively virtualizes that exact setup. Just point your workers at the same Redis.

1 Like

Thank you!

Can you please explain it a bit? I studied a bit around it and I think we need to place redash code on all machines. is that right? also can you please share how redash queues work? I mean how celery workers only get the tasks from redis which are assigned to those workers not the tasks from some other workers queue?

Appreciate your support.

Hello,

Can you please share some details how can I achieve this. Thank you in advance