Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

What about using Redis as a celery backend ? Redis has a pub sub mechanism which seems quite reliable, so no need to poll.


I used Redis for celery in production with great success for a year but then we started running some long running jobs that needed the ACKS_LATE setting and the Redis delivery timeout kept hurting us by resending the task to another worker. It's configurable but in the end we just switched to RabbitMQ. I found it quite painless to setup and migrate to.


Redis is still not an AMQP, but yes Redis's Pub/Sub works quite nicely. Out of all the brokers celery supports I'd recommend only RabbitMQ and Redis to people.


Yeah, I've been using redis with celery in production to perform lots of network io related tasks on a low end machine because

a > Redis uses less memory

b > Redis is easier to setup


With container solutions like Docker and prebuilt images the setup part is kinda eliminated. Although I don't remember having any special configuration issues with RabbitMQ as well, it just works TM. That's always nice right? :)


That's been my experience with Redis + Celery as well vs. the other brokers.


Redis works great as a results backend, but I'd still use RabbitMQ for the queue. RabbitMQ is designed to be a message queue, and it does a great job at it.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: