We merged our redis queue work we still
We merged our redis queue work, we still need to do more testing to switch production to it. It won't matter to you unless you want to run 500+jobs/secs. It's an optional replacement for our postgres queue.
4 Replies
1- I'm curious was nats considered for managing the queue instead of redis?
2- https://dagster.io/blog/skip-kafka-use-postgres-message-queue dagster was recommending postgres queue instead of kafka or redis. They didn't release any source code so god knows what was their implementation.
1. probably nats could have worked as well. and probably we could add nats one day if the needs arise. The rust implementation of rsmq fitted our bill well so we went with that. I believe nats and redis have similar performance characteristics but there are some invariants that were really nice about rsmq
2. I think dagster doesn't want to process jobs whose duration is 5ms like we'd like to be able to with the deno runtime so probably 500 by shard is plenty enough for them ?
they might care a lot less about latency in general too. For low latency we really needed UPDATE SKIP LOCKED and it's expensive.
I am genuinely grateful for your efforts in achieving optimal performance for windmill.
Thanks
1. it's cool 😎
2. it's the best way to convince devs to use such a tool
3. it allows for a lot of usage pattern that would not make sense otherwise, like constant polling of source and semi-streaming