Scheduled script stops scheduling after wait for executor
Hi!
We are experiencing some issues on one of our Windmill instances. It runs a few scheduled jobs and for a still unknown reason sometimes the scheduled jobs wait a very long time for an executor. Also sometimes it seems the run time is displayed as a negative number (which i'm guessing may be some sort of overflow due to crazy long wait times).
The jobs do seem to eventually run just fine, even if the run time is broken like that.
The problem is that after such a "broken" job runs, the schedule stops working and no further jobs are scheduled. We are trying to figure out what the cause behind the large executor wait time may be (as the instance is not under heavy load at all and the jobs don't take very long to execute).
I guess the problem consists of two parts
- The long executor wait time for whatever reason (despite the actual worker load being low)
- And the second reason which i want to ask about here: Why does the schedule seem to stop working after a job waits a while for an executor? Is this a known issue or possibly even expected behavior?
Sadly i have not managed to reproduce this issue on another Windmill instance..
We are experiencing some issues on one of our Windmill instances. It runs a few scheduled jobs and for a still unknown reason sometimes the scheduled jobs wait a very long time for an executor. Also sometimes it seems the run time is displayed as a negative number (which i'm guessing may be some sort of overflow due to crazy long wait times).
The jobs do seem to eventually run just fine, even if the run time is broken like that.
The problem is that after such a "broken" job runs, the schedule stops working and no further jobs are scheduled. We are trying to figure out what the cause behind the large executor wait time may be (as the instance is not under heavy load at all and the jobs don't take very long to execute).
I guess the problem consists of two parts
- The long executor wait time for whatever reason (despite the actual worker load being low)
- And the second reason which i want to ask about here: Why does the schedule seem to stop working after a job waits a while for an executor? Is this a known issue or possibly even expected behavior?
Sadly i have not managed to reproduce this issue on another Windmill instance..
