Waiting for workers
The Windmill cloud hosted instance seems to be missing out on workers. picniclabs-prd is the workspace.
Jobs are falling into Canceled status, or coming up with a warning message:
Time spent waiting for an executor: 108.449s
This job spent an unexpected amount of time waiting for a worker before starting.
In a healthy queue, jobs are expected to start in under 50ms.
I know that we are must share resources with other workspaces, but we had over 500 jobs in canceled status over the morning (6 hours from 7am through to 1pm)
We generally have less than 3 jobs a second, and 5 jobs a second would be a lot for our cloud instance.
A schedule job that runs every 3 minutes has the message: Queued Queue position: 341
Is there anything I can do about this?
Jobs are falling into Canceled status, or coming up with a warning message:
Time spent waiting for an executor: 108.449s
This job spent an unexpected amount of time waiting for a worker before starting.
In a healthy queue, jobs are expected to start in under 50ms.
I know that we are must share resources with other workspaces, but we had over 500 jobs in canceled status over the morning (6 hours from 7am through to 1pm)
We generally have less than 3 jobs a second, and 5 jobs a second would be a lot for our cloud instance.
A schedule job that runs every 3 minutes has the message: Queued Queue position: 341
Is there anything I can do about this?
