Experiencing slow to start scripts
We have started to have an issue in our self hosted env where there is a bit of a random slowness that has started. Its not every run, but the vast majority where scripts will take 60+ seconds to start, when they start the execution time is the usual 1s. E.g. run a flow,
script 1: 70 seconds,
script 2: 60 seconds,
script 3: 60 seconds.
Run the flow again,
script 1: 1 second
script 2: 65 seconds
script 3: 1 second
We are not sure where to find logs to work out where the issue lies, it feels like the jobs are sitting and waiting for the workers to pick them up, but cant find any evidence to support this. Can anyone point us to where we can find the logs for this so we can work out what we need to do with our workers? It is worth noting that nothing else is running at the time we are starting the scripts/flows.
6 Replies
you can look at your worker and server logs
There is probably something wrong with your setup because you're the first one to report such issue
Yeah we expect its something specific to us, as this was running perfectly. But it just started a couple weeks ago.
Are the logs just in /var/logs ?
they're emitted in stdout so it depends how your container orchestrator handle the logs
That will make it much easier to check, will have a look at the containers and see if there is some indications
Thank you for the help
the logs of the jobs themselves are not emitted in stdout but if you say it's the time to start, they won't be useful
Looking within Windmill itself, its as if nothing happens for the 60+ seconds. Then you get output from the script and it executes in its usual time. Its just a delay before they actually start