Schedule issue due to DST change

R

rubenf•3/11/25, 11:40 AM

Hi,

We’ve identified a potential issue with some rare jobs seemingly stuck and some rare schedules not triggering their next tick following the recent Daylight Saving Time (DST) change on March 9th. You're likely not affected or the issue already resolved itself as it only require a server restart. Out of transparency and for extra safety, you will want to still do the check. Below, you'll find details on how to detect if you're affected, how to fix it, and a technical explanation of what happened.

How to Fix It

Restart at least one server/app.

How to Check If You're Affected

Run the following query:

SELECT q.id, q.workspace_id, r.ping, COALESCE(zjc.counter, 0) as counter
FROM v2_job_queue q
JOIN v2_job j ON j.id = q.id
JOIN v2_job_runtime r ON r.id = j.id
LEFT JOIN zombie_job_counter zjc ON zjc.job_id = q.id
WHERE r.ping < now() - INTERVAL '30 seconds'
    AND q.running = true
    AND j.kind NOT IN ('flow', 'flowpreview', 'flownode', 'singlescriptflow')  -- j.kind from v2_job
    AND (zjc.counter IS NULL OR zjc.counter <= 3);

SELECT q.id, q.workspace_id, r.ping, COALESCE(zjc.counter, 0) as counter
FROM v2_job_queue q
JOIN v2_job j ON j.id = q.id
JOIN v2_job_runtime r ON r.id = j.id
LEFT JOIN zombie_job_counter zjc ON zjc.job_id = q.id
WHERE r.ping < now() - INTERVAL '30 seconds'
    AND q.running = true
    AND j.kind NOT IN ('flow', 'flowpreview', 'flownode', 'singlescriptflow')  -- j.kind from v2_job
    AND (zjc.counter IS NULL OR zjc.counter <= 3);

SELECT q.id, q.workspace_id, r.ping, COALESCE(zjc.counter, 0) as counter
FROM v2_job_queue q
JOIN v2_job j ON j.id = q.id
JOIN v2_job_runtime r ON r.id = j.id
LEFT JOIN zombie_job_counter zjc ON zjc.job_id = q.id
WHERE r.ping < now() - INTERVAL '30 seconds'
    AND q.running = true
    AND j.kind NOT IN ('flow', 'flowpreview', 'flownode', 'singlescriptflow')  -- j.kind from v2_job
    AND (zjc.counter IS NULL OR zjc.counter <= 3);

SELECT q.id, q.workspace_id, r.ping, COALESCE(zjc.counter, 0) as counter
FROM v2_job_queue q
JOIN v2_job j ON j.id = q.id
JOIN v2_job_runtime r ON r.id = j.id
LEFT JOIN zombie_job_counter zjc ON zjc.job_id = q.id
WHERE r.ping < now() - INTERVAL '30 seconds'
    AND q.running = true
    AND j.kind NOT IN ('flow', 'flowpreview', 'flownode', 'singlescriptflow')  -- j.kind from v2_job
    AND (zjc.counter IS NULL OR zjc.counter <= 3);

If you see jobs with a last ping older than 1 day, you are affected.

What Happened?

Windmill is designed with multiple layers of error detection and fail-safes to handle various issues, from job errors to network failures and worker crashes. Our approach ensures:

Explicit error handling (e.g., in Rust, we avoid unwrap() or expect() to prevent panics). Transactional job progression, where workers regularly ping to indicate progress.

However, during the recent DST change, a bug in a scheduling library caused an issue. The library failed to compute the next tick for jobs scheduled between 2 AM and 3 AM, treating it as a non-existent time. Our code, assuming continuous time progression, triggered a panic when attempting to schedule the next tick.

Impact

Workers could not complete their jobs because they panicked when scheduling the next tick.
Our monitoring system detected job stalls, attempted to compute the next tick, and also panicked if the tick fell within the affected DST period.
Tokio, the concurrency library we use, does not handle panics well in this case—it killed only the monitor task instead of the entire application.

Why Restarting a Server Fixes It

Restarting a server reinitializes the monitor task, allowing it to detect and recover stuck jobs. This restores normal scheduling behavior.

We sincerely apologize for any disruptions this may have caused. We're actively working with the library maintainer to resolve the underlying issue and will implement further safeguards to prevent similar incidents in the future.

Thank you for your understanding,
The Windmill Team

F

fr3fou•3/11/25, 12:16 PM

Thank you for the transparency and quick turnaround. I was wondering if you've considered publishing such "PSA"s to other channels, not only Discord. I have coworkers that aren't on Discord and would like to be up-to-date on critical issues [there was another incident with some faulty update several months ago that required a hot-fix update].
It would be nice if there was an email newsletter/posting on GitHub discussion announcements/in-app-notifications, similar to the changelogs part [which I think hasn't been updated in a while]. Would love to hear other ideas as well

Ffr3fou Thank you for the transparency and quick turnaround. I was wondering if you've c...

R

rubenf•3/11/25, 12:33 PM

we announce it to every customers on their dedicated channel & emails + give them precise instructions based on the data we have, for community at the moment it's on discord, we will see for the future. Github discussions seems nice