Alex Nunes
Alex Nunes•9mo ago

Windmill Server Stuck On Boot [ docker ]

After a reboot to caprover, the windmill server stopped working (can't open page). Workers keep working fine. It gets stuck after deleting tokens from the DB.
15 Replies
Alex Nunes
Alex NunesOP•9mo ago
Log from docker container
Alex Nunes
Alex NunesOP•9mo ago
I'm stumped with this 😅 I created a new PostgreSQL docker container, and windmill spawns alright. Looks like it is a DB issue, but can't put my finger on it.
Tiago Serafim
Tiago Serafim•9mo ago
See this: https://github.com/windmill-labs/windmill/issues/3233 maybe you're out of disk space? Caprover doesn't delete older images by default and on each upgrade they take a lot of disk space.
GitHub
bug: workers servers not migrating · Issue #3233 · windmill-labs/wi...
Describe the bug Happy user here. Just updated my docker image to the latest version (1.270). I'm only seeing the following messages: Started locking PG for migration purposes Locked PG for mig...
Alex Nunes
Alex NunesOP•9mo ago
I have removed all the dangling images Will double check, but pretty sure I had 20GB free, at least
Tiago Serafim
Tiago Serafim•9mo ago
Another user recently noticed that the migration can take hours https://discord.com/channels/930051556043276338/930051556043276341/1207355036250804295
Alex Nunes
Alex NunesOP•9mo ago
Can confirm, have 26GB free When trying to perform a pg_dump, it gets stuck in "dumping contents of table 'public.completed_job'". Maybe some data corruption on that table?
Tiago Serafim
Tiago Serafim•9mo ago
Or it's huge? I don't know PG that much.
Alex Nunes
Alex NunesOP•9mo ago
The backup size doesn't increase... It ended up throwing a "no space left on device", but don't understand why.. since the docker host still has 20+GB free..
Tiago Serafim
Tiago Serafim•9mo ago
What's the size of the database? Maybe it's bigger than that much
Alex Nunes
Alex NunesOP•9mo ago
Yeah, that was it Sorted it.. had to manually delete the records on the job table.. Then vaccum full the database Started windmill, all good
Tiago Serafim
Tiago Serafim•9mo ago
Great!
Alex Nunes
Alex NunesOP•9mo ago
Had 38gb of data on that one How does retention time work? Windmill purges data after that time?
Tiago Serafim
Tiago Serafim•9mo ago
The jobs, yes.
Alex Nunes
Alex NunesOP•9mo ago
So, ideally, the database shouldn't increase that much if I set it to like, 1 day, right? Or does one need to enable vacuum in the postgres instance?
Tiago Serafim
Tiago Serafim•9mo ago
I'm not sure