Windmill Server Stuck On Boot [ docker ]
After a reboot to caprover, the windmill server stopped working (can't open page). Workers keep working fine.
It gets stuck after deleting tokens from the DB.
15 Replies
Log from docker container
I'm stumped with this 😅
I created a new PostgreSQL docker container, and windmill spawns alright. Looks like it is a DB issue, but can't put my finger on it.
See this: https://github.com/windmill-labs/windmill/issues/3233 maybe you're out of disk space? Caprover doesn't delete older images by default and on each upgrade they take a lot of disk space.
GitHub
bug: workers servers not migrating · Issue #3233 · windmill-labs/wi...
Describe the bug Happy user here. Just updated my docker image to the latest version (1.270). I'm only seeing the following messages: Started locking PG for migration purposes Locked PG for mig...
I have removed all the dangling images
Will double check, but pretty sure I had 20GB free, at least
Another user recently noticed that the migration can take hours https://discord.com/channels/930051556043276338/930051556043276341/1207355036250804295
Can confirm, have 26GB free
When trying to perform a pg_dump, it gets stuck in "dumping contents of table 'public.completed_job'".
Maybe some data corruption on that table?
Or it's huge? I don't know PG that much.
The backup size doesn't increase... It ended up throwing a "no space left on device", but don't understand why.. since the docker host still has 20+GB free..
What's the size of the database? Maybe it's bigger than that much
Yeah, that was it
Sorted it.. had to manually delete the records on the job table..
Then vaccum full the database
Started windmill, all good
Great!
Had 38gb of data on that one
How does retention time work? Windmill purges data after that time?
The jobs, yes.
So, ideally, the database shouldn't increase that much if I set it to like, 1 day, right?
Or does one need to enable vacuum in the postgres instance?
I'm not sure