zoro
zoro2d ago

Error in DB Migration: v2 migrate from v1

I have decided to upgrade one of the worker groups and apply the new migrations but getting this error constantly: 2025-02-21T06:16:12.576439429Z 2025-02-21T06:16:12.575768Z INFO windmill-api/src/db.rs:93: Acquired global PG lock 2025-02-21T06:16:12.587240719Z 2025-02-21T06:16:12.587029Z INFO windmill-api/src/db.rs:130: Started applying migration 20250201124748: v2 migrate from v1 2025-02-21T06:18:32.191178681Z 2025-02-21T06:18:32.190955Z INFO windmill-api/src/db.rs:156: Finished applying migration 20250201124748 2025-02-21T06:18:32.232113713Z Error: Error migrating db: Migrating database: while executing migration 20250201124748: error returned from database: value too long for type character varying(50)
29 Replies
zoro
zoroOP2d ago
What should I check or attempt to rectify this?
rubenf
rubenf2d ago
GitHub
bug: Migration from v1 to v2 fails · Issue #5345 · windmill-labs/wi...
Describe the bug Updating the image on one of the worker groups started the migration process. Once it got to the v1 to v2 migration it fails after a few minutes and then the container terminates a...
rubenf
rubenf2d ago
curious how come your tags were so long, it would usually happen for dedicated workers
zoro
zoroOP2d ago
Cool. Let me check. Thanks @rubenf Interesting. I found this tag in the queue: "playground:flow/f/sql_flows/eflow_exceptions_investigate_byhubid_to_dwh"
rubenf
rubenf2d ago
it's a dedicated worker job
zoro
zoroOP2d ago
Looks like db migration has been completed but there is also an error from python:
rubenf
rubenf2d ago
you're not using our images are you?
zoro
zoroOP2d ago
No description
rubenf
rubenf2d ago
are you in a very tight environment where it wouldn't be able to download python ?
zoro
zoroOP2d ago
Nope. Running a docker swarm with 5 nodes that have full outbound access to the internet
rubenf
rubenf2d ago
investigating
zoro
zoroOP2d ago
Let me know if I can assist in anyway
rubenf
rubenf2d ago
what are your volume mounts?
zoro
zoroOP2d ago
For the workers I have this mount setup on each docker node which points to the same dedicate share:
No description
zoro
zoroOP2d ago
The same is for the worker dependncy cache and lsp cache and index
zoro
zoroOP2d ago
No description
zoro
zoroOP2d ago
@ramkubcp1:~$ docker volume inspect windmill_worker_logs_prod [ { "CreatedAt": "2024-09-17T13:48:11Z", "Driver": "local", "Labels": null, "Mountpoint": "/var/lib/docker/volumes/windmill_worker_logs_prod/_data", "Name": "windmill_worker_logs_prod", "Options": { "device": "//10.0.20.210/worker_logs", "o": "addr=10.0.20.210,username=windmill.service,password=***,vers=3.0", "type": "cifs" }, "Scope": "local" } ]
rubenf
rubenf2d ago
somehow your filesystem doesn't support symlinking
zoro
zoroOP2d ago
Odd. Is this something new in the newer images?
rubenf
rubenf2d ago
it uses uv which install python instead of relying on the system python but uv to install python require symlinking which your system doesn't seem to support for some reasons remove the windmill_worker_dependency_cache_prod volumes for now
zoro
zoroOP2d ago
Would it require the containers to run in priveledged mode?
rubenf
rubenf2d ago
no
zoro
zoroOP2d ago
Let me try
rubenf
rubenf2d ago
the error is this one:
2025-02-21T07:26:58.419929200Z Caused by: Failed to extract archive: cpython-3.11.11%2B20250106-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz

2025-02-21T07:26:58.419935700Z Caused by: failed to unpack `/tmp/windmill/cache/py_runtime/.temp/.tmpfuKQX6/python/bin/2to3`

2025-02-21T07:26:58.419959200Z Caused by: Operation not supported (os error 95) when symlinking 2to3-3.11 to /tmp/windmill/cache/py_runtime/.temp/.tmpfuKQX6/python/bin/2to3
2025-02-21T07:26:58.419929200Z Caused by: Failed to extract archive: cpython-3.11.11%2B20250106-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz

2025-02-21T07:26:58.419935700Z Caused by: failed to unpack `/tmp/windmill/cache/py_runtime/.temp/.tmpfuKQX6/python/bin/2to3`

2025-02-21T07:26:58.419959200Z Caused by: Operation not supported (os error 95) when symlinking 2to3-3.11 to /tmp/windmill/cache/py_runtime/.temp/.tmpfuKQX6/python/bin/2to3
zoro
zoroOP2d ago
ok removing the cache mount seems to resolve the error How will this affecting the caching on the scripts?
rubenf
rubenf2d ago
yes, they won't be persistent across worker restarts
zoro
zoroOP2d ago
ok cool I can live with that 👍 All looks good. Thanks Thank you for your time @rubenf I appreciate it
rubenf
rubenfthis hour
sorry for the trouble, i'm still not too sure how the symlink error happens
zoro
zoroOPthis hour
No worries. Just glad to be up and running again with Windmill 🙂

Did you find this page helpful?