look for the associated dependency job

T

tl_jacob•10/16/25, 6:23 PM

@rubenf I'm seeing the following migration error when upgrading from v1.554.0 to v1.561.0

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

Checking my db I don't see any null settings values

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

Which points me to the migration might be the thing that is trying to insert a null value: https://github.com/windmill-labs/windmill/blob/6dfa4ff5ba26bc4912b1be1a9709728cf6c2d119/backend/migrations/20251006143822_ducklake_instance_settings_better.down.sql#L3. I don't have ducklake settings

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

R

rubenfOP•10/16/25, 6:25 PM

@Diego looks to be your migration:

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

D

Diego•10/16/25, 6:27 PM

hm

Ttl_jacob @rubenf I'm seeing the following migration error when upgrading from v1.554.0 to...

D

Diego•10/16/25, 6:35 PM

Can you try to run this migration by hand

https://github.com/windmill-labs/windmill/blob/6dfa4ff5ba26bc4912b1be1a9709728cf6c2d119/backend/migrations/20250731132157_ducklake_instance_settings.up.sql

GitHub

windmill/backend/migrations/20250731132157_ducklake_instance_settin...

Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal...

D

Diego•10/16/25, 6:36 PM

Somehow ducklake_user_pg_pwd does not exist

DDiego Can you try to run this migration by hand https://github.com/windmill-labs/wind...

T

tl_jacob•10/16/25, 6:44 PM

okay that fixed the issue, I'm wondering how my windmill db got in that state though since I never run any manual migrations. Did I miss an upgrade message about needing to run a manual migration?

D

Diego•10/16/25, 6:45 PM

no this should have been run automatically liked other migrations

D

Diego•10/16/25, 6:45 PM

i don't understand either

D

Diego•10/16/25, 6:45 PM

@rubenf if you have an idea how it's possible that a migration got skipped

R

rubenfOP•10/16/25, 6:45 PM

Not possible

D

Diego•10/16/25, 6:46 PM

well it's magic

D

Diego•10/16/25, 6:46 PM

never happened before

Rrubenf look for the associated dependency job

R

rubberducky•10/16/25, 7:18 PM

sorry im not seeing any issues with depedency jobs eitehr

Rrubberducky sorry im not seeing any issues with depedency jobs eitehr

P

Pyra•10/16/25, 7:36 PM

Is it just having infinite spinner e.g. deployment in progress? Are lockfiles generated?

If there is no locks there is likely a djob somewhere still running.

Did you try to redeploy?

R

rubberducky•10/16/25, 7:36 PM

I actually wiped my entire windmill instance and did a fresh push

R

rubberducky•10/16/25, 7:37 PM

" Deployment in progress" on the flow page

P

Pyra•10/16/25, 7:38 PM

What version are you on? The minimal reproduction is to just create a flow and deploy it?

R

rubberducky•10/16/25, 7:38 PM

1.555.1

PPyra What version are you on? The minimal reproduction is to just create a flow and d...

P

Pyra•10/16/25, 7:39 PM

Any specific actions? Imports?

P

Pyra•10/16/25, 7:41 PM

And what is your infra? k8s? Docker? Native?

Ttl_jacob okay that fixed the issue, I'm wondering how my windmill db got in that state th...

D

Diego•10/16/25, 7:42 PM

Did you ever do something to your global settings that might have deleted it ?

D

Diego•10/16/25, 7:42 PM

I'm going through the code and assuming the migration ran, I don't see how this was possible

PPyra And what is your infra? k8s? Docker? Native?

R

rubberducky•10/16/25, 7:42 PM

right now a local docker running

R

rubberducky•10/16/25, 7:43 PM

its a rather complicated flow, all python is checked and valid with tests and i can confirm they work.

R

rubberducky•10/16/25, 7:43 PM

I can view all scripts etc on the frontend

DDiego Did you ever do something to your global settings that might have deleted it ?

T

tl_jacob•10/16/25, 7:44 PM

I've been setting instance-level settings using a

instance_settings.yaml

instance_settings.yaml

that doesn't have any ducklake settings

D

Diego•10/16/25, 7:44 PM

oh

D

Diego•10/16/25, 7:45 PM

there we go

thanks

T

tl_jacob•10/16/25, 7:47 PM

@Diego just to be clear I do feel like the way this has broken is not an optimal customer/developer experience. I don't think upgrading versions should break in this way.

D

Diego•10/16/25, 7:47 PM

yes it's a bug

Rrubberducky its a rather complicated flow, all python is checked and valid with tests and i ...

P

Pyra•10/16/25, 7:47 PM

Could it just be locking things? Could happen if you have poor connectivity.

But if it is complicated flow, I think it is the best to gradually reduce until it works

T

tl_jacob•10/16/25, 7:48 PM

@Diego gotcha! happy to be a point of contact on the fix if that's helpful

really appreciate how you + ruben have hopped in here so quickly

PPyra Could it just be locking things? Could happen if you have poor connectivity. Bu...

R

rubberducky•10/16/25, 7:51 PM

ill refactor it and start with each step until it breaks somewhere

P

Pyra•10/16/25, 7:52 PM

Yeah. But you should really check dependency jobs. You can filter by that job kind in run page

P

Pyra•10/16/25, 7:52 PM

I just think you have something heavy or it is somehow stuck on a a dependency

R

rubberducky•10/16/25, 7:53 PM

its a local instance, no jobs running except the deployment

P

Pyra•10/16/25, 7:54 PM

by heavy I mean you have hundreds of dependencies (including transitive) and your instance didn't finish fetching all necessarily metadata

R

rubberducky•10/16/25, 7:58 PM

they all finished, is there a chance one may have not?

P

Pyra•10/16/25, 8:11 PM

If it is a single flow and there is no relative imports, it would be a single dependency job. If that dependency job has finished then the flow should be viewable as deployed

M

metapontum•10/17/25, 11:02 AM

This is how simple it could be to interact with wmill storage / ducklake

M

metapontum•10/17/25, 11:04 AM

https://bun.com/blog/bun-v1.3

Bun

Bun 1.3

Bun 1.3 introduces zero-config frontend development, unified SQL API, built-in Redis client, security enhancements, package catalogs, async stack traces, VS Code test integration, and Node.js compatibility improvements.

M

metapontum•10/17/25, 11:16 AM

Very impressive from bun, when I was reading it I kept thinking that these can be made to interoperate with windmill by default

M

metapontum•10/17/25, 11:21 AM

Resources ≈ redis KV (types can be specified inline, defaulting to ‘state’ _resource_type: ‘s3’)
Variables ≈ secrets
Storage ≈ s3
* Ducklake ≈ sql

R

rubenfOP•10/17/25, 11:44 AM

Agreed @metapontum , we will probably want a simplified syntax, maybe a one function call in our sdk to setup the s3 client

Z

Zach•10/17/25, 3:01 PM

Have you thought about Bun to build/serve the raw apps, @rubenf? (I’ve actually been using raw apps to iframe to a subdomain with a cookie.) Was reading through some of the raw apps branch and was just curious?

R

rubenfOP•10/17/25, 3:07 PM

@Zach raw apps are purely static frontend rendering + windmill jobs for background runnables

R

rubenfOP•10/17/25, 3:47 PM

https://youtube.com/live/mrnmNgQ22jE?feature=share weekly!

YouTubeWindmill

Windmill Weekly 17th October, 2025

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Rrubenf https://youtube.com/live/mrnmNgQ22jE?feature=share weekly!

K

Kelora•10/17/25, 4:36 PM

excited for the ui polish, wanted to suggest some attention on this tiny papercut I run into a lot. Resizing these panes can be hard especially on high resolution/remote desktop.

R

rubenfOP•10/17/25, 4:56 PM

at very least we could make it bigger on hover I suppose

K

Kelora•10/17/25, 4:57 PM

i dont think it needs to be visibly bigger, but a few more pixels of "grip" would help a ton