@Trey yes but not very priority. But you can do a lot with bash directly

T

Trey•10/16/25, 4:53 PM

ok thanks!

Rrubenf @Trey yes but not very priority. But you can do a lot with bash directly

P

Pyra•10/16/25, 4:54 PM

or nu

TTrey ok thanks!

F

fjørdlek•10/16/25, 5:00 PM

specifically with the resource type "file"

R

rubberducky•10/16/25, 5:32 PM

what s the best way to debug a flow / script stuck in a deployment state?

R

rubenfOP•10/16/25, 5:33 PM

look for the associated dependency job

T

tl_jacob•10/16/25, 6:23 PM

@rubenf I'm seeing the following migration error when upgrading from v1.554.0 to v1.561.0

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

2025-10-16T18:12:51.094722Z  INFO windmill-api/src/db.rs:187: Finished applying migration 20251006143822
Error: Error migrating db: Migrating database: while executing migration 20251006143822: error returned from database: null value in column "value" of relation "global_settings" violates not-null constraint

Checking my db I don't see any null settings values

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

windmill=# SELECT * from global_settings where value is NULL;
 name | value | updated_at
------+-------+------------

Which points me to the migration might be the thing that is trying to insert a null value: https://github.com/windmill-labs/windmill/blob/6dfa4ff5ba26bc4912b1be1a9709728cf6c2d119/backend/migrations/20251006143822_ducklake_instance_settings_better.down.sql#L3. I don't have ducklake settings

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

windmill=# select name from global_settings;
         name
-----------------------
 jwt_secret
 dev_instance
 retention_period_secs
 disable_stats
 job_default_timeout
 oauths
 base_url
 license_key

R

rubenfOP•10/16/25, 6:25 PM

@Diego looks to be your migration:

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

INSERT INTO global_settings (name, value) VALUES (
  'ducklake_settings',
  (SELECT json_build_object(
    'ducklake_user_pg_pwd', g2.value,
    'instance_catalog_db_status', '{}'::json
  ) FROM global_settings g2 WHERE g2.name = 'ducklake_user_pg_pwd')
);

D

Diego•10/16/25, 6:27 PM

hm

Ttl_jacob @rubenf I'm seeing the following migration error when upgrading from v1.554.0 to...

D

Diego•10/16/25, 6:35 PM

Can you try to run this migration by hand

https://github.com/windmill-labs/windmill/blob/6dfa4ff5ba26bc4912b1be1a9709728cf6c2d119/backend/migrations/20250731132157_ducklake_instance_settings.up.sql

GitHub

windmill/backend/migrations/20250731132157_ducklake_instance_settin...

Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal...

D

Diego•10/16/25, 6:36 PM

Somehow ducklake_user_pg_pwd does not exist

DDiego Can you try to run this migration by hand https://github.com/windmill-labs/wind...

T

tl_jacob•10/16/25, 6:44 PM

okay that fixed the issue, I'm wondering how my windmill db got in that state though since I never run any manual migrations. Did I miss an upgrade message about needing to run a manual migration?

D

Diego•10/16/25, 6:45 PM

no this should have been run automatically liked other migrations

D

Diego•10/16/25, 6:45 PM

i don't understand either

D

Diego•10/16/25, 6:45 PM

@rubenf if you have an idea how it's possible that a migration got skipped

R

rubenfOP•10/16/25, 6:45 PM

Not possible

D

Diego•10/16/25, 6:46 PM

well it's magic

D

Diego•10/16/25, 6:46 PM

never happened before

Rrubenf look for the associated dependency job

R

rubberducky•10/16/25, 7:18 PM

sorry im not seeing any issues with depedency jobs eitehr

Rrubberducky sorry im not seeing any issues with depedency jobs eitehr

P

Pyra•10/16/25, 7:36 PM

Is it just having infinite spinner e.g. deployment in progress? Are lockfiles generated?

If there is no locks there is likely a djob somewhere still running.

Did you try to redeploy?

R

rubberducky•10/16/25, 7:36 PM

I actually wiped my entire windmill instance and did a fresh push

R

rubberducky•10/16/25, 7:37 PM

" Deployment in progress" on the flow page

P

Pyra•10/16/25, 7:38 PM

What version are you on? The minimal reproduction is to just create a flow and deploy it?

R

rubberducky•10/16/25, 7:38 PM

1.555.1

PPyra What version are you on? The minimal reproduction is to just create a flow and d...

P

Pyra•10/16/25, 7:39 PM

Any specific actions? Imports?

P

Pyra•10/16/25, 7:41 PM

And what is your infra? k8s? Docker? Native?

Ttl_jacob okay that fixed the issue, I'm wondering how my windmill db got in that state th...

D

Diego•10/16/25, 7:42 PM

Did you ever do something to your global settings that might have deleted it ?

D

Diego•10/16/25, 7:42 PM

I'm going through the code and assuming the migration ran, I don't see how this was possible

PPyra And what is your infra? k8s? Docker? Native?

R

rubberducky•10/16/25, 7:42 PM

right now a local docker running

R

rubberducky•10/16/25, 7:43 PM

its a rather complicated flow, all python is checked and valid with tests and i can confirm they work.

R

rubberducky•10/16/25, 7:43 PM

I can view all scripts etc on the frontend

DDiego Did you ever do something to your global settings that might have deleted it ?

T

tl_jacob•10/16/25, 7:44 PM

I've been setting instance-level settings using a

instance_settings.yaml

instance_settings.yaml

instance_settings.yaml

instance_settings.yaml that doesn't have any ducklake settings

D

Diego•10/16/25, 7:44 PM

oh

D

Diego•10/16/25, 7:45 PM

there we go

thanks

T

tl_jacob•10/16/25, 7:47 PM

@Diego just to be clear I do feel like the way this has broken is not an optimal customer/developer experience. I don't think upgrading versions should break in this way.

D

Diego•10/16/25, 7:47 PM

yes it's a bug

Rrubberducky its a rather complicated flow, all python is checked and valid with tests and i ...

P

Pyra•10/16/25, 7:47 PM

Could it just be locking things? Could happen if you have poor connectivity.

But if it is complicated flow, I think it is the best to gradually reduce until it works

T

tl_jacob•10/16/25, 7:48 PM

@Diego gotcha! happy to be a point of contact on the fix if that's helpful

really appreciate how you + ruben have hopped in here so quickly

PPyra Could it just be locking things? Could happen if you have poor connectivity. Bu...

R

rubberducky•10/16/25, 7:51 PM

ill refactor it and start with each step until it breaks somewhere

P

Pyra•10/16/25, 7:52 PM

Yeah. But you should really check dependency jobs. You can filter by that job kind in run page

P

Pyra•10/16/25, 7:52 PM

I just think you have something heavy or it is somehow stuck on a a dependency

R

rubberducky•10/16/25, 7:53 PM

its a local instance, no jobs running except the deployment

P

Pyra•10/16/25, 7:54 PM

by heavy I mean you have hundreds of dependencies (including transitive) and your instance didn't finish fetching all necessarily metadata

R

rubberducky•10/16/25, 7:58 PM

they all finished, is there a chance one may have not?

P

Pyra•10/16/25, 8:11 PM

If it is a single flow and there is no relative imports, it would be a single dependency job. If that dependency job has finished then the flow should be viewable as deployed

M

metapontum•10/17/25, 11:02 AM

This is how simple it could be to interact with wmill storage / ducklake

M

metapontum•10/17/25, 11:04 AM

https://bun.com/blog/bun-v1.3

Bun

Bun 1.3

Bun 1.3 introduces zero-config frontend development, unified SQL API, built-in Redis client, security enhancements, package catalogs, async stack traces, VS Code test integration, and Node.js compatibility improvements.

M

metapontum•10/17/25, 11:16 AM

Very impressive from bun, when I was reading it I kept thinking that these can be made to interoperate with windmill by default

M

metapontum•10/17/25, 11:21 AM

Resources ≈ redis KV (types can be specified inline, defaulting to ‘state’ _resource_type: ‘s3’)
Variables ≈ secrets
Storage ≈ s3
* Ducklake ≈ sql

R

rubenfOP•10/17/25, 11:44 AM

Agreed @metapontum , we will probably want a simplified syntax, maybe a one function call in our sdk to setup the s3 client

Z

Zach•10/17/25, 3:01 PM

Have you thought about Bun to build/serve the raw apps, @rubenf? (I’ve actually been using raw apps to iframe to a subdomain with a cookie.) Was reading through some of the raw apps branch and was just curious?