Error Ducklake: Writing parquet file fails.
ExecutionErr: execution error:
"HTTP Error: Unable to connect to URL http://localhost:36867/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-01997aa2-1513-75ee-a66e-ff5a6e3d9aa3.parquet?uploads=: Forbidden (HTTP code 403)"
Metadata within Postgres (Neon) looks good, S3 Browser up- and download looks good. Using a stand alone DuckDB / Ducklake with same S3 and Postgres infrastructure works just fine, so I think, something is wrong with my Windmill configuration.
Has anybody had similar problems? Does the s3_proxy translate "default"? Do I have to set special permissions? I have tried legacy permissions and new advanced permissions.
Can somebody give me guidance on how to diagnose the issue further? I am stuck, therefore any help is appreciated. Thank you.
I am running self hosted docker version v1.547 with EE image and license on a Windows machine.
6 Replies
you're using azure storage ?
What does the backend logs says ? There should be more info on the 403 there
DuckDB has issues with Azure, therefore we are using S3 on Cloudflare.
Backend log = Worker log? There are obvious no errors. Do I have to set a log level?
2025-09-24T15:26:11.367005Z INFO windmill-common/src/ee.rs:1286: disk stats for "wk-default-58dc978a819b-f1SpF": "/" - 964.8 GB,"/tmp/windmill/logs" - 964.8 GB
2025-09-24T15:26:15.854059Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:21.910671Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=31MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:27.976075Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:34.039443Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:40.081071Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:41.369825Z INFO windmill-common/src/ee.rs:1286: disk stats for "wk-default-58dc978a819b-f1SpF": "/" - 964.8 GB,"/tmp/windmill/logs" - 964.8 GB
2025-09-24T15:26:46.154590Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:52.219735Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1SpF hostname=58dc978a819b
2025-09-24T15:26:58.288481Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=30MB, windmill=21MB worker=wk-default-58dc978a819b-f1Sp
No, I was indeed talking about backend logs
The worker queries the S3 Proxy running on the backend
(I apologize for the unhelpful error message, I will be looking for a way to show nicer errors from S3 Proxy when I have time)
You can see those logs in Logs > Service Logs > Server if you have the right permissions
