fr3fou
fr3fou
PostsComments
WWindmill
Created by fr3fou on 11/3/2024 in #help
Relative imported module doesn't respect optional/non-imported Python dependencies
I have the following file in f/main/requirements.txt : 
duckdb==1.1.2
polars[pyarrow]==1.12.0
duckdb==1.1.2
polars[pyarrow]==1.12.0
I also have a script called f/main/get_db, which only imports duckdb: 
from typing import TypedDict
import duckdb
import textwrap
...
from typing import TypedDict
import duckdb
import textwrap
...
Running wmill script generate-metadata produces a script.lockfile for the get_db script correctly: 
duckdb==1.1.2
polars==1.12.0
pyarrow==18.0.0
duckdb==1.1.2
polars==1.12.0
pyarrow==18.0.0
(the pyarrow & polars dependencies are required if you want to convert duckdb results to a polars dataframe) In my flow, f/main/my_flow, inside an inline script, I import the get_db module: 
from typing import TypedDict
from f.main.get_db import get_db
from typing import TypedDict
from f.main.get_db import get_db
The generated lockfile, however, doesn't include the optional polars & pyarrow dependencies: 
duckdb==1.1.2
duckdb==1.1.2
I've tried running wmill script generate-metadata and wmill flow generate-locks, but none of them seem to produce the desired output. A workaround is for me to include the pyarrow / polars dependencies in a comment in my inline script (under a extra_requirements section), but this means that I have to pin 2 versions separately which is undesired.
2 replies
WWindmill
Created by fr3fou on 10/31/2024 in #help
What's the recommended way to run dbt transformations?
Currently migrating from a Prefect pipeline which uses https://github.com/datarootsio/prefect-dbt-flow and was wondering how I can run my dbt models in Windmill. This library automatically makes a new Task/Node in the DAG for each dbt model. I'm not really sure what the best way is to pull your dbt files (.sql models & yaml configurations) into Windmill, as well as running them using the dbt CLI. One way I thought of is to package my dbt models (and dbt CLI tool) in a Docker image, which I would then pull from a private registry inside Windmill. This, however, won't handle the breaking-down-into-multiple-tasks (scripts in Windmill) part. Any ideas? I'm also considering trying sqlmesh but I've never used it, but I think it has a similar "deployment" model.
2 replies
WWindmill
Created by fr3fou on 10/30/2024 in #help
Dev/Prod envs using 2 instances (Community Edition)
Would I encounter any issues if I have 2 GitHub branches: - dev - master and use wmill sync push in a GitHub Action [with 2 different domains / tokens]?
16 replies
WWindmill
Created by fr3fou on 7/22/2024 in #help
`ApiError: Internal Server Error` when writing a 2MB csv file
Using the wmill.writeS3File function to write a small csv file from ~20k lines and it seems to give this error. 
ApiError: Internal Server Error
ApiError: Internal Server Error
    at catchErrorCodes (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:224:15)
    at file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:269:45
    at Generator.next (<anonymous>)
    at fulfilled (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:5:58)
    at eventLoopTick (ext:core/01_core.js:168:7)
ApiError: Internal Server Error
ApiError: Internal Server Error
    at catchErrorCodes (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:224:15)
    at file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:269:45
    at Generator.next (<anonymous>)
    at fulfilled (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:5:58)
    at eventLoopTick (ext:core/01_core.js:168:7)
If I try the same script with less data (thus making the file smaller), e.g. 5k csv lines it works fine. Upping it to 10k (it's ~1MB then), it still crashes. I'm also confident that it's not the serialization that's failing and it is indeed the call to writeS3File. Running CE v1.365.0, self hosted, Deno TypeScript, Self Hosted Minio
3 replies
WWindmill
Created by fr3fou on 10/9/2023 in #help
Always-on scripts/flows?
Is there a way to have some HTTP server / blocking function to run and never stop?
2 replies
WWindmill
Created by fr3fou on 10/7/2023 in #help
Nested for loops?
No description
2 replies