@Kelora not yet but we will have such api in the future as we want to make it easy to build your own

@Kelora not yet but we will have such api in the future as we want to make it easy to build your own workers in any sdk and maybe deploy some workers on cloudfare workers/lambda etc

rubenfOP•6/4/24, 7:23 AM

We also need to implement the sdk for pinging the db while executing jobs so that those can process jobs longer than 30s without us considering it dead

Kelora•6/4/24, 7:36 AM

ok good to hear

really like what im seeing so far, doing some trials to see how windmill can fit our needs

Rrubenf @Kelora not yet but we will have such api in the future as we want to make it ea...

fr3fou•6/4/24, 11:08 AM

will this be available in OSS? seems incredibly useful and would make it very comparable with something like Prefect

rubenfOP•6/4/24, 11:15 AM

debatable, it would be competitive with our normal offering that is per vCPU, we will see when we get there, we might introduce a pricing per event for those

Joldz 🔮•6/4/24, 2:54 PM

In the documentation I see reference to a Table component: https://www.windmill.dev/docs/apps/app_configuration_settings/table
but I don't see an option for a component just titled "Table" in the list of components in the App Editor. is a simple table display no longer an option or is there something I'm missing? I did notice there was a TanStack Table component that looks similar to the Table in the docs, but I see it says it is deprecated in the App Editor and that we should use the AgGrid Table instead. Is that just what is recommended for simple table display now?

Table | Windmill

The table component allows you to display a table.

nsadeh•6/4/24, 7:42 PM

Following up on my question from above: I have a docker image in a protected container registry. My docker on my host machine where I have Windmill deployed can run

docker pull $IMAGE

docker pull $IMAGE

no problem, in Windmill, I cannot pull my docker image as it's not authenticated. How can I authenticate Windmill? This is the script I run locally to auth

gcloud auth print-access-token \
  | docker login \
  -u oauth2accesstoken \
  --password-stdin us-east4-docker.pkg.dev

gcloud auth print-access-token \
  | docker login \
  -u oauth2accesstoken \
  --password-stdin us-east4-docker.pkg.dev

Nnsadeh Following up on my question from above: I have a docker image in a protected con...

nsadeh•6/4/24, 7:45 PM

My docker-compose worker has this line NOT commented out:

      - /var/run/docker.sock:/var/run/docker.sock

      - /var/run/docker.sock:/var/run/docker.sock

Rr0ck Forgive me for the very basic question. I am trying to move a step in a flow. I ...

rubenfOP•6/4/24, 8:16 PM

you have to click on the paste icon that replace the + icon in this mode

Nnsadeh Following up on my question from above: I have a docker image in a protected con...

rubenfOP•6/4/24, 8:17 PM

you should echo every part of this to make sure the password is what you expect

JJoldz 🔮In the documentation I see reference to a Table component: https://www.windmill....

rubenfOP•6/4/24, 8:17 PM

aggrid is the recommended table

Rrubenf you should echo every part of this to make sure the password is what you expect

nsadeh•6/4/24, 8:18 PM

So I am authenticating outside Windmill and hoping that Windmill is then authenticated. I don't know if I can install gcloud on Windmill. I did manage to get around that by pulling the image outside WM and then running on WM without pulling, but it's kind of annoying to have to SSH into my VM, pull, and run every time.

nsadeh•6/4/24, 8:20 PM

Another question: how do I pass custom environment variables to my docker process from WM? I read this: https://www.windmill.dev/docs/script_editor/custom_environment_variables but not sure how to actually use it.

FWIW, I am migrating from Cloud Run Jobs (GCP) in case it's interesting for you to see how potential customers would be able to migrate

Custom Environment Variables | Windmill

In a self-hosted environment, Windmill allows you to set custom environment variables for your scripts. This feature is useful when a script needs an environment variable prior to the main function executing itself. For instance, some libraries in Go do some setup in the 'init' function that depends on environment variables.

DDevid hey everyone 🙂 maybe this is the wrong place to ask. If so pls call me out 🙃 ...

antonioalves•6/4/24, 10:52 PM

what do you mean by inern database? I know that windmill uses postgres only

DDevid hey everyone 🙂 maybe this is the wrong place to ask. If so pls call me out 🙃 ...

henri-c•6/5/24, 7:02 AM

Hi!

"Windmill is not designed to store heavy data that extends beyond the execution of a script or flow. Indeed, for each computation the worker executing is not the same as the previous computation, so the data would have to be retrieved from another location.

Instead, Windmill is very convenient to use alongside data storage providers to manipulate big amounts of data."

You can check: https://www.windmill.dev/docs/core_concepts/persistent_storage/within_windmill

Within Windmill: Not recommended | Windmill

This page is part of our section on Persistent Storage & Databases which covers where to effectively store and manage the data manipulated by Windmill. Check that page for more options on data storage.

andness•6/5/24, 7:35 AM

Hi, I have a script that is suddenly failing quite often with the error

ExecutionErr: error during execution of the script:
process terminated by signal: Some(
9,
), stopped_signal: None, core_dumped: false

In the past I've seen this with OOMs, but this script is only peaking at 116MB on a worker with 2GB of memory so I don't understand how it can be memory related.

JJosh There's a bug with form field orders. I have a form that when you first view it,...

rubenfOP•6/5/24, 1:47 PM

I will take a look asap (probably a few days)

rubenfOP•6/5/24, 4:58 PM

@Josh I cannot reproduce a single instance of the form being out-of-order

sledge•6/5/24, 7:27 PM

I'm working on using windmill as our platform for all integrations of our ERP with external APIs. We need to integrate with stripe for credit card processing. We want to use the stripe checkout form, which means redirecting to their URL after running some code to create the checkout session. I built a script that is triggered at this URL:

https://app.windmill.dev/api/w/velvac-ifs-integrations/jobs/run_wait_result/p/f/stripe/internal_checkout

With token-in-query, I should be able to just hit this from a link in the browser. But since I do not have access to the response object in that typescript function, I do not see how I can redirect to the URL for the stripe checkout. Is there a way to get access to the response object so I can do a redirect?

rubenfOP•6/5/24, 9:50 PM

@Josh I can't reproduce at all, and both yaml order look to be exactly the same

rubenfOP•6/5/24, 9:52 PM

Coud you try with firefox, and reproducing by importing a yaml and the steps from there

wayne•6/6/24, 1:34 AM

can I somehow do something like that? It doesn't seem to work.
@rubenf

the use case is that users each have their own secret.

Wwayne can I somehow do something like that? It doesn't seem to work. @rubenf the use...

rubenfOP•6/6/24, 6:52 AM

Input transform don't support env variable, do that fetch in a separate step

Cctx for some reason, the scheduledFor when calling a workflow or script won't work. ...

rubenfOP•6/6/24, 7:29 AM

it should work so you might be doing something wrong.

Alper•6/6/24, 8:05 AM

any recommendations for python scripts throwing "RuntimeError: can't start new thread"? i thought about reducing the number of workers on each server (some have 8). but maybe there is a better way like increasing limits?

D3SL•6/6/24, 11:31 AM

I'm trying to understand the pricing for self-hosting. For example would 2 workers with 1 cpu each running for 1460 combined hours and 1 worker with 4 cpus that ran for 365 hours both equal $120 a month, since 1460/2 and 365*4 both equal the ~730 hours in a month?

rubenfOP•6/6/24, 11:38 AM

@D3SL we bill per provisioned vCPU (limit on helm/docker-compose) and we're not granular enough to adapt to auto-scaling (coming soon) but for now if you tell us that the telemetry data is incorrect because you auto-scaled up and down and we will trust that your vCPU count is correct and do a pro-rata of it being privisionned or not (if node is up only 1/4 of the month, it counts as 1/4 vCPU)

D3SL•6/6/24, 11:43 AM

So basically for now the billing system severely penalizes parallelizable workloads since it would see that I have workers with many cores, not that they're only actually running a fraction of the month?

D3SL•6/6/24, 11:45 AM

And by adapting to autoscaling I'm assuming that's what you're referring to, being able to track actual vcpu-hours as opposed to just what's in the compose.

rubenfOP•6/6/24, 11:47 AM

@D3SL I'm not saying that, I'm saying we will actually count vCPU-mins but our automated system is not fully ready so we will trust the vCPU numbers you give us until we deploy the new algorithm (ETA 1/2 weeks)

D3SL•6/6/24, 11:48 AM

Okay I misunderstood what you meant then. In either case it sounds like the whole issue is moot quite soon.

rubenfOP•6/6/24, 11:49 AM

exactly

D3SL•6/6/24, 11:52 AM

Spiffy, thanks for trying to explain it at least. I've only in the last month or two found out basically the entire field of "data engineering" exists. This is a major step up from bash aliases and cron jobs running R scripts.

rubenfOP•6/6/24, 11:55 AM

And we're only getting started, we're happy about where windmill is as a workflow engine, we have quite a bit of improvement to do for it to be a data pipeline engine (like giving you a pre-defined structure for incremental pipelines, maintaining column-wise lineage, etc)

D3SL•6/6/24, 12:40 PM

Don't sell yourself short. I've tried this, mage, reactive apps, DAG based R packages, and cronjobs. The only real hiccup I ran into was with the automatic serialization.

Kelora•6/6/24, 5:01 PM

does windmill have hard coded fixes for various automatic python imports? I've found a few that don't work out of the box and was wondering if I should report them to be fixed

rubenfOP•6/6/24, 5:02 PM

better, you can contribute them directly: https://github.com/windmill-labs/windmill/blob/main/backend/parsers/windmill-parser-py-imports/src/lib.rs#L57

GitHub

windmill/backend/parsers/windmill-parser-py-imports/src/lib.rs at m...

Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool. - windmill-labs/windmill