Following up on my question from above: I have a docker image in a protected container registry. My

Following up on my question from above: I have a docker image in a protected container registry. My docker on my host machine where I have Windmill deployed can run

docker pull $IMAGE

docker pull $IMAGE

no problem, in Windmill, I cannot pull my docker image as it's not authenticated. How can I authenticate Windmill? This is the script I run locally to auth

gcloud auth print-access-token \
  | docker login \
  -u oauth2accesstoken \
  --password-stdin us-east4-docker.pkg.dev

gcloud auth print-access-token \
  | docker login \
  -u oauth2accesstoken \
  --password-stdin us-east4-docker.pkg.dev

Nnsadeh Following up on my question from above: I have a docker image in a protected con...

nsadehOP•6/4/24, 7:45 PM

My docker-compose worker has this line NOT commented out:

Rr0ck Forgive me for the very basic question. I am trying to move a step in a flow. I ...

rubenf•6/4/24, 8:16 PM

you have to click on the paste icon that replace the + icon in this mode

Nnsadeh Following up on my question from above: I have a docker image in a protected con...

rubenf•6/4/24, 8:17 PM

you should echo every part of this to make sure the password is what you expect

JJoldz 🔮In the documentation I see reference to a Table component: https://www.windmill....

rubenf•6/4/24, 8:17 PM

aggrid is the recommended table

Rrubenf you should echo every part of this to make sure the password is what you expect

nsadehOP•6/4/24, 8:18 PM

So I am authenticating outside Windmill and hoping that Windmill is then authenticated. I don't know if I can install gcloud on Windmill. I did manage to get around that by pulling the image outside WM and then running on WM without pulling, but it's kind of annoying to have to SSH into my VM, pull, and run every time.

nsadehOP•6/4/24, 8:20 PM

Another question: how do I pass custom environment variables to my docker process from WM? I read this: https://www.windmill.dev/docs/script_editor/custom_environment_variables but not sure how to actually use it.

FWIW, I am migrating from Cloud Run Jobs (GCP) in case it's interesting for you to see how potential customers would be able to migrate

Custom Environment Variables | Windmill

In a self-hosted environment, Windmill allows you to set custom environment variables for your scripts. This feature is useful when a script needs an environment variable prior to the main function executing itself. For instance, some libraries in Go do some setup in the 'init' function that depends on environment variables.

DDevid hey everyone 🙂 maybe this is the wrong place to ask. If so pls call me out 🙃 ...

antonioalves•6/4/24, 10:52 PM

what do you mean by inern database? I know that windmill uses postgres only

DDevid hey everyone 🙂 maybe this is the wrong place to ask. If so pls call me out 🙃 ...

henri-c•6/5/24, 7:02 AM

Hi!

"Windmill is not designed to store heavy data that extends beyond the execution of a script or flow. Indeed, for each computation the worker executing is not the same as the previous computation, so the data would have to be retrieved from another location.

Instead, Windmill is very convenient to use alongside data storage providers to manipulate big amounts of data."

You can check: https://www.windmill.dev/docs/core_concepts/persistent_storage/within_windmill

Within Windmill: Not recommended | Windmill

This page is part of our section on Persistent Storage & Databases which covers where to effectively store and manage the data manipulated by Windmill. Check that page for more options on data storage.

andness•6/5/24, 7:35 AM

Hi, I have a script that is suddenly failing quite often with the error

ExecutionErr: error during execution of the script:
process terminated by signal: Some(
9,
), stopped_signal: None, core_dumped: false

In the past I've seen this with OOMs, but this script is only peaking at 116MB on a worker with 2GB of memory so I don't understand how it can be memory related.

JJosh There's a bug with form field orders. I have a form that when you first view it,...

rubenf•6/5/24, 1:47 PM

I will take a look asap (probably a few days)

rubenf•6/5/24, 4:58 PM

@Josh I cannot reproduce a single instance of the form being out-of-order

sledge•6/5/24, 7:27 PM

I'm working on using windmill as our platform for all integrations of our ERP with external APIs. We need to integrate with stripe for credit card processing. We want to use the stripe checkout form, which means redirecting to their URL after running some code to create the checkout session. I built a script that is triggered at this URL:

https://app.windmill.dev/api/w/velvac-ifs-integrations/jobs/run_wait_result/p/f/stripe/internal_checkout

With token-in-query, I should be able to just hit this from a link in the browser. But since I do not have access to the response object in that typescript function, I do not see how I can redirect to the URL for the stripe checkout. Is there a way to get access to the response object so I can do a redirect?

rubenf•6/5/24, 9:50 PM

@Josh I can't reproduce at all, and both yaml order look to be exactly the same

rubenf•6/5/24, 9:52 PM

Coud you try with firefox, and reproducing by importing a yaml and the steps from there

wayne•6/6/24, 1:34 AM

can I somehow do something like that? It doesn't seem to work.
@rubenf

the use case is that users each have their own secret.

Wwayne can I somehow do something like that? It doesn't seem to work. @rubenf the use...

rubenf•6/6/24, 6:52 AM

Input transform don't support env variable, do that fetch in a separate step

Cctx for some reason, the scheduledFor when calling a workflow or script won't work. ...

rubenf•6/6/24, 7:29 AM

it should work so you might be doing something wrong.

Alper•6/6/24, 8:05 AM

any recommendations for python scripts throwing "RuntimeError: can't start new thread"? i thought about reducing the number of workers on each server (some have 8). but maybe there is a better way like increasing limits?

D3SL•6/6/24, 11:31 AM

I'm trying to understand the pricing for self-hosting. For example would 2 workers with 1 cpu each running for 1460 combined hours and 1 worker with 4 cpus that ran for 365 hours both equal $120 a month, since 1460/2 and 365*4 both equal the ~730 hours in a month?

rubenf•6/6/24, 11:38 AM

@D3SL we bill per provisioned vCPU (limit on helm/docker-compose) and we're not granular enough to adapt to auto-scaling (coming soon) but for now if you tell us that the telemetry data is incorrect because you auto-scaled up and down and we will trust that your vCPU count is correct and do a pro-rata of it being privisionned or not (if node is up only 1/4 of the month, it counts as 1/4 vCPU)

D3SL•6/6/24, 11:43 AM

So basically for now the billing system severely penalizes parallelizable workloads since it would see that I have workers with many cores, not that they're only actually running a fraction of the month?

D3SL•6/6/24, 11:45 AM

And by adapting to autoscaling I'm assuming that's what you're referring to, being able to track actual vcpu-hours as opposed to just what's in the compose.

rubenf•6/6/24, 11:47 AM

@D3SL I'm not saying that, I'm saying we will actually count vCPU-mins but our automated system is not fully ready so we will trust the vCPU numbers you give us until we deploy the new algorithm (ETA 1/2 weeks)

D3SL•6/6/24, 11:48 AM

Okay I misunderstood what you meant then. In either case it sounds like the whole issue is moot quite soon.

rubenf•6/6/24, 11:49 AM

exactly

D3SL•6/6/24, 11:52 AM

Spiffy, thanks for trying to explain it at least. I've only in the last month or two found out basically the entire field of "data engineering" exists. This is a major step up from bash aliases and cron jobs running R scripts.

rubenf•6/6/24, 11:55 AM

And we're only getting started, we're happy about where windmill is as a workflow engine, we have quite a bit of improvement to do for it to be a data pipeline engine (like giving you a pre-defined structure for incremental pipelines, maintaining column-wise lineage, etc)

D3SL•6/6/24, 12:40 PM

Don't sell yourself short. I've tried this, mage, reactive apps, DAG based R packages, and cronjobs. The only real hiccup I ran into was with the automatic serialization.

Kelora•6/6/24, 5:01 PM

does windmill have hard coded fixes for various automatic python imports? I've found a few that don't work out of the box and was wondering if I should report them to be fixed

rubenf•6/6/24, 5:02 PM

better, you can contribute them directly: https://github.com/windmill-labs/windmill/blob/main/backend/parsers/windmill-parser-py-imports/src/lib.rs#L57

GitHub

windmill/backend/parsers/windmill-parser-py-imports/src/lib.rs at m...

Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool. - windmill-labs/windmill

Kelora•6/6/24, 5:04 PM

oh right on, what are your thoughts on post-install content that is required? Here is my workaround

import asyncio
import os
from playwright.async_api import async_playwright
from PIL import Image, ImageFont, ImageDraw
import base64
from io import BytesIO
import datetime
import logging
import subprocess
import sys


import cv2
import numpy as np
from PIL import Image


def install_playwright():
    try:
        subprocess.check_call([sys.executable, "-m", "pip", "install", "playwright"])
        subprocess.check_call([sys.executable, "-m", "pip", "install", "azure-storage-blob"])
        subprocess.check_call([sys.executable, "-m", "pip", "install", "azure-core"])
        subprocess.check_call([sys.executable, "-m", "playwright", "install"])
        subprocess.check_call([sys.executable, "-m", "playwright", "install-deps"])
        print("Playwright installed successfully.")
    except subprocess.CalledProcessError as e:
        print(f"An error occurred while installing Playwright: {e}")


install_playwright()

import asyncio
import os
from playwright.async_api import async_playwright
from PIL import Image, ImageFont, ImageDraw
import base64
from io import BytesIO
import datetime
import logging
import subprocess
import sys


import cv2
import numpy as np
from PIL import Image


def install_playwright():
    try:
        subprocess.check_call([sys.executable, "-m", "pip", "install", "playwright"])
        subprocess.check_call([sys.executable, "-m", "pip", "install", "azure-storage-blob"])
        subprocess.check_call([sys.executable, "-m", "pip", "install", "azure-core"])
        subprocess.check_call([sys.executable, "-m", "playwright", "install"])
        subprocess.check_call([sys.executable, "-m", "playwright", "install-deps"])
        print("Playwright installed successfully.")
    except subprocess.CalledProcessError as e:
        print(f"An error occurred while installing Playwright: {e}")


install_playwright()

Kelora•6/6/24, 5:05 PM

playwrite needs browsers installed after the pip install or else it can't really do anything

rubenf•6/6/24, 5:05 PM

you should use init scripts for that

Kelora•6/6/24, 5:06 PM

ok gotcha, ill just look into fixing the azure ones then, those are direct swaps but I think it is a many to one relationship which might not be as easy as the one to one relationships in the code you linked

Alper•6/6/24, 5:47 PM

@Kelora i am also using playwright in my scripts. I solved it by creating a custom worker image with preinstalled browser and dependencies

Kelora•6/6/24, 5:49 PM

i thought about doing that, but i have a working setup now, just wanted to see what best practice was

Kelora•6/6/24, 5:50 PM

and i installed with defaults which disables docker and didnt want to jump into that if i didnt have to