Hey @_pai03 ! I have created a small App that uses 3 components: - A file input that only accepts C

Hey @_pai03 !

I have created a small App that uses 3 components:

A file input that only accepts CSV
A button that processes the base64 CSV in GO into an array of object
A table that displays the data

The result of the file input component is connected to the input of the Go function. ( see docs: https://docs.windmill.dev/docs/apps/app_outputs#connecting-inputs)

I've attached the JSON of the app. You can import it from the home page -> Little arrow next to + App" -> Import from Raw JSON

Outputs | Windmill

On the left-hand side of the app editor, you can find the app outputs. They are three types of outputs:

FatonOP•4/25/23, 9:56 AM

app-csv.json6.59KB

rubenf•4/25/23, 10:35 AM

We're happy to provide this level of support on the enterprise level contract but currently we do not have an ecosystem of "windmill contractors"

rubenf•4/25/23, 12:47 PM

@jrklabs implemented "run now" https://github.com/windmill-labs/windmill/pull/1475 on latest

GitHub

feat(frontend): Schedules run now by fatonramadani · Pull Request #...

rubenf•4/25/23, 2:45 PM

It's a deno/MySQL lib issue

rubenf•4/25/23, 2:45 PM

That version of the MySQL lib doesn't work with latest deno

rubenf•4/25/23, 2:46 PM

Could you try with https://deno.land/x/mysql@v2.11.0

rubenf•4/25/23, 3:22 PM

Ah yes, sorry, I didn't see you were not actually using mysql

rubenf•4/25/23, 3:23 PM

So here is the issue, our previous version of the client bundled a mysql client, but the mysql dependency was using non pinned version of the std

rubenf•4/25/23, 3:23 PM

So we fixed that issue by removing the bundling of the mysql client

rubenf•4/25/23, 3:23 PM

We will update the tutorial, thanks for catching it

rubenf•4/25/23, 3:23 PM

for a bit of context, we had to bump all the versions in the hub as well

Rrubenf @jrklabs implemented "run now" https://github.com/windmill-labs/windmill/pull/14...

JRKLabs-Jeff Knorr-USA•4/25/23, 4:22 PM

I tried this out today--it's great!! Thank you!

rubenf•4/25/23, 7:05 PM

That's a great idea. I'm actually reworking our public helm charts to first update them as they are terribly obsolete but also showcase how to use worker groups. I think what you suggest might just work, spinning a worker that kill itself after one job. The issue though is that caching won't work very well unless you use a persistent disk

rubenf•4/25/23, 7:06 PM

But if you're up for it, i will add an env variable for number of job before the worker suicide itself

rubenf•4/25/23, 7:10 PM

So it all depends on how much cold start matter to you

rubenf•4/25/23, 7:18 PM

You could use retries for that. Make your job fail until the data is there, and configure retry to try again later with either exponential or constant backoff

rubenf•4/25/23, 7:18 PM

To fail a job, you can just throw/raise an exception

rubenf•4/25/23, 7:19 PM

when a job is failed, it is first retried if retries are defined for that step, and then passed to the error handler if error handler is defined

rubenf•4/25/23, 7:19 PM

so in your example, you could set 12 attempts of constant retries, after 300 seconds each.

Sindre•4/25/23, 7:38 PM

oh, having workers that killed them self for long running jobs would by nice, could maybe combined with worker groups so that you can have a cheap node always on, running most of the jobs, and then when you need that GPU or bigger node then thats on-demand. Would be a nice tradeoff, then you can have your cake and eat it tooe

rubenf•4/25/23, 7:39 PM

Yeah I think @cheddarwhizzy | Gelotto insight is the right one with: /jobs/queue/list/jobs/queue/list, I just need to add a filter by tag

rubenf•4/25/23, 7:41 PM

but I kind of think that to downscale, you might actually not want the worker to kill itself because it can then act as a timeout. "If no new job for 5m, kill yourself because cpu average too low, otherwise stay alive"

rubenf•4/25/23, 7:42 PM

maybe I will rebrand windmill as the universal kubernetes arbitrary code execution layer.

rubenf•4/25/23, 7:42 PM

workflows + retool

rubenf•4/25/23, 9:28 PM

I don't mind using a network file system for multiple pods/workers to mount for cache purposes. I see the enterprise edition supports S3 cache, so if that gets open sourced for self-hosted, a local minio setup would be useful there.

It's a tough question. The code is actually already public, just under a rust feature flag. It's not particularly hard code, but we need to have an incentive for people to give us a bit of money so we can continue writing good code and making our investors happy. On the other hand, it feels a bit silly to force people into a contrived inferior network mount so they don't use our shiny Enterprise Edition

While you tinker, I will also brainstorm some ideas of how to better autoscale windmill. I'm quite sure that /jobs/queue/list/jobs/queue/list is the right upscale API. For the downscale, I think it would likely be something like "kill yourself after X minutes without a job" which is a capitalist utopia.

rubenf•4/25/23, 11:25 PM

@cheddarwhizzy | Gelotto we already have the prom metrics: queue_push_countqueue_push_count

rubenf•4/25/23, 11:33 PM

And we have the prom metrics queue_pull_count, with those 2 you can have a nice overview of the throughput of jobs. But we have even better as metrics:
worker_execution_duration_counter
worker_sleep_duration_counter
worker_pull_duration_counter

rubenf•4/26/23, 6:29 AM

@samurai Is it possible that you had to refresh the flow in the app for the fields to show up as you were doing both in different tabs ?

You should be able to pass in all the content of your input fields by connecting them as the output of those input fields. You can click the little plug on the right of the field.

rubenf•4/26/23, 12:28 PM

With a transformer you can

rubenf•4/26/23, 12:28 PM

transformer or background script in javascript

rubenf•4/26/23, 12:29 PM

return {field1: a.result, field2: b.result }return {field1: a.result, field2: b.result } is one possibility for instance

rubenf•4/26/23, 2:21 PM

Glad to hear, and always curious to see what we can still improve wrt to the smooth experience that n8n offers. For the logos, it would work automatically as long the sso is google / gitlab / microsoft / github normally