For anyone that updated their instance on latest between 48h and 24h, I did a mistake and your instance will be unable to start on the next update unless you apply the following to your db:
DSo if I skip 219 version and upgrade directly to 220, then I should be safe, correct ?
RIt's not on the released versions
Rjust for docker-compose users if they pulled latest in between those timestamps
MThanks, needed this.
Acan any one tell me how to get query parameter in get webhook ?
FHi @Angad, in synchronous webhoobs, you can use the query paramater to pipe data as args of your script/flow. Check out our doc:
https://www.windmill.dev/docs/core_concepts/webhooks#synchronous
Also, please use the #help or #feedback channels for questions!
https://www.windmill.dev/docs/core_concepts/webhooks#synchronous
Also, please use the #help or #feedback channels for questions!
Webhooks are a way to interact with Windmill using standard web technologies.
A@Faton Thanks alot for answering
R@Binarytide are you sure you put everything down, and then up again ?
Rno
Rag-grid was updated to 0.31.0
RYou can now parametrize the visibility of fields in forms based on the value of other fields:
GNew feature: you can now sync your workspace with a Git repo. Any updated script, flow or app will be automatically pushed to the repo upon deploy! Only for EE. More details -> https://www.windmill.dev/docs/advanced/git_sync
RYou can now declare some steps as being sensitive and having their args/logs/results removed at the completion of the flow (ee only):
HNEW [Major] 
Schedule App Reports
Send a PDF or PNG snapshot of any app at a given schedule to slack / discord / email or via any custom script.
Enabling this feature will create a flow and a schedule in your workspace.
Docs: https://www.windmill.dev/docs/apps/schedule_reports
Tutorial: https://www.youtube.com/watch?v=4VSAZ-WTh4Y
Schedule App Reports
Send a PDF or PNG snapshot of any app at a given schedule to slack / discord / email or via any custom script.
Enabling this feature will create a flow and a schedule in your workspace.
Docs: https://www.windmill.dev/docs/apps/schedule_reports
Tutorial: https://www.youtube.com/watch?v=4VSAZ-WTh4Y
GNEW [Major]
Running Services with Perpetual Scripts
Perpetual scripts restart upon ending unless canceled.
Docs: https://www.windmill.dev/docs/script_editor/perpetual_scripts
Video: https://www.youtube.com/watch?v=5uw3JWiIFp0
Blog on Service script pattern in Windmill using Kafka: https://www.windmill.dev/blog/service-script-kafka
Running Services with Perpetual Scripts
Perpetual scripts restart upon ending unless canceled.
Docs: https://www.windmill.dev/docs/script_editor/perpetual_scripts
Video: https://www.youtube.com/watch?v=5uw3JWiIFp0
Blog on Service script pattern in Windmill using Kafka: https://www.windmill.dev/blog/service-script-kafka
RNEW [Major]
Bun smart assistant working with any dependency, all in browser
Bun now has a smart assistant that does proper completion on any npm packages, and we achieved this feat WITHOUT using LSP. Your browser is now doing all the work of fetching the types and running the smart assistant locally. It even work with relative imports!
Bun smart assistant working with any dependency, all in browser
Bun now has a smart assistant that does proper completion on any npm packages, and we achieved this feat WITHOUT using LSP. Your browser is now doing all the work of fetching the types and running the smart assistant locally. It even work with relative imports!
RNEW [Minor]
Summaries can now be given to schedules
The summary will be displayed in the schedule list and the search feature can also search by summary
Summaries can now be given to schedules
The summary will be displayed in the schedule list and the search feature can also search by summary
Nthis doesn't seem to work if you version lock dependencies
RNEW [Minor]
Schedules flows can now be toggled to never overlap over their full execution
Schedules flows can now be toggled to never overlap over their full execution
Rthis is fixed in latest
RThere are lot of unannounced change (we didn't have time to write proper changelog). But PSA:
the .wmill folder was source of much confusion. It's not needed anymore in a world where Git Sync exist.
wmill sync --raw will soon become the default and it doesn't rely on the .wmill folder
Since not everyone can use Git Sync since it's under EE, the previous behavior will now be flagged under --stateful.
We will keep a transition period for a few weeks and then change the default, update and add --stateful to your wmill commands if you rely on this behavior.
the .wmill folder was source of much confusion. It's not needed anymore in a world where Git Sync exist.
wmill sync --raw will soon become the default and it doesn't rely on the .wmill folder
Since not everyone can use Git Sync since it's under EE, the previous behavior will now be flagged under --stateful.
We will keep a transition period for a few weeks and then change the default, update and add --stateful to your wmill commands if you rely on this behavior.
RWe have an alpha of a NodeJS compatibility mode (a true one, where NodeJS is the actual runtime). We are looking for enterprise beta testers to test it with your legacy NodeJS code.
RPSA, we are deprecating .wmillignore in favor of a wmill.yaml that has an includes field. It's already the case in 1.250.0 of the CLI (which is back-compatible, nothing has broken). We noticed that it was way more common to have whitelist patterns than blacklist and we will centralize every windmill setting in this wmill.yaml
See example here: https://github.com/windmill-labs/windmill-sync-example/commit/62949af56be1c54a3f9e6364663d93e71c1cab73#diff-cb4c7b506a5f402c[…]6d342bb74b338806fb402
Lots of very exciting changes coming to the CLI for a great fully local development experience. More to share soon
See example here: https://github.com/windmill-labs/windmill-sync-example/commit/62949af56be1c54a3f9e6364663d93e71c1cab73#diff-cb4c7b506a5f402c[…]6d342bb74b338806fb402
Lots of very exciting changes coming to the CLI for a great fully local development experience. More to share soon
HNEW [Major]
We're starting to catch up on our changelogs that haven't been announced yet. The first is the Decision Tree component, which allows you to create TypeForm-like dynamic forms embedding scripts & flows.
Changelog: https://www.windmill.dev/changelog/decision-tree
Tutorial: https://www.windmill.dev/blog/dynamic-form
We're starting to catch up on our changelogs that haven't been announced yet. The first is the Decision Tree component, which allows you to create TypeForm-like dynamic forms embedding scripts & flows.
Changelog: https://www.windmill.dev/changelog/decision-tree
Tutorial: https://www.windmill.dev/blog/dynamic-form
HWe're taking this opportunity to announce our new Changelog page, where you can view our major releases as well as the date and version in which they were deployed: https://www.windmill.dev/changelog
HNEW [Major] Rich table display
Released on 23/01/2024 under v1.251.1
Display arrays of objects as interactive tables (search, filter, hide, view, download as csv)
Changelog: https://www.windmill.dev/changelog/rich-render
Docs: https://www.windmill.dev/docs/core_concepts/rich_display_rendering#rich-table-display
Rich table displayReleased on 23/01/2024 under v1.251.1
Display arrays of objects as interactive tables (search, filter, hide, view, download as csv)
Changelog: https://www.windmill.dev/changelog/rich-render
Docs: https://www.windmill.dev/docs/core_concepts/rich_display_rendering#rich-table-display
The result renderer in Windmill supports rich display rendering, allowing you to customize the display format of your results. By leveraging specific formats, you can display images, files, tables, HTML, JSON, and more.
RIn latest, we did a few improvements to the runs page which should feel snappier. And on the admins workspace, you can now see the runs of all workspaces at once:
HNEW [Major] Database Studio
Released on 23/01/2024 under v1.251.1
In apps, interaction with database content made easy: from a PostgreSQL or MySQL resource, display, edit, add rows, delete rows ... and connect to other components.
Changelog: https://www.windmill.dev/changelog/database-studio
Docs: https://www.windmill.dev/docs/apps/app_configuration_settings/database_studio
Database StudioReleased on 23/01/2024 under v1.251.1
In apps, interaction with database content made easy: from a PostgreSQL or MySQL resource, display, edit, add rows, delete rows ... and connect to other components.
Changelog: https://www.windmill.dev/changelog/database-studio
Docs: https://www.windmill.dev/docs/apps/app_configuration_settings/database_studio
Rrunnable fields can now be individually selected to not trigger recompute
HNEW [Major] Flow & Metadata Copilot
Released on 15/02/2024 under v1.270.0
The Flow & Metadata Copilot is an assistant powered by an OpenAI resource that simplifies your script & flows building experience by population fields (summaries, descriptions, step input expressions) automatically based on context and prompts.
Changelog: https://www.windmill.dev/changelog/ai-copilot
Docs: https://www.windmill.dev/docs/core_concepts/ai_generation#step-input-copilot
Flow & Metadata CopilotReleased on 15/02/2024 under v1.270.0
The Flow & Metadata Copilot is an assistant powered by an OpenAI resource that simplifies your script & flows building experience by population fields (summaries, descriptions, step input expressions) automatically based on context and prompts.
Changelog: https://www.windmill.dev/changelog/ai-copilot
Docs: https://www.windmill.dev/docs/core_concepts/ai_generation#step-input-copilot
RI will write a blog post asap on it, but this weekend we achieved a very cool property for a distributed sytem:
- Scripts were always 100% reliable as in, they would either execute and be completed with a success or failure, or retried if the worker crashed at ANY point (and I really mean any, even mid transaction, that's the beauty on relying on the beast that is Postgresql). It was achieved using atomic statement for pulling jobs and writing back their progress timestamps, regularly and on completion.
- Flows were 99% reliable but had some extremely ephemeral point-in-time where if a crash happened, a flow could be stuck forever. Those events were so rare and unlikely on a modern infra that we didn't prioritize improving that but that is now done:
Flows are now guaranteed to complete when they are scheduled given that enough workers are there to process them. This is done through a series of atomic statements in the right places of the finite state machine that runs the flows. If such crash on the machine happen, the flow will be guaranteed to progress in a finite amount of time and propagate the error back up, and then have it be treated by error handlers if any making windmill 100% observable.
R(sadly, I broke some eggs with the change above, if you upgraded in the last 2 days and were using the "No flow overlap" feature of schedules for flows, then your schedules silently turned off). Update to latest release and re-enable those schedules to fix.
RSecond PSA, if you use parrallel branches/for-loop and there is a time difference of ending branch > 30s, and you updated in the last 3 days, please update to 1.279.0. There is an edge-case that would make your flow being caught and cancelled by the new timeout progress monitor.
RWe are releasing in beta Workflow as code for Python and Typescript. No more excuse to use Airflow or Prefect: https://www.windmill.dev/docs/core_concepts/workflows_as_code
Flows are not the only way to write distributed programs that execute distinct jobs. Another approach is to write a program that defines the jobs and their dependencies, and then execute that program. This is known as workflows as code.
RBun is now the default language in the UI:
R(In not too long, we will make language order and starting templates configurable)
RWindmill supports many languages, but organizations usually only use a few, now you can configure the languages that are visible and their orders:
The setting apply to scripts, flows and apps and is global to all users within a workspace but only configurable by admins
The setting apply to scripts, flows and apps and is global to all users within a workspace but only configurable by admins
RAll about logs
One of our large scale customers noticed that their database disk usage was much much higher than they anticipated. After investigation, we realized that our use of the database for streaming was very suboptimal in a few ways due to the nature of update in postgres. When you update a row in Postgres, it will actually keep the prior row as a dead tuple until it is collected. It doesn't matter in much case but it will if you're appending a few log lines to a 25MB log row, every 500ms.
We have completely refactored the way we deal with logs in major ways and starting on 1.295.0 you should feel comfortable having extremely large logs on Windmill
First action we took was to extract the logs from the queue table to a separate table. That was to avoid update unrelated to logs creating dead tuples. Second action was to make the streaming rate from the worker adaptive to the the duration of the job: a longer job does not need to update its log every 500ms, every 2.5s is reasonable for jobs of more than 10s, 5s for 60s+, etc...
But that was still not enough, even every 2.5s an update on a 25mb log would create lots of heavy dead tuples. And 25MB is not that large, our customer should feel confident streaming GBs of logs per job with jobs that run for months.
So we completely revisited the way we store logs to only treat the database as a buffer for streaming purpose rather than long term storage. We keep the db as a 5000 char buffer to still provide the same instant preview as before but:
One of our large scale customers noticed that their database disk usage was much much higher than they anticipated. After investigation, we realized that our use of the database for streaming was very suboptimal in a few ways due to the nature of update in postgres. When you update a row in Postgres, it will actually keep the prior row as a dead tuple until it is collected. It doesn't matter in much case but it will if you're appending a few log lines to a 25MB log row, every 500ms.
We have completely refactored the way we deal with logs in major ways and starting on 1.295.0 you should feel comfortable having extremely large logs on Windmill
First action we took was to extract the logs from the queue table to a separate table. That was to avoid update unrelated to logs creating dead tuples. Second action was to make the streaming rate from the worker adaptive to the the duration of the job: a longer job does not need to update its log every 500ms, every 2.5s is reasonable for jobs of more than 10s, 5s for 60s+, etc...
But that was still not enough, even every 2.5s an update on a 25mb log would create lots of heavy dead tuples. And 25MB is not that large, our customer should feel confident streaming GBs of logs per job with jobs that run for months.
So we completely revisited the way we store logs to only treat the database as a buffer for streaming purpose rather than long term storage. We keep the db as a 5000 char buffer to still provide the same instant preview as before but:
- On EE, the logs will be streamed to S3 if you connected your instance to S3, everything is seamless and you can still download the entire log, windmill will take care of streaming from S3
- non EE, the excess log (>10000 chars) will be stored on disk of the worker (mount /tmp/windmill/logs to persist those)
RVery related update, but our use of the logs in the frontend was also suboptimal, we were loading many of them unecessarily before the user was in a view that actually rendered them, for instance in the flow status viewer, runs page detail and other log views. We now do it just-in-time, as soon as the user need them which will make windmill overall feel more reactive, avoid unecessary network calls, especially in the flow status viewer were the logs of all jobs were fetched even though none were displayed.
RAlso, we refactored our s3 distributed python cache to not rely on rclone anymore but to be done natively in rust:
- Much better and parallelized performance for dependency fetching from distributed cache
- less s3 usage as due to the increase in performance, there is no need to sync the cache at start, everything can be done just in time
Rand last but not least, @Hugo implemented custom contextual variables that will act as env variable for all jobs within a workspace. We still recommend using variables but in some cases (for instance your imports depend on env variables), this might be a good escape hatch.
RIntroducing custom flow states:
You can now set and retrieve a value given a key from any step of flow and it will be available from within the flow globally. That state will be stored in the flow state itself and thus has the same lifetime than the flow job itself.
Make it a powerful escape hatch when passing data as output/input is not feasible and using getResource/setResource has the issue of cluttering the workspace and inconvenient UX
You can now set and retrieve a value given a key from any step of flow and it will be available from within the flow globally. That state will be stored in the flow state itself and thus has the same lifetime than the flow job itself.
Make it a powerful escape hatch when passing data as output/input is not feasible and using getResource/setResource has the issue of cluttering the workspace and inconvenient UX
Ralso QOL improvements, webhook specific tokens have a default unique label that is displayed as the username in runs make it easy to distinguish when a run is from a user or a webhook:
