WindmillWWindmill
Powered by
fr3fouF
Windmill•16mo ago•
1 reply
fr3fou

What's the recommended way to run dbt transformations?

Currently migrating from a Prefect pipeline which uses https://github.com/datarootsio/prefect-dbt-flow and was wondering how I can run my dbt models in Windmill.
This library automatically makes a new Task/Node in the DAG for each dbt model. I'm not really sure what the best way is to pull your dbt files (.sql models & yaml configurations) into Windmill, as well as running them using the
dbt
dbt
CLI. One way I thought of is to package my dbt models (and dbt CLI tool) in a Docker image, which I would then pull from a private registry inside Windmill. This, however, won't handle the breaking-down-into-multiple-tasks (scripts in Windmill) part. Any ideas? I'm also considering trying sqlmesh but I've never used it, but I think it has a similar "deployment" model.
GitHub
GitHub - datarootsio/prefect-dbt-flow: prefect integration for runn...
prefect integration for running dbt . Contribute to datarootsio/prefect-dbt-flow development by creating an account on GitHub.
GitHub - datarootsio/prefect-dbt-flow: prefect integration for runn...
WindmillJoin
3,362Members
Resources

Similar Threads

Was this page helpful?
Recent Announcements
Recent Announcements
henri-c

Weekly kenote to tell you about our latest updates https://discord.com/channels/930051556043276338/1278977038430240813 https://youtube.com/live/2dGd9TdT8xs?feature=share

henri-c · 4d ago

Pyra

### HTTP tracing (EE) Capture HTTP requests made by job scripts as observability spans Features: - View HTTP request traces (method, URL, status, timing) in the job details UI - Auto-instrumentation for Native TypeScript, MITM proxy for other languages - Integrates with external OpenTelemetry collectors changelog: https://www.windmill.dev/changelog/http-tracing docs: https://www.windmill.dev/docs/advanced/instance_settings#http-tracing Additionally jobs memory metrics are now fully OSS!

Pyra · 2w ago

henri-c

First keynote of the year here https://discord.com/channels/930051556043276338/1278977038430240813 🙂

henri-c · 4w ago

Similar Threads

Latency on waiting, What's the recommended setup for workers for concurrent webhook calls?
connorCconnor / help
17mo ago
What is the best way to create a service user?
andnessAandness / help
2y ago
What does Filter-by-args-set do on run's page
ym1198Yym1198 / help
3y ago
Is there a way to run windmill under a subpath?
d10sfanDd10sfan / help
2y ago