What's the recommended way to run dbt transformations?
Currently migrating from a Prefect pipeline which uses https://github.com/datarootsio/prefect-dbt-flow and was wondering how I can run my dbt models in Windmill.
This library automatically makes a new Task/Node in the DAG for each dbt model. I'm not really sure what the best way is to pull your dbt files (.sql models & yaml configurations) into Windmill, as well as running them using the
dbt CLI. One way I thought of is to package my dbt models (and dbt CLI tool) in a Docker image, which I would then pull from a private registry inside Windmill. This, however, won't handle the breaking-down-into-multiple-tasks (scripts in Windmill) part. Any ideas? I'm also considering trying sqlmesh but I've never used it, but I think it has a similar "deployment" model.GitHub
GitHub - datarootsio/prefect-dbt-flow: prefect integration for runn...
prefect integration for running dbt . Contribute to datarootsio/prefect-dbt-flow development by creating an account on GitHub.
1 Reply
It's a good question. I'm not too certain right now but a great dbt integration is planned for the next 3 months