Best practices and guidelines
Hi Team, I am new to windmill.dev, but I have loads of experience as a data engineer using other orchestration tools like Airflow.
In my previous experience, we would always keep the heavy data loads and data processing outside of Airflow like in aws batch, spark and orchestrate the jobs via Airflow as a workflow engine.
Is it the same practice with Windmill.dev is there any docs or guidelines that explains best practices and what to do and what not do at enterprise scale ? Eg: processing millions of records per hour
1 Reply
We have native integration with duckdb and are about to merge native integration with ducklake. I think in most cases you will gain a lot of performance by leveraging those if your scale enable it rather than spark