Hi, I'm trying out Windmill for
Hi, I'm trying out Windmill for orchestrating our analytics workflows, and as part of that I tried porting some existing code. The code interfaces with Clickhouse, so I have a small shared component for running Clickhouse queries. For simplicity say my
script module depends the shared module. I got this working fine, but then I was going to write script2 which also uses the shared module, and as part of that I added a dependency to shared.
After doing that I noticed that the schedule that is invoking script is failing with the error "ModuleNotFoundError: No module named 'clickhouse_driver'". "clichouse_driver" is the module I added to shared.
To me it appears that the automatic module installation doesn't notice that the upstream shared dependency has had a dependency change so when it attempt to run script it doesn't run pip install and as a result this fails.
To check this theory I made a spurious change to script and saved it again, and that fixed it.
I'm not sure I like this automagic dependency management, I think I would prefer to just manage a requirements.txt file like I'm used to and I don't consider that a hassle at all. But the issue I ran into here is a showstopper really as it would cause all scripts to break every time a dependency is added to one of the shared modules.2 Replies
This was very easy to reproduce so I created a GH issue: https://github.com/windmill-labs/windmill/issues/2882
GitHub
bug: Scheduled script with relative import fails when import added ...
Describe the bug When using shared code from a common module scripts that run on a schedule and depend on the common module fail if new imports are added to the common module. To reproduce Create a...
Thanks for the report and apologies for the issue.
As I wrote on the issue, that was related to the way common module and deployment work.
If you'd like to manually declare your imports as requirements.txt, you can do it:
https://www.windmill.dev/docs/advanced/imports#pinning-dependencies
Dependency Management & Imports | Windmill
Windmill's strength lies in its ability to run scripts without having to deal with separate dependency files. This is achieved by automatically parsing the imports and resolving the dependencies.