I want to import a python dependency
I want to import a python dependency directly from a gitlab repo using a deploy token:
i'll put the error in thread
18 Replies
oh that's an easy fix away!
@guillaume , we need to replace ('/') with _ in the path to create the lock
context, we use lock for pip to work well in context of multiple workers doing concurrent pip installs
I can take a look in a bit
thanks a lot
I'm looking into it, I think it's a bit more tricky than doing a replace_all, but I'll have something for you soon
@Alper the fix merged yesterday, if you pull latest you should be able to use python package from git repo
thanks @guillaume! is it available for the helm chart as well? it looks like the latest release is 3 days old
not yet, will do a release asap
Unknown User•11mo ago
Message Not Public
Sign In & Join Server To View
you can't @victorL87
Unknown User•11mo ago
Message Not Public
Sign In & Join Server To View
I agree with you, but implementation wise it would have terrible consequences if we made the lockfile something that you evaluate later on rather than a raw value
I don't have a good solution right now other than:
- you should scope that token in a way that it doesn't have more privilege than just reading that repo
- use private pypi registry and set that at the instance level
Unknown User•11mo ago
Message Not Public
Sign In & Join Server To View
it actually shoudn't install them on the worker's system pip and it does some variation of what you describe
We have our own concept of pip caches that is derived from our global cache needs. We had to create the concept of piptars for it: https://www.windmill.dev/docs/misc/s3_cache
S3 Distributed Dependency Cache | Windmill
Workers cache aggressively the dependencies (and each version of them since every script has its own lockfile with a specific version for each dependency) so they are never pulled nor installed twice on the same worker. However, with a bigger cluster, for each script, the likelihood of being seen by a worker for the first time increases (and the...
you can find all the desired dependency in /tmp/windmill/cache/pip/...
the reason why Victor asks is that we get lots of these while the worker installs the requirements:
we do run pip as root, but we do not store it on the root default layout
but it's not really ran as root, it's ran as the docker user
in our case the warning can be very safely ignored
okay thanks. also for fixing the issue so quickly