Alper•11mo ago

I want to import a python dependency

I want to import a python dependency directly from a gitlab repo using a deploy token:

#extra_requirements:
#git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library

#extra_requirements:
#git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library

i'll put the error in thread

18 Replies

Alper•11mo ago

resolving dependencies...
content of requirements:
git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library
library

WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
--- PIP INSTALL ---

charset-normalizer==2.0.12 is being installed for the first time.
 It will be cached for all ulterior uses.
Collecting charset-normalizer==2.0.12
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Installing collected packages: charset-normalizer
Successfully installed charset-normalizer-2.0.12
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
--- PIP INSTALL ---

integrations @ git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library is being installed for the first time.
 It will be cached for all ulterior uses.
flock: cannot open lock file /tmp/windmill/cache/lock/pip-library @ git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library.lock: No such file or directory

resolving dependencies...
content of requirements:
git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library
library

WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
--- PIP INSTALL ---

charset-normalizer==2.0.12 is being installed for the first time.
 It will be cached for all ulterior uses.
Collecting charset-normalizer==2.0.12
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Installing collected packages: charset-normalizer
Successfully installed charset-normalizer-2.0.12
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
--- PIP INSTALL ---

integrations @ git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library is being installed for the first time.
 It will be cached for all ulterior uses.
flock: cannot open lock file /tmp/windmill/cache/lock/pip-library @ git+https://gitlab+deploy-token-MY_SECRET_TOKEN@gitlab.mycompany.net/repo/location.git/#subdirectory=my/shared/library.lock: No such file or directory

rubenf•11mo ago

oh that's an easy fix away! @guillaume , we need to replace ('/') with _ in the path to create the lock context, we use lock for pip to work well in context of multiple workers doing concurrent pip installs

guillaume•11mo ago

I can take a look in a bit

Alper•11mo ago

thanks a lot

guillaume•11mo ago

I'm looking into it, I think it's a bit more tricky than doing a replace_all, but I'll have something for you soon @Alper the fix merged yesterday, if you pull latest you should be able to use python package from git repo

Alper•11mo ago

thanks @guillaume! is it available for the helm chart as well? it looks like the latest release is 3 days old

rubenf•11mo ago

not yet, will do a release asap

Unknown User•11mo ago

Message Not Public

rubenf•11mo ago

you can't @victorL87

Unknown User•11mo ago

Message Not Public

rubenf•11mo ago

I agree with you, but implementation wise it would have terrible consequences if we made the lockfile something that you evaluate later on rather than a raw value I don't have a good solution right now other than: - you should scope that token in a way that it doesn't have more privilege than just reading that repo - use private pypi registry and set that at the instance level

Unknown User•11mo ago

Message Not Public

rubenf•11mo ago

it actually shoudn't install them on the worker's system pip and it does some variation of what you describe

rubenf•11mo ago

We have our own concept of pip caches that is derived from our global cache needs. We had to create the concept of piptars for it: https://www.windmill.dev/docs/misc/s3_cache

S3 Distributed Dependency Cache | Windmill

Workers cache aggressively the dependencies (and each version of them since every script has its own lockfile with a specific version for each dependency) so they are never pulled nor installed twice on the same worker. However, with a bigger cluster, for each script, the likelihood of being seen by a worker for the first time increases (and the...

rubenf•11mo ago

you can find all the desired dependency in /tmp/windmill/cache/pip/...

Alper•11mo ago

the reason why Victor asks is that we get lots of these while the worker installs the requirements:

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

rubenf•11mo ago

we do run pip as root, but we do not store it on the root default layout but it's not really ran as root, it's ran as the docker user in our case the warning can be very safely ignored

Alper•11mo ago

okay thanks. also for fixing the issue so quickly