Aleksey Posts - Windmill

Aleksey

•Created by Aleksey on 12/12/2024 in #help

Having trouble running python scripts that depend on packages from AWS codeartifact.

We're an enterprise customer, Vareto, and we self host. We use amazon web services (AWS) CodeArtifact to publish our python packages, and we need Windmill to use our index when setting up python environments. The problem is that AWS expects the url to contain a token, and the token expires after at most 12 hours. Using the enterprise "Pip Index Url" and "Pip Extra Index Url" features doesn't solve the problem because there is no way to periodically update them with a new token. In the past, we got around this by having windmill periodically trigger a script on the workers which would update the $HOME/.config/pip/pip.conf file on the worker with a new index-url. Similar to this:

aws sts get-caller-identity

export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token <other required flags> --query authorizationToken --output text)

pip config --global set global.extra-index-url "https://aws:${CODEARTIFACT_TOKEN}@<vareto repo address>"

aws sts get-caller-identity

export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token <other required flags> --query authorizationToken --output text)

pip config --global set global.extra-index-url "https://aws:${CODEARTIFACT_TOKEN}@<vareto repo address>"

However, UV doesn't use the pip.conf file. From a security standpoint, this is no doubt better, and UV's performance is phenomenal, but we no longer have a way of pulling packages from CodeArtifact. I have tried using a URL without the token in the "Pip Index Url" and "Pip Extra Index Url" eneterprise settings, but it doesn't work:

https://user-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/our-python/simple

https://user-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/our-python/simple

I've also tried setting the UV_INDEX_URL, UV_EXTRA_INDEX_URL, PIP_INDEX_URL, and PIP_EXTRA_INDEX_URL environment variables on both the server and the worker AWS task definitions. It does pick them up, and I see them appear on the UV command line, but it appears to revert to pypi anyway (perhaps it goes to pypi after it fails to connect to CodeArtifact?) I'm currently attempting to work around this by setting the USE_PIP_COMPILE and USE_PIP_INSTALL environmental variables so we can go back to using the $HOME/.config/pip/pip.conf file for now. Can you please point me in the right direction?

8 replies