Having trouble running python scripts that depend on packages from AWS codeartifact.
We're an enterprise customer, Vareto, and we self host.
We use amazon web services (AWS) CodeArtifact to publish our python packages, and we need Windmill to use our index when setting up python environments. The problem is that AWS expects the url to contain a token, and the token expires after at most 12 hours. Using the enterprise "Pip Index Url" and "Pip Extra Index Url" features doesn't solve the problem because there is no way to periodically update them with a new token.
In the past, we got around this by having windmill periodically trigger a script on the workers which would update the $HOME/.config/pip/pip.conf file on the worker with a new index-url. Similar to this:
However, UV doesn't use the pip.conf file. From a security standpoint, this is no doubt better, and UV's performance is phenomenal, but we no longer have a way of pulling packages from CodeArtifact.
I have tried using a URL without the token in the "Pip Index Url" and "Pip Extra Index Url" eneterprise settings, but it doesn't work:
I've also tried setting the UV_INDEX_URL, UV_EXTRA_INDEX_URL, PIP_INDEX_URL, and PIP_EXTRA_INDEX_URL environment variables on both the server and the worker AWS task definitions. It does pick them up, and I see them appear on the UV command line, but it appears to revert to pypi anyway (perhaps it goes to pypi after it fails to connect to CodeArtifact?)
I'm currently attempting to work around this by setting the USE_PIP_COMPILE and USE_PIP_INSTALL environmental variables so we can go back to using the $HOME/.config/pip/pip.conf file for now.
Can you please point me in the right direction?
4 Replies
if you use the string EPHEMERAL_TOKEN in your pip url and pass as env variable:
EPHEMERAL_TOKEN_CMD it will replace EPHEMERAL_TOKEN with the output of that command
that command should be used to generate a token for codeartifact using the aws clientOh wow. So to make sure I understand correctly...
I would put this into the "Pip Index Url" and "Pip Extra Index Url" configuration options:
and set this as a custom contextual variable (or maybe as a secret?)
https://www.windmill.dev/docs/core_concepts/variables_and_secrets#custom-contextual-variables
(pseudocode b/c this would be in the windmill UI)
?
Followups:
1) Does windmill try to evaluate all variables as commands, or only one named EPHEMERAL_TOKEN?
2) What shell does windmill use/emulate when it evaluates them? bash?
3) Is a custom contextual variable the right call, or should I use a secret for this?
Thanks!
Variables and secrets | Windmill
When writing scripts, you may want to reuse variables, or safely pass secrets to
you have to set it as env variable of your worker, it can't be done from the UI
only one named EPHEMERAL_TOKEN
it doesn't use bash, it runs commands directly
got it. In our case, we run two commands:
I can try running just the
aws codeartifact command on its own, but if I need the aws sts get-caller-identity command as well, then I'd do that by extending your docker container in another container and adding an executable script that runs multiple commands?