Can't get Python to work with TLS Interception, always UnknownIssuer
Hi guys, after updating to the current release, all Python applications fail to download their dependencies.
Before the update, everything worked fine (to note, I just downgraded and tested again, the issue persists. So I am unsure if this is actually related to the update). I have the following env's in place.
my init_script runs and adds the certificates to the system store. If I run either
pip install httpx
or uv pip install httpx
in the shell of the worker, I don't face the issue. I feel like the worker is creating a separate environment that does not respect the placed env's. I also addedWHITELIST_ENVS
with no effect. If I run a simple Python script that prints the values env's above, the values are printed correctly. I do not have any issue pulling dependencies for rust or typescript. Any help would be appreciated.17 Replies
yes the change to uv likely make ca not work the same as before
are you an existing ee customer (for priority backlog) ?
I am not 😦
we will add to backlog but it might take a bit of time
all right, the confirmation that this is releated to the update is already some kind of help, thx.
is USE_PIP_COMPILE=true available for ce users? https://github.com/windmill-labs/windmill/blob/e47dd697f96883f8f97bf469e69d3344f12d9633/backend/windmill-worker/src/python_executor.rs#L469 and whould it help in this case?
GitHub
windmill/backend/windmill-worker/src/python_executor.rs at e47dd697...
Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal...
yes
but we will remove it altogether in the coming release
(we're removing pip support to simplify our support burden)
I can understand that, but still, that would be rather unfortunate for ce users without any other workaround. Should
USE_PIP_COMPILE
be set as env and than just work? Or is there some clean up todo as the uv lock file is already present in the PostgreSQL (that is how I understand it from a brief look at the code)
for new scripts it does attempt to fallback to pip but looks like it is still using uv anyway
got it, I also had to add USE_PIP_INSTALL@huschplay I think you can use pip until we resolve the issue, once we done you can simply update to latest
thx, I will do that. Could you mention that as breaking change, once you remove it?
I can, yes. I will write to changelog when pip is removed
@huschplay Hello, I think issue should be resolved on latest. If you have installed certs to system store, you can try to set
PY_NATIVE_CERT=true
so uv uses it (by default it uses it's own).
If it does not work, you can try PY_INDEX_CERT=/custom-certs/root-ca.crt
If that does work neither, you can whitelist your domain PY_TRUSTED_HOST=pypi.org
I hope that works 😊
Let me know if it does or notthx, for letting me know. I will check next week and get back to you
I can confirm this is working, thank you very much. I would suggest updating the documentation at https://www.windmill.dev/docs/advanced/self_host#configure-python-requests--httpx-trust
Self-host | Windmill
Self-host Windmill on your own infrastructure.
I will update the issue "feature: add support for custom/corporate certificate authorities #1564", so people are aware
Glad to know. Which solution did the trick in the end?
According to your last edit on gh issue, it was
PY_NATIVE_CERT=true
?hi yes, that is correct
I mount the folder
/usr/local/share/ca-certificates:ro
from the host system and use a init_script to update the certs on the worker@huschplay Hello again 👋
Would you mind helping us a little bit? We have to verify if uv can work with self-signed certificates or not. :) Are you using self-signed certificates or they are signed by someone else?
We have a firewall doing TLS interception with a custom CA, which is enrolled on all devices. So not self signed in the usal way
@Pyra let me know If I can test something for you