huschplay
huschplay•4w ago

Can't get Python to work with TLS Interception, always UnknownIssuer

Hi guys, after updating to the current release, all Python applications fail to download their dependencies.
Error while installing certifi==2024.12.14:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/certifi/`
Caused by: Request failed after 3 retries
Caused by: error sending request for url (https://pypi.org/simple/certifi/)
Caused by: client error (Connect)
Caused by: invalid peer certificate: UnknownIssuer
Error while installing certifi==2024.12.14:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/certifi/`
Caused by: Request failed after 3 retries
Caused by: error sending request for url (https://pypi.org/simple/certifi/)
Caused by: client error (Connect)
Caused by: invalid peer certificate: UnknownIssuer
Before the update, everything worked fine (to note, I just downgraded and tested again, the issue persists. So I am unsure if this is actually related to the update). I have the following env's in place.
- INIT_SCRIPT=/tmp/use_ca.sh
- REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
- DENO_TLS_CA_STORE=system,mozilla
- INIT_SCRIPT=/tmp/use_ca.sh
- REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
- DENO_TLS_CA_STORE=system,mozilla
my init_script runs and adds the certificates to the system store. If I run either pip install httpx or uv pip install httpx in the shell of the worker, I don't face the issue. I feel like the worker is creating a separate environment that does not respect the placed env's. I also addedWHITELIST_ENVSwith no effect. If I run a simple Python script that prints the values env's above, the values are printed correctly. I do not have any issue pulling dependencies for rust or typescript. Any help would be appreciated.
17 Replies
rubenf
rubenf•4w ago
yes the change to uv likely make ca not work the same as before are you an existing ee customer (for priority backlog) ?
huschplay
huschplayOP•4w ago
I am not 😦
rubenf
rubenf•4w ago
we will add to backlog but it might take a bit of time
huschplay
huschplayOP•4w ago
all right, the confirmation that this is releated to the update is already some kind of help, thx.
huschplay
huschplayOP•4w ago
GitHub
windmill/backend/windmill-worker/src/python_executor.rs at e47dd697...
Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal...
rubenf
rubenf•4w ago
yes but we will remove it altogether in the coming release (we're removing pip support to simplify our support burden)
huschplay
huschplayOP•4w ago
I can understand that, but still, that would be rather unfortunate for ce users without any other workaround. Should USE_PIP_COMPILE be set as env and than just work? Or is there some clean up todo as the uv lock file is already present in the PostgreSQL (that is how I understand it from a brief look at the code)
resolving dependencies...
content of requirements:
wmill

Fallback to pip-compile (Deprecated!)

--- UV PIP INSTALL ---

To be installed:

anyio==4.8.0
certifi==2024.12.14
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
idna==3.10
sniffio==1.3.1
typing-extensions==4.12.2
wmill==1.448.1

Starting installation... (20 tasks in parallel)

Error while installing wmill==1.448.1:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/wmill/`
Caused by: Request failed after 3 retries
Caused by: error sending request for url (https://pypi.org/simple/wmill/)
Caused by: client error (Connect)
Caused by: invalid peer certificate: UnknownIssuer
resolving dependencies...
content of requirements:
wmill

Fallback to pip-compile (Deprecated!)

--- UV PIP INSTALL ---

To be installed:

anyio==4.8.0
certifi==2024.12.14
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
idna==3.10
sniffio==1.3.1
typing-extensions==4.12.2
wmill==1.448.1

Starting installation... (20 tasks in parallel)

Error while installing wmill==1.448.1:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/wmill/`
Caused by: Request failed after 3 retries
Caused by: error sending request for url (https://pypi.org/simple/wmill/)
Caused by: client error (Connect)
Caused by: invalid peer certificate: UnknownIssuer
for new scripts it does attempt to fallback to pip but looks like it is still using uv anyway got it, I also had to add USE_PIP_INSTALL
Pyra
Pyra•4w ago
@huschplay I think you can use pip until we resolve the issue, once we done you can simply update to latest
huschplay
huschplayOP•4w ago
thx, I will do that. Could you mention that as breaking change, once you remove it?
Pyra
Pyra•3w ago
I can, yes. I will write to changelog when pip is removed @huschplay Hello, I think issue should be resolved on latest. If you have installed certs to system store, you can try to set PY_NATIVE_CERT=true so uv uses it (by default it uses it's own). If it does not work, you can try PY_INDEX_CERT=/custom-certs/root-ca.crt If that does work neither, you can whitelist your domain PY_TRUSTED_HOST=pypi.org I hope that works 😊 Let me know if it does or not
huschplay
huschplayOP•3w ago
thx, for letting me know. I will check next week and get back to you
huschplay
huschplayOP•3w ago
I can confirm this is working, thank you very much. I would suggest updating the documentation at https://www.windmill.dev/docs/advanced/self_host#configure-python-requests--httpx-trust
Self-host | Windmill
Self-host Windmill on your own infrastructure.
huschplay
huschplayOP•3w ago
I will update the issue "feature: add support for custom/corporate certificate authorities #1564", so people are aware
Pyra
Pyra•3w ago
Glad to know. Which solution did the trick in the end? According to your last edit on gh issue, it was PY_NATIVE_CERT=true?
huschplay
huschplayOP•3w ago
hi yes, that is correct I mount the folder /usr/local/share/ca-certificates:ro from the host system and use a init_script to update the certs on the worker
Pyra
Pyra•7d ago
@huschplay Hello again 👋 Would you mind helping us a little bit? We have to verify if uv can work with self-signed certificates or not. :) Are you using self-signed certificates or they are signed by someone else?
huschplay
huschplayOP•5d ago
We have a firewall doing TLS interception with a custom CA, which is enrolled on all devices. So not self signed in the usal way @Pyra let me know If I can test something for you

Did you find this page helpful?