Can't get Python to work with TLS Interception, always UnknownIssuer

Hi guys, after updating to the current release, all Python applications fail to download their dependencies.

Error while installing certifi==2024.12.14:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/certifi/`
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://pypi.org/simple/certifi/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

Error while installing certifi==2024.12.14:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/certifi/`
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://pypi.org/simple/certifi/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

Before the update, everything worked fine (to note, I just downgraded and tested again, the issue persists. So I am unsure if this is actually related to the update). I have the following env's in place.

      - INIT_SCRIPT=/tmp/use_ca.sh
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
      - DENO_TLS_CA_STORE=system,mozilla

      - INIT_SCRIPT=/tmp/use_ca.sh
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
      - DENO_TLS_CA_STORE=system,mozilla

my init_script runs and adds the certificates to the system store. If I run either

pip install httpx

pip install httpx

uv pip install httpx

uv pip install httpx

in the shell of the worker, I don't face the issue. I feel like the worker is creating a separate environment that does not respect the placed env's. I also added

WHITELIST_ENVS

WHITELIST_ENVS

with no effect. If I run a simple Python script that prints the values env's above, the values are printed correctly. I do not have any issue pulling dependencies for rust or typescript. Any help would be appreciated.

rubenf•1/24/25, 10:30 AM

yes the change to uv likely make ca not work the same as before

rubenf•1/24/25, 10:31 AM

are you an existing ee customer (for priority backlog) ?

huschplayOP•1/24/25, 10:33 AM

I am not

rubenf•1/24/25, 10:33 AM

we will add to backlog but it might take a bit of time

huschplayOP•1/24/25, 10:34 AM

all right, the confirmation that this is releated to the update is already some kind of help, thx.

huschplayOP•1/24/25, 11:33 AM

is USE_PIP_COMPILE=true available for ce users? https://github.com/windmill-labs/windmill/blob/e47dd697f96883f8f97bf469e69d3344f12d9633/backend/windmill-worker/src/python_executor.rs#L469 and whould it help in this case?

GitHub

windmill/backend/windmill-worker/src/python_executor.rs at e47dd697...

Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal...

rubenf•1/24/25, 11:35 AM

yes

rubenf•1/24/25, 11:35 AM

but we will remove it altogether in the coming release

rubenf•1/24/25, 11:36 AM

(we're removing pip support to simplify our support burden)

huschplayOP•1/24/25, 11:39 AM

I can understand that, but still, that would be rather unfortunate for ce users without any other workaround. Should

USE_PIP_COMPILE

USE_PIP_COMPILE

be set as env and than just work? Or is there some clean up todo as the uv lock file is already present in the PostgreSQL (that is how I understand it from a brief look at the code)

huschplayOP•1/24/25, 11:44 AM

resolving dependencies...
content of requirements:
wmill

Fallback to pip-compile (Deprecated!)

--- UV PIP INSTALL ---

To be installed: 

anyio==4.8.0 
certifi==2024.12.14 
h11==0.14.0 
httpcore==1.0.7 
httpx==0.28.1 
idna==3.10 
sniffio==1.3.1 
typing-extensions==4.12.2 
wmill==1.448.1 

Starting installation... (20 tasks in parallel) 

Error while installing wmill==1.448.1:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/wmill/`
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://pypi.org/simple/wmill/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

resolving dependencies...
content of requirements:
wmill

Fallback to pip-compile (Deprecated!)

--- UV PIP INSTALL ---

To be installed: 

anyio==4.8.0 
certifi==2024.12.14 
h11==0.14.0 
httpcore==1.0.7 
httpx==0.28.1 
idna==3.10 
sniffio==1.3.1 
typing-extensions==4.12.2 
wmill==1.448.1 

Starting installation... (20 tasks in parallel) 

Error while installing wmill==1.448.1:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/wmill/`
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://pypi.org/simple/wmill/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

for new scripts it does attempt to fallback to pip but looks like it is still using uv anyway

huschplayOP•1/24/25, 11:48 AM

got it, I also had to add USE_PIP_INSTALL

Pyra•1/24/25, 11:57 AM

@huschplay I think you can use pip until we resolve the issue, once we done you can simply update to latest

huschplayOP•1/24/25, 12:00 PM

thx, I will do that. Could you mention that as breaking change, once you remove it?

Hhuschplay thx, I will do that. Could you mention that as breaking change, once you remove ...

Pyra•1/24/25, 12:01 PM

I can, yes. I will write to changelog when pip is removed

Pyra•1/31/25, 11:31 AM

@huschplay Hello, I think issue should be resolved on latest. If you have installed certs to system store, you can try to set

PY_NATIVE_CERT=true

PY_NATIVE_CERT=true

so uv uses it (by default it uses it's own).

If it does not work, you can try

PY_INDEX_CERT=/custom-certs/root-ca.crt

PY_INDEX_CERT=/custom-certs/root-ca.crt

If that does work neither, you can whitelist your domain

PY_TRUSTED_HOST=pypi.org

PY_TRUSTED_HOST=pypi.org

I hope that works

Let me know if it does or not

huschplayOP•1/31/25, 10:27 PM

thx, for letting me know. I will check next week and get back to you

huschplayOP•2/3/25, 11:22 AM

I can confirm this is working, thank you very much. I would suggest updating the documentation at https://www.windmill.dev/docs/advanced/self_host#configure-python-requests--httpx-trust

Self-host | Windmill

Self-host Windmill on your own infrastructure.

huschplayOP•2/3/25, 11:24 AM

I will update the issue "feature: add support for custom/corporate certificate authorities #1564", so people are aware

Hhuschplay I can confirm this is working, thank you very much. I would suggest updating the...

Pyra•2/3/25, 11:36 AM

Glad to know. Which solution did the trick in the end?

Pyra•2/3/25, 11:37 AM

According to your last edit on gh issue, it was

PY_NATIVE_CERT=true

PY_NATIVE_CERT=true

huschplayOP•2/3/25, 11:49 AM

hi yes, that is correct

huschplayOP•2/3/25, 11:51 AM

I mount the folder

/usr/local/share/ca-certificates:ro

/usr/local/share/ca-certificates:ro

from the host system and use a init_script to update the certs on the worker

Pyra•2/13/25, 4:19 PM

@huschplay Hello again

Would you mind helping us a little bit? We have to verify if uv can work with self-signed certificates or not. :) Are you using self-signed certificates or they are signed by someone else?

huschplayOP•2/15/25, 7:22 PM

We have a firewall doing TLS interception with a custom CA, which is enrolled on all devices. So not self signed in the usal way

huschplayOP•2/15/25, 7:23 PM

@Pyra let me know If I can test something for you

Pyra•2/20/25, 2:27 PM

Thank you for information @huschplay . Original Issue is solved, so it is ok.

You asked for announcement when we remove fallback to pip. So fallback to pip since 1.466.0 is no more.