berbw
berbw•7d ago

Synchronize Hub Ressource types with instance throws "Error: Could not reauth"

Hi there! We are just getting started evaluating Windmill self-hosted for a possible EE-deployment. We are currently running version Windmill CE v1.478.1-5-g46797f072 behind an explicit proxy. Proxy settings are passed to the docker containers via env variables HTTP_PROXY=http://redacted.local:3128, HTTPS_PROXY=http://redacted.local:3128, NO_PROXY=localhost,windmill.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16. I can see in the proxy logs, that Windmill uses the explicit proxy for outbound connections. Running the default Script Synchronize Hub Resource types with instance results in an error and Windmill Resources are not loaded. However: Windmill Hub Scripts/Flows/Apps are loaded correctly. What could cause this? Thank you! 
{
    "error": {
        "name": "Error",
        "stack": "Error: Could not reauth\n    at <anonymous> (/tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a:57432:22)\n    at processTicksAndRejections (native:7:39)",
        "message": "Could not reauth"
    }
}
{
    "error": {
        "name": "Error",
        "stack": "Error: Could not reauth\n    at <anonymous> (/tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a:57432:22)\n    at processTicksAndRejections (native:7:39)",
        "message": "Could not reauth"
    }
}
Logs: 
job=0195b845-2aaf-e9e3-60cb-56ad91bb69ef tag=bun worker=wk-default-c67bf71951c7-SxCO2 hostname=c67bf71951c7

loaded from local cache: /tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a


--- BUN BUNDLE SNAPSHOT EXECUTION ---

INFO ! Could not reach API given existing credentials. Attempting to reauth...
INFO Not a TTY, can't login interactively.
57427 |   try {
57428 |     return await exports_deps.UserService.globalWhoami();
57429 |   } catch {
57430 |     exports_mod9.info("! Could not reach API given existing credentials. Attempting to reauth...");
57431 |     let newToken = await loginInteractive(workspace.remote);
57432 |       throw new Error("Could not reauth");
                    ^
error: Could not reauth
      at <anonymous> (/tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a:57432:13)
job=0195b845-2aaf-e9e3-60cb-56ad91bb69ef tag=bun worker=wk-default-c67bf71951c7-SxCO2 hostname=c67bf71951c7

loaded from local cache: /tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a


--- BUN BUNDLE SNAPSHOT EXECUTION ---

INFO ! Could not reach API given existing credentials. Attempting to reauth...
INFO Not a TTY, can't login interactively.
57427 |   try {
57428 |     return await exports_deps.UserService.globalWhoami();
57429 |   } catch {
57430 |     exports_mod9.info("! Could not reach API given existing credentials. Attempting to reauth...");
57431 |     let newToken = await loginInteractive(workspace.remote);
57432 |       throw new Error("Could not reauth");
                    ^
error: Could not reauth
      at <anonymous> (/tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a:57432:13)
7 Replies
Hugo
Hugo•7d ago
are the proxy settings also passed to the workers?
berbw
berbwOP•7d ago
Thank you for the reply! Yes, to server, workers and native-workers. The error happens more or less instantaneously - it does not appear like the application runs into a timeout. I just deleted all containers and volumes and started from scratch for further testing. The first run of the script also successfully installs the windmill-cli and some packages but then again runs into the error. 
job=0195b8c9-8fd2-83cf-172e-ad6985afb475 tag=bun worker=wk-default-6e8fbbe3dc86-8ZSod hostname=6e8fbbe3dc86

bun install v1.2.4 (fd9a5ea6)
Saved lockfile
+ windmill-cli@1.393.2
103 packages installed [6.13s]

--- BUN CODE EXECUTION ---


wrote cached binary: /tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a (backed by EE distributed object store: false)


INFO ! Could not reach API given existing credentials. Attempting to reauth...
INFO Not a TTY, can't login interactively.
57427 |   try {
57428 |     return await exports_deps.UserService.globalWhoami();
57429 |   } catch {
57430 |     exports_mod9.info("! Could not reach API given existing credentials. Attempting to reauth...");
57431 |     let newToken = await loginInteractive(workspace.remote);
57432 |       throw new Error("Could not reauth");
                    ^
error: Could not reauth
      at <anonymous> (/tmp/windmill/wk-default-6e8fbbe3dc86-8ZSod/0195b8c9-8fd2-83cf-172e-ad6985afb475/main.js:57432:13)
job=0195b8c9-8fd2-83cf-172e-ad6985afb475 tag=bun worker=wk-default-6e8fbbe3dc86-8ZSod hostname=6e8fbbe3dc86

bun install v1.2.4 (fd9a5ea6)
Saved lockfile
+ windmill-cli@1.393.2
103 packages installed [6.13s]

--- BUN CODE EXECUTION ---


wrote cached binary: /tmp/windmill/cache/bun/c568fcc2fbf82e82cd6db9f9a40abb6366c832daece3b3364c8028f9f293ae9a (backed by EE distributed object store: false)


INFO ! Could not reach API given existing credentials. Attempting to reauth...
INFO Not a TTY, can't login interactively.
57427 |   try {
57428 |     return await exports_deps.UserService.globalWhoami();
57429 |   } catch {
57430 |     exports_mod9.info("! Could not reach API given existing credentials. Attempting to reauth...");
57431 |     let newToken = await loginInteractive(workspace.remote);
57432 |       throw new Error("Could not reauth");
                    ^
error: Could not reauth
      at <anonymous> (/tmp/windmill/wk-default-6e8fbbe3dc86-8ZSod/0195b8c9-8fd2-83cf-172e-ad6985afb475/main.js:57432:13)
rubenf
rubenf•7d ago
can you try to curl/ping the target domain in bash and see the errors you get if you get any
berbw
berbwOP•7d ago
I can reach the target URL of the Windmill Hub hub.windmill.dev from within the docker container. Downloading Windmill Hub Scripts/Flows/App works. Network connectivity to the Internet/... works. It's only the Windmill Resources that are not downloaded. (And I of course don't know if other steps of the Script are not executed and therefore something else is not working as it should)
rubenf
rubenf•7d ago
>Windmill Hub hub.windmill.dev from within the docker container. Downloading Windmill Hub Scripts/Flows/App works. Network connectivity to the Internet/... works. How can you tell
berbw
berbwOP•7d ago
If the proxy envs are passed to the container Scripts, etc. are downloaded, but resources are not added. If i don't pass the proxy envs neither is downloaded and the Synchronize Hub Ressource script runs into a network connectivity timeout. If I docker exec -it windmill-server /bin/sh into the container I can curl hub.windmill.dev Screenshot of the added Scripts/Flows/Apps after executing the script "Synchronize Hub Resource types with instance" (that results in the mentioned error) and the ressources that are not added:
No description
No description
berbw
berbwOP•6d ago
Have to split this message into two message, since it exceeds the 2.000 character limit: Part 1: Just did some more testing/digging around, and managed to solve my issue but came across a suspected issue in the way Windmill or a dependency of Windmill uses the NO_PROXY envs. Since the error is thrown in windmill/cli/context.ts requireLogin I suspected, that the error is a local issue. I however was able to successfully query the API of my Windmill server from my machine. 
$headers=@{}
$headers.Add("Authorization", "Bearer redacted")
$response = Invoke-WebRequest -Uri 'https://redacted.local/api/w/test/resources/list' -Method GET -Headers $headers
$headers=@{}
$headers.Add("Authorization", "Bearer redacted")
$response = Invoke-WebRequest -Uri 'https://redacted.local/api/w/test/resources/list' -Method GET -Headers $headers
Results in $response.Content containing the default app theme of my test workspace 
[{"workspace_id":"test","path":"f/app_themes/theme_0","value":null,"description":"The default app theme","resource_type":"app_theme","extra_perms":{},"created_by":"berbw","edite
d_at":"2025-03-22T08:02:02.841596Z","is_linked":false,"is_refreshed":null,"is_oauth":null,"is_expired":null,"refresh_error":null,"account":null}]
[{"workspace_id":"test","path":"f/app_themes/theme_0","value":null,"description":"The default app theme","resource_type":"app_theme","extra_perms":{},"created_by":"berbw","edite
d_at":"2025-03-22T08:02:02.841596Z","is_linked":false,"is_refreshed":null,"is_oauth":null,"is_expired":null,"refresh_error":null,"account":null}]
API access therefore works. Part 2: I then created a simple Windmill script to query a not existing resource via the API, to see if the Windmill worker can access the Windmill API. 
import * as wmill from "windmill-client"

export async function main() {
  return wmill.getResource('does-not-exist');
}
import * as wmill from "windmill-client"

export async function main() {
  return wmill.getResource('does-not-exist');
}
This resulted in the explicit proxy returning an error 403 Forbidden. The requested URL was http://127.0.0.1:42783/api/w/admins/resources/get_value_interpolated/does-not-exist. The windmill CLI did not correctly use the env NO_PROXY=localhost,windmill.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16. The env contains 127.0.0.0/8 but windmill CLI used a proxy for the connection to 127.0.0.1 - the Windmill server itself. I therefore explicitly added 127.0.0.1 to the env NO_PROXY=localhost,windmill.local,127.0.0.1,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 and was then able to successfully execute my test script without a proxy error. The script now correctly does not use a Proxy for connections to 127.0.0.1 and throws an error Resource not found at does-not-exist or not visible to you: Not found: Resource does-not-exist not found - which is expected, since the resource does not exist. 🙂 I afterwards executed the script Synchronize Hub Resource types with instance which now also correctly imports the Hub Resources (additionally to the Scripts/etc. which already worked). My issue is therefore solved. It seems to me, that the way Windmill or a dependency of Windmill processes the NO_PROXY env does not take into account entries in CIDR notation.

Did you find this page helpful?