i did, using `docker compose down && docker compose up -d`

AAlper i did, using `docker compose down && docker compose up -d`

R

rubenf•10/30/23, 12:11 PM

It's from a container that has not restarted

R

rubenf•10/30/23, 12:11 PM

Or that isn't on same version

JJim Is there a way to see when a worker is busy?

R

rubenf•10/30/23, 12:12 PM

It's exposed in the Prometheus metrics

A

AlperOP•10/30/23, 12:18 PM

@rubenf you're right i had to manually

docker pull

docker pull

docker pull

docker pull to ensure that all containers are up to date

new problem now though as the web app is still not showing up. docker logs for caddy show this:

caddy-1                   | {"level":"error","ts":1698668186.8462517,"logger":"http.log.error","msg":"dial tcp 172.24.0.15:8000: connect: connection refused","request":{"remote_ip":"99.222.22.99","remote_port":"11906","proto":"HTTP/1.1","method":"GET","host":"something.net:9000","uri":"/","headers":{"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7,lb;q=0.6"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Cookie":[]}},"duration":0.002089665,"status":502,"err_id":"0rzabz7tt","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}

caddy-1                   | {"level":"error","ts":1698668186.8462517,"logger":"http.log.error","msg":"dial tcp 172.24.0.15:8000: connect: connection refused","request":{"remote_ip":"99.222.22.99","remote_port":"11906","proto":"HTTP/1.1","method":"GET","host":"something.net:9000","uri":"/","headers":{"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7,lb;q=0.6"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Cookie":[]}},"duration":0.002089665,"status":502,"err_id":"0rzabz7tt","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}

caddy-1                   | {"level":"error","ts":1698668186.8462517,"logger":"http.log.error","msg":"dial tcp 172.24.0.15:8000: connect: connection refused","request":{"remote_ip":"99.222.22.99","remote_port":"11906","proto":"HTTP/1.1","method":"GET","host":"something.net:9000","uri":"/","headers":{"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7,lb;q=0.6"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Cookie":[]}},"duration":0.002089665,"status":502,"err_id":"0rzabz7tt","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}

caddy-1                   | {"level":"error","ts":1698668186.8462517,"logger":"http.log.error","msg":"dial tcp 172.24.0.15:8000: connect: connection refused","request":{"remote_ip":"99.222.22.99","remote_port":"11906","proto":"HTTP/1.1","method":"GET","host":"something.net:9000","uri":"/","headers":{"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7,lb;q=0.6"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Cookie":[]}},"duration":0.002089665,"status":502,"err_id":"0rzabz7tt","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}

S

Sindre•10/30/23, 12:20 PM

if the connection is refused you should check the logs on the server. I assume there is some kind of error message there.

A

AlperOP•10/30/23, 12:22 PM

this is the output from

docker compose logs

docker compose logs

therefore there are no other server logs than those from

caddy

caddy

caddy

caddy as seen above

S

Sindre•10/30/23, 12:26 PM

no other server logs? you should def be seeing other logs.
what does

docker logs windmill_server

docker logs windmill_server

docker logs windmill_server

docker logs windmill_server where "windmill_server" is the container show? nothing?

A

AlperOP•10/30/23, 12:32 PM

# docker compose logs windmill_server
windmill_server-1  | 2023-10-30T12:14:11.758240Z  INFO windmill: Mode not specified, defaulting to standalone
windmill_server-1  | 2023-10-30T12:14:11.758295Z  INFO windmill: Connecting to database...
windmill_server-1  | 2023-10-30T12:14:11.784252Z  INFO windmill: Database connected
windmill_server-1  | 2023-10-30T12:14:11.786103Z  INFO windmill: PostgreSQL version: PostgreSQL 14.9 (Debian 14.9-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit (windmill require PG >= 14)

# docker compose logs windmill_server
windmill_server-1  | 2023-10-30T12:14:11.758240Z  INFO windmill: Mode not specified, defaulting to standalone
windmill_server-1  | 2023-10-30T12:14:11.758295Z  INFO windmill: Connecting to database...
windmill_server-1  | 2023-10-30T12:14:11.784252Z  INFO windmill: Database connected
windmill_server-1  | 2023-10-30T12:14:11.786103Z  INFO windmill: PostgreSQL version: PostgreSQL 14.9 (Debian 14.9-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit (windmill require PG >= 14)

# docker compose logs windmill_server
windmill_server-1  | 2023-10-30T12:14:11.758240Z  INFO windmill: Mode not specified, defaulting to standalone
windmill_server-1  | 2023-10-30T12:14:11.758295Z  INFO windmill: Connecting to database...
windmill_server-1  | 2023-10-30T12:14:11.784252Z  INFO windmill: Database connected
windmill_server-1  | 2023-10-30T12:14:11.786103Z  INFO windmill: PostgreSQL version: PostgreSQL 14.9 (Debian 14.9-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit (windmill require PG >= 14)

# docker compose logs windmill_server
windmill_server-1  | 2023-10-30T12:14:11.758240Z  INFO windmill: Mode not specified, defaulting to standalone
windmill_server-1  | 2023-10-30T12:14:11.758295Z  INFO windmill: Connecting to database...
windmill_server-1  | 2023-10-30T12:14:11.784252Z  INFO windmill: Database connected
windmill_server-1  | 2023-10-30T12:14:11.786103Z  INFO windmill: PostgreSQL version: PostgreSQL 14.9 (Debian 14.9-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit (windmill require PG >= 14)

nothing more than that

A

AlperOP•10/30/23, 12:32 PM

logs suspicously similar to windmill worker outputs

A

AlperOP•10/30/23, 12:34 PM

this is the docker compose section for that service:

  windmill_server:
    image: ${WM_IMAGE}
    pull_policy: always
    deploy:
      replicas: 1
    restart: unless-stopped
    expose:
      - 8000
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - RUST_LOG=info
      ## You can set the number of workers to 1 and not need any separate worker service but not recommended
      - NUM_WORKERS=0
      - DISABLE_SERVER=false
      - METRICS_ADDR=false # (ee only, if set to true, metrics will be exposed on port 8001)
      # LICENSE_KEY is only needed for the enterprise edition
      - LICENSE_KEY=${WM_LICENSE_KEY}
    depends_on:
      db:
        condition: service_healthy

  windmill_server:
    image: ${WM_IMAGE}
    pull_policy: always
    deploy:
      replicas: 1
    restart: unless-stopped
    expose:
      - 8000
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - RUST_LOG=info
      ## You can set the number of workers to 1 and not need any separate worker service but not recommended
      - NUM_WORKERS=0
      - DISABLE_SERVER=false
      - METRICS_ADDR=false # (ee only, if set to true, metrics will be exposed on port 8001)
      # LICENSE_KEY is only needed for the enterprise edition
      - LICENSE_KEY=${WM_LICENSE_KEY}
    depends_on:
      db:
        condition: service_healthy

  windmill_server:
    image: ${WM_IMAGE}
    pull_policy: always
    deploy:
      replicas: 1
    restart: unless-stopped
    expose:
      - 8000
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - RUST_LOG=info
      ## You can set the number of workers to 1 and not need any separate worker service but not recommended
      - NUM_WORKERS=0
      - DISABLE_SERVER=false
      - METRICS_ADDR=false # (ee only, if set to true, metrics will be exposed on port 8001)
      # LICENSE_KEY is only needed for the enterprise edition
      - LICENSE_KEY=${WM_LICENSE_KEY}
    depends_on:
      db:
        condition: service_healthy

  windmill_server:
    image: ${WM_IMAGE}
    pull_policy: always
    deploy:
      replicas: 1
    restart: unless-stopped
    expose:
      - 8000
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - RUST_LOG=info
      ## You can set the number of workers to 1 and not need any separate worker service but not recommended
      - NUM_WORKERS=0
      - DISABLE_SERVER=false
      - METRICS_ADDR=false # (ee only, if set to true, metrics will be exposed on port 8001)
      # LICENSE_KEY is only needed for the enterprise edition
      - LICENSE_KEY=${WM_LICENSE_KEY}
    depends_on:
      db:
        condition: service_healthy

R

rubenf•10/30/23, 12:34 PM

Do docker ps -a

A

AlperOP•10/30/23, 12:37 PM

message.txt3.38KB

R

rubenf•10/30/23, 12:38 PM

Can you just show the relevant containers part

A

AlperOP•10/30/23, 12:40 PM

what do you mean? above are all containers from the docker-compose file. this is just the server:

CONTAINER ID   IMAGE                                       COMMAND                  CREATED          STATUS                    PORTS                                                                                  NAMES
ab9bd601d95b   ghcr.io/windmill-labs/windmill:main         "windmill"               47 seconds ago   Up 33 seconds             8000/tcp                                                                               windmill_server-1

CONTAINER ID   IMAGE                                       COMMAND                  CREATED          STATUS                    PORTS                                                                                  NAMES
ab9bd601d95b   ghcr.io/windmill-labs/windmill:main         "windmill"               47 seconds ago   Up 33 seconds             8000/tcp                                                                               windmill_server-1

CONTAINER ID   IMAGE                                       COMMAND                  CREATED          STATUS                    PORTS                                                                                  NAMES
ab9bd601d95b   ghcr.io/windmill-labs/windmill:main         "windmill"               47 seconds ago   Up 33 seconds             8000/tcp                                                                               windmill_server-1

CONTAINER ID   IMAGE                                       COMMAND                  CREATED          STATUS                    PORTS                                                                                  NAMES
ab9bd601d95b   ghcr.io/windmill-labs/windmill:main         "windmill"               47 seconds ago   Up 33 seconds             8000/tcp                                                                               windmill_server-1

R

rubenf•10/30/23, 12:48 PM

docker logs ab9bd601d95b

R

rubenf•10/30/23, 1:24 PM

one small issue is that it's one more state and more queries to sync

R

rubenf•10/30/23, 1:24 PM

which means it degrade performance overall

R

rubenf•10/30/23, 1:24 PM

but I think it can be done

A

AlperOP•10/30/23, 1:26 PM

message.txt23KB

R

rubenf•10/30/23, 1:27 PM

looks like it's running fine

A

AlperOP•10/30/23, 1:27 PM

whoops, you're right, it works now. startup took like 30 minutes though

R

rubenf•10/30/23, 1:27 PM

and that somehow your caddy is not proxying to it

AAlper i did, using `docker compose down && docker compose up -d`

S

Sindre•10/30/23, 1:34 PM

tip: I do

docker compose pull && docker compose up -d --no-deps

docker compose pull && docker compose up -d --no-deps

docker compose pull && docker compose up -d --no-deps

docker compose pull && docker compose up -d --no-deps That way the service is still running when you pull and it do not restart services that do not get a new image. that helped a lot for me getting the services to start faster. Caddy rarly needs to be restarted. If I remember correctly it creats a new cert each time it started by default.

R

rubenf•10/30/23, 1:35 PM

The issue here is that your migration was particularly long which might be the case if you didn't update for a while and had lots of completed_jobs

A

AlperOP•10/30/23, 1:56 PM

can confirm that base64 display works in latest version

R

rubenf•10/30/23, 2:04 PM

unauthorized means the token (as bearer token or query arg) was not passed or is invalid

R

rubenf•10/30/23, 2:50 PM

@Alpius did you set baseUrl in the helm values?

R

rubenf•10/30/23, 2:55 PM

no that's way too specific

R

rubenf•10/30/23, 2:56 PM

Did you set any instance settings and saved ? If yes, did you also change the base url in those settings ?

R

rubenf•10/30/23, 2:57 PM

instance settings are shown during setup

R

rubenf•10/30/23, 3:01 PM

@Alpius can you look at the response from the login endpoint when you login

R

rubenf•10/30/23, 3:02 PM

if it's 200 and if the domain it sets the cookie to is the right one

R

rubenf•10/30/23, 3:02 PM

also can you look at your cookies and check the domain

R

rubenf•10/30/23, 3:06 PM

yeah could you look at the response headers of login

R

rubenf•10/30/23, 3:06 PM

just want to make sure it's a base url issue

R

rubenf•10/30/23, 3:10 PM

there should be a domain in there

R

rubenf•10/30/23, 3:10 PM

also I noticed you have an envoy proxy in there

R

rubenf•10/30/23, 3:11 PM

can you confirm that the request asking for workspaces is not passing your token?

R

rubenf•10/30/23, 3:11 PM

(as a cookie)

R

rubenf•10/30/23, 3:12 PM

if it is, then it's your proxy not passing the auth token to the final server

i did, using `docker compose down && docker compose up -d`

Similar Threads

Similar Threads

Similar Threads