Since helm release 2.0.195 getting logged out

Getting logged out intermittently

18 Replies

rubenf•5mo ago

we can't investigate with just that as info but noted, if we have similar feedbacks we will use that as a note

let me know how I can help, provide a reproduction or some more detailed logs from app maybe? Might have been due to still some old app pods hanging out, will let you know if it keeps happening. It stopped 😛

rubenf•5mo ago

Yes, most likely you're hitting a not authorized at some point which will log you out if you can isolate what not authorized it is, it will help us

pixeleet•5mo ago

hmm nvm it keeps happening

rubenf•5mo ago

which you can get from your server logs

pixeleet•5mo ago

{"timestamp":"2024-05-27T09:55:13.612518Z","level":"INFO","message":"response","latency":"0","status":400,"target":"windmill_api::tracing_init","span":{"method":"GET","uri":"/api/w/voja/jobs_u/completed/get_result_maybe/Unauthorized","name":"request"}}

{"timestamp":"2024-05-27T09:55:13.612518Z","level":"INFO","message":"response","latency":"0","status":400,"target":"windmill_api::tracing_init","span":{"method":"GET","uri":"/api/w/voja/jobs_u/completed/get_result_maybe/Unauthorized","name":"request"}}

rubenf•5mo ago

If you can isolate how you got to make that request that will help us ideally we can repro in a fresh isolated environment

pixeleet•5mo ago

I have a perpetual script that has a token attached to it, that is from my user. That script checks if a flow it triggered has been completed. So I guess that fails.

rubenf•5mo ago

only requests you make from the browser will log you out also this is status 400 you need to look for status 401 or status 403

pixeleet•5mo ago

{"timestamp":"2024-05-27T09:53:00.102601Z","level":"INFO","message":"response","latency":"3","status":200,"target":"windmill_api::tracing_init","span":{"method":"GET","uri":"/api/w/voja/jobs_u/completed/get_result_maybe/018fb974-9439-5aa4-3011-6c287f39e492","name":"request"}} {"timestamp":"2024-05-27T09:53:00.118615Z","level":"INFO","message":"response","latency":"0","status":200,"target":"windmill_api::tracing_init","span":{"method":"GET","uri":"/_app/immutable/nodes/4.BDKe7_wv.js","name":"request"}} {"timestamp":"2024-05-27T09:53:00.184896Z","level":"INFO","message":"response","latency":"0","status":401,"target":"windmill_api::tracing_init","span":{"method":"GET","uri":"/api/workspaces/users","name":"request"}} I am not doing anything really, just log in, then get logged out intermittently

rubenf•5mo ago

How often do you get disconnected ?

pixeleet•5mo ago

Every 30s or so

rubenf•5mo ago

oh, then probably you have a misconfiguration somewhere in your ingress cause otherwise windmill would be unusable for everyone

pixeleet•5mo ago

Didn't change configuration, went from 2.0.178 to 2.0.195 everything seems okay except this Not seeing error logs around the 401, just these random infos

rubenf•5mo ago

Yes but we have hundreds of enterprise customers with many many users themselves, and app.windmill.dev, that are on latest or close so. If it was our helm the issue, we would know like every 30s is very very noticeable for everyone so there is probably something wrong with your setup if you can come back with more info we can investigate

pixeleet•5mo ago

fair point

{"timestamp":"2024-05-27T10:06:48.187597Z","level":"ERROR","message":"job 018fb983-ce87-142d-3948-9e5686d40179 did not succeed: {\"error\":{\"message\":\"Job timed out after no ping from job since 2024-05-27 10:06:13.989558 UTC (ZOMBIE_JOB_TIMEOUT: 30)\",\"name\":\"InternalErr\"}}","target":"windmill_queue::jobs","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}
{"timestamp":"2024-05-27T10:06:48.195166Z","level":"INFO","message":"inserted completed job: 018fb983-ce87-142d-3948-9e5686d40179 (success: false)","job_id":"018fb983-ce87-142d-3948-9e5686d40179","root_job":"\"018fb97a-d240-0e60-d436-b5250ed34ad6\"","path":"f/voja/places_google_maps_handler","job_kind":"Script","started_at":"\"2024-05-27 10:06:13.989558 UTC\"","duration":"34219","permissioned_as":"\"u/david\"","email":"\"david@voja.ai\"","created_by":"david","is_flow_step":true,"language":"Some(Deno)","success":false,"target":"windmill_queue::jobs","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}
{"timestamp":"2024-05-27T10:06:48.195243Z","level":"ERROR","message":"handle job error, updating flow status: Object {\"message\": String(\"Job timed out after no ping from job since 2024-05-27 10:06:13.989558 UTC (ZOMBIE_JOB_TIMEOUT: 30)\"), \"name\": String(\"InternalErr\")}","parent_flow":"018fb97a-d240-0e60-d436-b5250ed34ad6","subflow":"018fb983-ce87-142d-3948-9e5686d40179","target":"windmill_worker::worker","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}
{"timestamp":"2024-05-27T10:06:48.291751Z","level":"INFO","message":"update flow status","id":"018fb97a-d240-0e60-d436-b5250ed34ad6","root_id":"none","target":"windmill_worker::worker_flow","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}

{"timestamp":"2024-05-27T10:06:48.187597Z","level":"ERROR","message":"job 018fb983-ce87-142d-3948-9e5686d40179 did not succeed: {\"error\":{\"message\":\"Job timed out after no ping from job since 2024-05-27 10:06:13.989558 UTC (ZOMBIE_JOB_TIMEOUT: 30)\",\"name\":\"InternalErr\"}}","target":"windmill_queue::jobs","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}
{"timestamp":"2024-05-27T10:06:48.195166Z","level":"INFO","message":"inserted completed job: 018fb983-ce87-142d-3948-9e5686d40179 (success: false)","job_id":"018fb983-ce87-142d-3948-9e5686d40179","root_job":"\"018fb97a-d240-0e60-d436-b5250ed34ad6\"","path":"f/voja/places_google_maps_handler","job_kind":"Script","started_at":"\"2024-05-27 10:06:13.989558 UTC\"","duration":"34219","permissioned_as":"\"u/david\"","email":"\"david@voja.ai\"","created_by":"david","is_flow_step":true,"language":"Some(Deno)","success":false,"target":"windmill_queue::jobs","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}
{"timestamp":"2024-05-27T10:06:48.195243Z","level":"ERROR","message":"handle job error, updating flow status: Object {\"message\": String(\"Job timed out after no ping from job since 2024-05-27 10:06:13.989558 UTC (ZOMBIE_JOB_TIMEOUT: 30)\"), \"name\": String(\"InternalErr\")}","parent_flow":"018fb97a-d240-0e60-d436-b5250ed34ad6","subflow":"018fb983-ce87-142d-3948-9e5686d40179","target":"windmill_worker::worker","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}
{"timestamp":"2024-05-27T10:06:48.291751Z","level":"INFO","message":"update flow status","id":"018fb97a-d240-0e60-d436-b5250ed34ad6","root_id":"none","target":"windmill_worker::worker_flow","span":{"job_id":"018fb983-ce87-142d-3948-9e5686d40179","name":"job_error"}}

This is the only error I see really

rubenf•5mo ago

those are logs from the workers alternatively, you can rollback to 2.0.178 and see if you do not have the error anymore can try on a fresh instance as well, going to 2.0.178, testing, then 2.0.195, testing

pixeleet•5mo ago

Yep, thanks for the support 🙂