pixeleet•9mo ago

Worker stuck in migration mode

Error: Migrating database: while executing migrations: error returned from database: cannot execute CREATE TABLE in a read-only transaction

3 Replies

pixeleet•9mo ago

{"timestamp":"2024-02-15T09:17:32.843504Z","level":"INFO","message":"Binary is in 'worker' mode","target":"windmill"}
{"timestamp":"2024-02-15T09:17:32.843580Z","level":"INFO","message":"Connecting to database...","target":"windmill"}
{"timestamp":"2024-02-15T09:17:33.038938Z","level":"INFO","message":"Database connected","target":"windmill"}
{"timestamp":"2024-02-15T09:17:33.043342Z","level":"INFO","message":"PostgreSQL version: PostgreSQL 15.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20231011 (Red Hat 13.2.1-4), 64-bit (windmill require PG >= 14)","target":"windmill"}
{"timestamp":"2024-02-15T09:17:33.044931Z","level":"INFO","message":"Started locking PG for migration purposes","target":"windmill_api::db"}
{"timestamp":"2024-02-15T09:17:33.044962Z","level":"INFO","message":"Locked PG for migration purposes","target":"windmill_api::db"}

{"timestamp":"2024-02-15T09:17:32.843504Z","level":"INFO","message":"Binary is in 'worker' mode","target":"windmill"}
{"timestamp":"2024-02-15T09:17:32.843580Z","level":"INFO","message":"Connecting to database...","target":"windmill"}
{"timestamp":"2024-02-15T09:17:33.038938Z","level":"INFO","message":"Database connected","target":"windmill"}
{"timestamp":"2024-02-15T09:17:33.043342Z","level":"INFO","message":"PostgreSQL version: PostgreSQL 15.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20231011 (Red Hat 13.2.1-4), 64-bit (windmill require PG >= 14)","target":"windmill"}
{"timestamp":"2024-02-15T09:17:33.044931Z","level":"INFO","message":"Started locking PG for migration purposes","target":"windmill_api::db"}
{"timestamp":"2024-02-15T09:17:33.044962Z","level":"INFO","message":"Locked PG for migration purposes","target":"windmill_api::db"}

I am using an external DB and the app connects without an issue, but the workers since yesterday started misbehaving.

{"timestamp":"2024-02-15T09:27:13.310325Z","level":"ERROR","message":"Error deleting token: error returned from database: cannot execute DELETE in a read-only transaction","target":"windmill::monitor"}
{"timestamp":"2024-02-15T09:27:13.311775Z","level":"ERROR","message":"Error deleting pip_resolution: error returned from database: cannot execute DELETE in a read-only transaction","target":"windmill::monitor"}
{"timestamp":"2024-02-15T09:27:13.313390Z","level":"ERROR","message":"Error deleting cache resource error returned from database: cannot execute DELETE in a read-only transaction","target":"windmill::monitor"}
{"timestamp":"2024-02-15T09:27:13.315804Z","level":"ERROR","message":"Error deleting expired job stats: Database(PgDatabaseError { severity: Error, code: \"25006\", message: \"cannot execute DELETE in a read-only transaction\", detail: None, hint: None, position: None, where: None, schema: None, table: None, column: None, data_type: None, constraint: None, file: Some(\"utility.c\"), line: Some(414), routine: Some(\"PreventCommandIfReadOnly\") })","target":"windmill::monitor"}

{"timestamp":"2024-02-15T09:27:13.310325Z","level":"ERROR","message":"Error deleting token: error returned from database: cannot execute DELETE in a read-only transaction","target":"windmill::monitor"}
{"timestamp":"2024-02-15T09:27:13.311775Z","level":"ERROR","message":"Error deleting pip_resolution: error returned from database: cannot execute DELETE in a read-only transaction","target":"windmill::monitor"}
{"timestamp":"2024-02-15T09:27:13.313390Z","level":"ERROR","message":"Error deleting cache resource error returned from database: cannot execute DELETE in a read-only transaction","target":"windmill::monitor"}
{"timestamp":"2024-02-15T09:27:13.315804Z","level":"ERROR","message":"Error deleting expired job stats: Database(PgDatabaseError { severity: Error, code: \"25006\", message: \"cannot execute DELETE in a read-only transaction\", detail: None, hint: None, position: None, where: None, schema: None, table: None, column: None, data_type: None, constraint: None, file: Some(\"utility.c\"), line: Some(414), routine: Some(\"PreventCommandIfReadOnly\") })","target":"windmill::monitor"}

rubenf•9mo ago

It's saying the user used for the database url is read only

pixeleet•9mo ago

yeah totally my bad we ran out of space 😄 thank you for getting back to me