Postgres trigger: replication slot LSNs never advance — WAL retention grows unbounded
Environment
Windmill CE v1.686.0 on AWS EKS (Helm)
Postgres: Aurora PostgreSQL 14.17
Single publication + single logical slot (windmill_slot), output plugin pgoutput
Trigger on INSERT/UPDATE for a small set of tables
Symptom
INSERT/UPDATE events are received correctly and the bound workflows complete end-to-end with no errors. However, confirmed_flush_lsn and restart_lsn on the slot stay static. Retained WAL grows continuously and eventually exceeds max_slot_wal_keep_size.
Evidence
1. Slot state — both LSNs identical and frozen from connection start
2. Slot recreate experiment
Disabled the Windmill trigger → confirmed slot inactive → pg_drop_replication_slot('windmill_slot') → re-enabled the trigger → slot was recreated by Windmill, active = true. LSNs immediately freeze at the new starting position. Firing fresh INSERT/UPDATE events runs the workflows successfully but does not advance the LSNs.
3. Filter is not the cause
Reconfigured the trigger to watch specific tables vs. all tables — identical frozen-LSN behavior in both cases.
4. No replication-protocol activity in any Windmill pod logs
kubectl logs across all Windmill pods (windmill-app ×2, windmill-extra, windmill-workers-default ×2, windmill-workers-native ×5) filtered for confirm|flush|ack|lsn|replic|standby|keepalive|pgoutput|slot|wal returns nothing related to the replication protocol
Questions
Is this a known issue in Windmill CE v1.686.0? Anyone running Postgres triggers long-term who has not hit unbounded WAL growth?
Is there a config option to enable/force confirm_flush_lsn from the trigger consumer, or is the expectation that operators advance the slot externally?
Happy to share more diagnostics, run a debug build, or test a patch.
Windmill CE v1.686.0 on AWS EKS (Helm)
Postgres: Aurora PostgreSQL 14.17
Single publication + single logical slot (windmill_slot), output plugin pgoutput
Trigger on INSERT/UPDATE for a small set of tables
Symptom
INSERT/UPDATE events are received correctly and the bound workflows complete end-to-end with no errors. However, confirmed_flush_lsn and restart_lsn on the slot stay static. Retained WAL grows continuously and eventually exceeds max_slot_wal_keep_size.
Evidence
1. Slot state — both LSNs identical and frozen from connection start
2. Slot recreate experiment
Disabled the Windmill trigger → confirmed slot inactive → pg_drop_replication_slot('windmill_slot') → re-enabled the trigger → slot was recreated by Windmill, active = true. LSNs immediately freeze at the new starting position. Firing fresh INSERT/UPDATE events runs the workflows successfully but does not advance the LSNs.
3. Filter is not the cause
Reconfigured the trigger to watch specific tables vs. all tables — identical frozen-LSN behavior in both cases.
4. No replication-protocol activity in any Windmill pod logs
kubectl logs across all Windmill pods (windmill-app ×2, windmill-extra, windmill-workers-default ×2, windmill-workers-native ×5) filtered for confirm|flush|ack|lsn|replic|standby|keepalive|pgoutput|slot|wal returns nothing related to the replication protocol
Questions
Is this a known issue in Windmill CE v1.686.0? Anyone running Postgres triggers long-term who has not hit unbounded WAL growth?
Is there a config option to enable/force confirm_flush_lsn from the trigger consumer, or is the expectation that operators advance the slot externally?
Happy to share more diagnostics, run a debug build, or test a patch.
