Fix handling of synchronous replication for stopping WAL senders

This fixes an oversight from c6c3334 which has introduced a more strict
ordering in the way WAL senders are stopped to prevent current WAL
activity when a shutdown checkpoint is created.  After all backends are
stopped, all WAL senders are requested to stop which makes them stop any
activity, and switching their state as stopping.  Once the checkpointer
knows that all WAL senders are in a stopping state, the shutdown
checkpoint can begin, with all WAL senders activated, waiting for their
clients to flush the shutdown checkpoint record.

If a subset of WAL senders are stopping and in a sync state, other WAL
senders could still be waiting for a WAL position to be synced while
committing a transaction, however the subset of stopping senders would
not release waiters, potentially breaking synchronous replication
guarantees.  This commit makes sure that even WAL senders stopping are
able to release waiters properly.

On 9.4, this can also trigger an assertion failure when setting for
example max_wal_senders to 1 where a WAL sender is not able to find
itself as in synchronous state when the instance stops.

Reported-by: Paul Guo
Author: Paul Guo, Michael Paquier
Discussion: https://postgr.es/m/CAEET0ZEv8VFqT3C-cQm6byOB4r4VYWcef1J21dOX-gcVhCSpmA@mail.gmail.com
Backpatch-through: 9.4

diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index caf16efa168f4a83a5520a7fdc4c58a96062eea0..651525f59f5785782b15734f80b7d562ea8913a9 100644 (file)
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -379,10 +379,12 @@ SyncRepReleaseWaiters(void)
     * If this WALSender is serving a standby that is not on the list of
     * potential sync standbys then we have nothing to do. If we are still
     * starting up, still running base backup or the current flush position
-    * is still invalid, then leave quickly also.
+    * is still invalid, then leave quickly also. Streaming or stopping WAL
+    * senders are allowed to release waiters.
     */
    if (MyWalSnd->sync_standby_priority == 0 ||
-       MyWalSnd->state < WALSNDSTATE_STREAMING ||
+       (MyWalSnd->state != WALSNDSTATE_STREAMING &&
+        MyWalSnd->state != WALSNDSTATE_STOPPING) ||
        XLogRecPtrIsInvalid(MyWalSnd->flush))
        return;
 
@@ -400,7 +402,8 @@ SyncRepReleaseWaiters(void)
        volatile WalSnd *walsnd = &walsndctl->walsnds[i];
 
        if (walsnd->pid != 0 &&
-           walsnd->state == WALSNDSTATE_STREAMING &&
+           (walsnd->state == WALSNDSTATE_STREAMING ||
+            walsnd->state == WALSNDSTATE_STOPPING) &&
            walsnd->sync_standby_priority > 0 &&
            (priority == 0 ||
             priority > walsnd->sync_standby_priority) &&

diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 4c7d55488ed0517c9ab4e22a0ee85d881f73848d..2e944a93905f915ec216695940f535f4442a265c 100644 (file)
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -2941,12 +2941,14 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
            /*
             * Treat a standby such as a pg_basebackup background process
             * which always returns an invalid flush location, as an
-            * asynchronous standby.
+            * asynchronous standby.  WAL sender must be streaming or
+            * stopping.
             */
            sync_priority[i] = XLogRecPtrIsInvalid(walsnd->flush) ?
                0 : walsnd->sync_standby_priority;
 
-           if (walsnd->state == WALSNDSTATE_STREAMING &&
+           if ((walsnd->state == WALSNDSTATE_STREAMING ||
+                walsnd->state == WALSNDSTATE_STOPPING) &&
                walsnd->sync_standby_priority > 0 &&
                (priority == 0 ||
                 priority > walsnd->sync_standby_priority) &&