Add simple VACUUM progress reporting.

There's a lot more that could be done here yet - in particular, this
reports only very coarse-grained information about the index vacuuming
phase - but even as it stands, the new pg_stat_progress_vacuum can
tell you quite a bit about what a long-running vacuum is actually
doing.

Amit Langote and Robert Haas, based on earlier work by Vinayak Pokale
and Rahila Syed.

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index ec5328ea8fdbe8139d0f5da3530d31a016a6a98a..de79fde6be3d6b7a396de901c992be5a3cc2372c 100644 (file)
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -507,6 +507,12 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       yet included in pg_stat_user_functions).
      
 
+     
+      pg_stat_progress_vacuumpg_stat_progress_vacuum
+      One row for each backend (including autovacuum worker processes) running
+      VACUUM, showing current progress.
+      See .
+     
     
    
   
@@ -2490,6 +2496,207 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
   
  
 
+ 
+  Progress Reporting
+
+  
+   PostgreSQL has the ability to report the progress of
+   certain commands during command execution.  Currently, the only command
+   which supports progress reporting is VACUUM.  This may be
+   expanded in the future.
+  
+
+ 
+  VACUUM Progress Reporting
+
+  
+   Whenever VACUUM is running, the
+   pg_stat_progress_vacuum view will contain
+   one row for each backend (including autovacuum worker processes) that is
+   currently vacuuming.  The tables below describe the information
+   that will be reported and provide information about how to interpret it.
+   Progress reporting is not currently supported for VACUUM FULL
+   and backends running VACUUM FULL will not be listed in this
+   view.
+  
+
+  
+   <structname>pg_stat_progress_vacuum</structname> View
+   
+    
+    
+      Column
+      Type
+      Description
+     
+    
+
+   
+    
+     pid
+     integer
+     Process ID of backend.
+    
+    
+     datid
+     oid
+     OID of the database to which this backend is connected.
+    
+    
+     datname
+     name
+     Name of the database to which this backend is connected.
+    
+    
+     relid
+     oid
+     OID of the table being vacuumed.
+    
+    
+     phase
+     text
+     
+       Current processing phase of vacuum.  See .
+     
+    
+    
+     heap_blks_total
+     bigint
+     
+       Total number of heap blocks in the table.  This number is reported
+       as of the beginning of the scan; blocks added later will not be (and
+       need not be) visited by this VACUUM.
+     
+    
+    
+     heap_blks_scanned
+     bigint
+     
+       Number of heap blocks scanned.  Because the
+       visibility map is used to optimize scans,
+       some blocks will be skipped without inspection; skipped blocks are
+       included this total, so that this number will eventually become
+       equal to heap_blks_total when the vacuum is complete.
+       This counter only advances when the phase is scanning heap.
+     
+    
+    
+     heap_blks_vacuumed
+     bigint
+     
+       Number of heap blocks vacuumed.  Unless the table has no indexes, this
+       counter only advances when the phase is vacuuming heap.
+       Blocks that contain no dead tuples are skipped, so the counter may
+       sometimes skip forward in large increments.
+     
+    
+    
+     index_vacuum_count
+     bigint
+     
+       Number of completed index vacuums cycles.
+     
+    
+    
+     max_dead_tuples
+     bigint
+     
+      Number of dead tuples that we can store before needing to perform
+      an index vacuum cycle, based on
+      .
+     
+    
+    
+     num_dead_tuples
+     bigint
+     
+       Number of dead tuples collected since the last index vacuum cycle.
+     
+    
+   
+   
+  
+
+  
+   VACUUM phases
+   
+    
+    
+      Phase
+      Description
+     
+    
+
+   
+    
+     initializing
+     
+       VACUUM is preparing to begin scanning the heap.  This
+       phase is expected to be very brief.
+     
+    
+    
+     scanning heap
+     
+       VACUUM is currently scanning the heap.  It will prune and
+       defragment each page if required, and possibly perform freezing
+       activity.  The heap_blks_scanned column can be used
+       to monitor the progress of the scan.
+     
+    
+    
+     vacuuming indexes
+     
+       VACUUM is currently vacuuming the indexes.  If a table has
+       any indexes, this will happen at least once per vacuum, after the heap
+       has been completely scanned.  It may happen multiple times per vacuum
+       if  is insufficient to
+       store the number of dead tuples found.
+     
+    
+    
+     vacuuming heap
+     
+       VACUUM is currently vacuuming the heap.  Vacuuming the heap
+       is distinct from scanning the heap, and occurs after each instance of
+       vacuuming indexes.  If heap_blks_scanned is less than
+       heap_blks_total, the system will return to scanning
+       the heap after this phase is completed; otherwise, it will begin
+       cleaning up indexes after this phase is completed.
+     
+    
+    
+     cleaning up indexes
+     
+       VACUUM is currently cleaning up indexes.  This occurs after
+       the heap has been completely scanned and all vacuuming of the indexes
+       and the heap has been completed.
+     
+    
+    
+     truncating heap
+     
+       VACUUM is currently truncating the heap so as to return
+       empty pages at the end of the relation to the operating system.  This
+       occurs after cleaning up indexes.
+     
+    
+    
+     performing final cleanup
+     
+       VACUUM is performing final cleanup.  During this phase,
+       VACUUM will vacuum the free space map, update statistics
+       in pg_class, and report statistics to the statistics
+       collector.  When this phase is completed, VACUUM will end.
+     
+    
+   
+   
+  
+
+ 
+ 
+
  
   Dynamic Tracing
 

diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 84aa06148efccc73fef09dfb3d00f706f33a0a56..fef67bdd4cdc1f9c7012fa5a96c2abd81499ac62 100644 (file)
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -796,6 +796,24 @@ CREATE VIEW pg_stat_bgwriter AS
         pg_stat_get_buf_alloc() AS buffers_alloc,
         pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
 
+CREATE VIEW pg_stat_progress_vacuum AS
+   SELECT
+       S.pid AS pid, S.datid AS datid, D.datname AS datname,
+       S.relid AS relid,
+       CASE S.param1 WHEN 0 THEN 'initializing'
+                     WHEN 1 THEN 'scanning heap'
+                     WHEN 2 THEN 'vacuuming indexes'
+                     WHEN 3 THEN 'vacuuming heap'
+                     WHEN 4 THEN 'cleaning up indexes'
+                     WHEN 5 THEN 'truncating heap'
+                     WHEN 6 THEN 'performing final cleanup'
+                     END AS phase,
+       S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
+       S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+       S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+    FROM pg_stat_get_progress_info('VACUUM') AS S
+        JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,

diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index fe87243f4c53b71fe9dc3d52659c089fd729fcb7..163c121560d20dc47eb7c8f286d2c5ce02ee2785 100644 (file)
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -48,6 +48,7 @@
 #include "catalog/catalog.h"
 #include "catalog/storage.h"
 #include "commands/dbcommands.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "pgstat.h"
@@ -272,6 +273,10 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params,
    if (should_attempt_truncation(vacrelstats))
        lazy_truncate_heap(onerel, vacrelstats);
 
+   /* Report that we are now doing final cleanup */
+   pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                PROGRESS_VACUUM_PHASE_FINAL_CLEANUP);
+
    /* Vacuum the Free Space Map */
    FreeSpaceMapVacuum(onerel);
 
@@ -457,6 +462,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
    bool        skipping_blocks;
    xl_heap_freeze_tuple *frozen;
    StringInfoData buf;
+   const int   initprog_index[] = {
+       PROGRESS_VACUUM_PHASE,
+       PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
+       PROGRESS_VACUUM_MAX_DEAD_TUPLES
+   };
+   int64       initprog_val[3];
 
    pg_rusage_init(&ru0);
 
@@ -481,6 +492,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
    lazy_space_alloc(vacrelstats, nblocks);
    frozen = palloc(sizeof(xl_heap_freeze_tuple) * MaxHeapTuplesPerPage);
 
+   /* Report that we're scanning the heap, advertising total # of blocks */
+   initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
+   initprog_val[1] = nblocks;
+   initprog_val[2] = vacrelstats->max_dead_tuples;
+   pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
+
    /*
     * Except when aggressive is set, we want to skip pages that are
     * all-visible according to the visibility map, but only when we can skip
@@ -572,6 +589,8 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
 #define FORCE_CHECK_PAGE() \
        (blkno == nblocks - 1 && should_attempt_truncation(vacrelstats))
 
+       pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
+
        if (blkno == next_unskippable_block)
        {
            /* Time to advance next_unskippable_block */
@@ -652,6 +671,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
        if ((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
            vacrelstats->num_dead_tuples > 0)
        {
+           const int   hvp_index[] = {
+               PROGRESS_VACUUM_PHASE,
+               PROGRESS_VACUUM_NUM_INDEX_VACUUMS
+           };
+           int64       hvp_val[2];
+
            /*
             * Before beginning index vacuuming, we release any pin we may
             * hold on the visibility map page.  This isn't necessary for
@@ -667,11 +692,26 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
            /* Log cleanup info before we touch indexes */
            vacuum_log_cleanup_info(onerel, vacrelstats);
 
+           /* Report that we are now vacuuming indexes */
+           pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                        PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+
            /* Remove index entries */
            for (i = 0; i < nindexes; i++)
                lazy_vacuum_index(Irel[i],
                                  &indstats[i],
                                  vacrelstats);
+
+           /*
+            * Report that we are now vacuuming the heap.  We also increase
+            * the number of index scans here; note that by using
+            * pgstat_progress_update_multi_param we can update both
+            * parameters atomically.
+            */
+           hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
+           hvp_val[1] = vacrelstats->num_index_scans + 1;
+           pgstat_progress_update_multi_param(2, hvp_index, hvp_val);
+
            /* Remove tuples from heap */
            lazy_vacuum_heap(onerel, vacrelstats);
 
@@ -682,6 +722,10 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
             */
            vacrelstats->num_dead_tuples = 0;
            vacrelstats->num_index_scans++;
+
+           /* Report that we are once again scanning the heap */
+           pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                        PROGRESS_VACUUM_PHASE_SCAN_HEAP);
        }
 
        /*
@@ -1182,6 +1226,10 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
            RecordPageWithFreeSpace(onerel, blkno, freespace);
    }
 
+   /* report that everything is scanned and vacuumed */
+   pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
+   pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_VACUUMED, blkno);
+
    pfree(frozen);
 
    /* save stats for use later */
@@ -1208,19 +1256,41 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
    /* XXX put a threshold on min number of tuples here? */
    if (vacrelstats->num_dead_tuples > 0)
    {
+       const int   hvp_index[] = {
+           PROGRESS_VACUUM_PHASE,
+           PROGRESS_VACUUM_NUM_INDEX_VACUUMS
+       };
+       int64       hvp_val[2];
+
        /* Log cleanup info before we touch indexes */
        vacuum_log_cleanup_info(onerel, vacrelstats);
 
+       /* Report that we are now vacuuming indexes */
+       pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                    PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+
        /* Remove index entries */
        for (i = 0; i < nindexes; i++)
            lazy_vacuum_index(Irel[i],
                              &indstats[i],
                              vacrelstats);
+
+       /* Report that we are now vacuuming the heap */
+       hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
+       hvp_val[1] = vacrelstats->num_index_scans + 1;
+       pgstat_progress_update_multi_param(2, hvp_index, hvp_val);
+
        /* Remove tuples from heap */
+       pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                    PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
        lazy_vacuum_heap(onerel, vacrelstats);
        vacrelstats->num_index_scans++;
    }
 
+   /* report we're now in the cleanup phase */
+   pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+
    /* Do post-vacuum cleanup and statistics update for each index */
    for (i = 0; i < nindexes; i++)
        lazy_cleanup_index(Irel[i], indstats[i], vacrelstats);
@@ -1350,6 +1420,8 @@ lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
    TransactionId visibility_cutoff_xid;
    bool        all_frozen;
 
+   pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_VACUUMED, blkno);
+
    START_CRIT_SECTION();
 
    for (; tupindex < vacrelstats->num_dead_tuples; tupindex++)
@@ -1607,6 +1679,10 @@ lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats)
 
    pg_rusage_init(&ru0);
 
+   /* Report that we are now truncating */
+   pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+                                PROGRESS_VACUUM_PHASE_TRUNCATE);
+
    /*
     * Loop until no more truncating can be done.
     */
@@ -1887,6 +1963,8 @@ lazy_record_dead_tuple(LVRelStats *vacrelstats,
    {
        vacrelstats->dead_tuples[vacrelstats->num_dead_tuples] = *itemptr;
        vacrelstats->num_dead_tuples++;
+       pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+                                    vacrelstats->num_dead_tuples);
    }
 }
 

diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 14afef61fef27ce6fa940f29f32207ce83ecdf11..bfe70fc41dbe687bc025251fc46756a34f9b1157 100644 (file)
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -2902,6 +2902,35 @@ pgstat_progress_update_param(int index, int64 val)
    pgstat_increment_changecount_after(beentry);
 }
 
+/*-----------
+ * pgstat_progress_update_params() -
+ *
+ * Automatically update multiple members in st_progress_param[] of own backend
+ * entry.
+ *-----------
+ */
+void
+pgstat_progress_update_multi_param(int nparam, const int *index,
+                                  const int64 *val)
+{
+   volatile PgBackendStatus *beentry = MyBEEntry;
+   int     i;
+
+   if (!beentry || !pgstat_track_activities || nparam == 0)
+       return;
+
+   pgstat_increment_changecount_before(beentry);
+
+   for (i = 0; i < nparam; ++i)
+   {
+       Assert(index[i] >= 0 && index[i] < PGSTAT_NUM_PROGRESS_PARAM);
+
+       beentry->st_progress_param[index[i]] = val[i];
+   }
+
+   pgstat_increment_changecount_after(beentry);
+}
+
 /*-----------
  * pgstat_progress_end_command() -
  *

diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 8cfc9a4ca98c4e1d655c84478dd81a2e4deb50c5..7f410c178c54c85f8b3e14729229c7f7de000547 100644 (file)
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
  */
 
 /*                         yyyymmddN */
-#define CATALOG_VERSION_NO 201603111
+#define CATALOG_VERSION_NO 201603151
 
 #endif

diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
new file mode 100644 (file)
index 0000000..c48d23b
--- /dev/null
+++ b/src/include/commands/progress.h
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * progress.h
+ *   Constants used with the progress reporting facilities defined in
+ *   pgstat.h.  These are possibly interesting to extensions, so we
+ *   expose them via this header file.  Note that if you update these
+ *   constants, you probably also need to update the views based on them
+ *   in system_views.sql.
+ *
+ * Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/commands/progress.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PROGRESS_H
+#define PROGRESS_H
+
+/* Progress parameters for (lazy) vacuum */
+#define PROGRESS_VACUUM_PHASE                  0
+#define PROGRESS_VACUUM_TOTAL_HEAP_BLKS            1
+#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED      2
+#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED     3
+#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS      4
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLES            5
+#define PROGRESS_VACUUM_NUM_DEAD_TUPLES            6
+
+/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
+#define PROGRESS_VACUUM_PHASE_SCAN_HEAP            1
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX     2
+#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP      3
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP        4
+#define PROGRESS_VACUUM_PHASE_TRUNCATE         5
+#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP        6
+
+#endif

diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e7fbf1e39267c8b9f4c2ffaa6b18c3c14e4cabbc..1fc45acc23918dc0db19f4d7642b402b4da24107 100644 (file)
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -978,6 +978,8 @@ extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
 extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
                              Oid relid);
 extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_update_multi_param(int nparam, const int *index,
+                                  const int64 *val);
 extern void pgstat_progress_end_command(void);
 
 extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id);

diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 22ea06c150553d375ed49ae55055f3a7748593f9..79f9b2325649c53645f3b68adb2f122c38a064fa 100644 (file)
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1747,6 +1747,28 @@ pg_stat_database_conflicts| SELECT d.oid AS datid,
     pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin,
     pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock
    FROM pg_database d;
+pg_stat_progress_vacuum| SELECT s.pid,
+    s.datid,
+    d.datname,
+    s.relid,
+        CASE s.param1
+            WHEN 0 THEN 'initializing'::text
+            WHEN 1 THEN 'scanning heap'::text
+            WHEN 2 THEN 'vacuuming indexes'::text
+            WHEN 3 THEN 'vacuuming heap'::text
+            WHEN 4 THEN 'cleaning up indexes'::text
+            WHEN 5 THEN 'truncating heap'::text
+            WHEN 6 THEN 'performing final cleanup'::text
+            ELSE NULL::text
+        END AS phase,
+    s.param2 AS heap_blks_total,
+    s.param3 AS heap_blks_scanned,
+    s.param4 AS heap_blks_vacuumed,
+    s.param5 AS index_vacuum_count,
+    s.param6 AS max_dead_tuples,
+    s.param7 AS num_dead_tuples
+   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+     JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_replication| SELECT s.pid,
     s.usesysid,
     u.rolname AS usename,