linkend="postmaster-start-failures">.) The relevant kernel
|
SHMMAX>>
Maximum size of shared memory segment (bytes)>
- 250kB + 8.2 kB * shared_buffers> + 14.2 kB * max_connections> or infinity
+ 250 kB + 8.2 kB * shared_buffers> + 14.2 kB * max_connections> up to infinity
|
|
SEMMNI>>
Maximum number of semaphore identifiers (i.e., sets)>
- >= ceil(max_connections / 16)>
+ at least ceil(max_connections / 16)>
|
SEMMNS>>
Maximum number of semaphores system-wide>
- ceil(max_connections / 16) * 17 + room for other applications>
+ ceil(max_connections / 16) * 17 plus room for other applications>
|
SEMMSL>>
Maximum number of semaphores per set>
- >= 17>
+ at least 17>
|
|
SEMVMX>>
Maximum value of semaphore>
- >= 255 (The default is often 32767, don't change unless asked to.)>
+ at least 255 (The default is often 32767, don't change unless asked to.)>
exist on the system at one time. Hence this parameter must be at
least ceil(max_connections / 16)>. Lowering the number
of allowed connections is a temporary workaround for failures,
- which are usually confusingly worded <quote><errorname>No space
- left on device>>, from the function semget()>.
+ which are usually confusingly worded No space
+ left on device>, from the function semget>.
-
By default, only 4 MB of shared memory is supported. Keep in
mind that shared memory is not pageable; it is locked in RAM.
- To increase the number of shared buffers supported by the
- postmaster, add the following to your kernel configuration
+ To increase the amount of shared memory supported by your
+ system, add the following to your kernel configuration
file. A SHMALL> value of 1024 represents 4 MB of
shared memory. The following increases the maximum shared
memory area to 32 MB:
options "SHMALL=8192"
options "SHMMAX=\(SHMALL*PAGE_SIZE\)"
+ For those running 4.1 or later, just make the above changes,
+ recompile the kernel, and reboot.
- For those running 4.1 or later, just make the above changes,
- recompile the kernel, and reboot. For those running earlier
- releases, use bpatch> to find the
- sysptsize> value in the current kernel. This is
- computed dynamically at boot time.
+ For those running earlier releases, use bpatch> to
+ find the sysptsize> value in the current
+ kernel. This is computed dynamically at boot time.
$ bpatch -r sysptsize>
0x9 = 9>
Next, add SYSPTSIZE> as a hard-coded value in the
kernel configuration file. Increase the value you found using
- <application>bpatch>. Add 1 for every additional 4 MB of
+ <command>bpatch>. Add 1 for every additional 4 MB of
shared memory you desire.
options "SYSPTSIZE=16"
Semaphores>
- You may need to increase the number of semaphores. By default,
-
PostgreSQL> allocates 34 semaphores, which is
- over half the default system total of 60.
-
-
-
- Set the values you want in your kernel configuration file, e.g.:
+ You may need to increase the number of semaphores. By
+ default,
PostgreSQL> allocates 34 semaphores,
+ which is over half the default system total of 60. Set the
+ values you want in your kernel configuration file, e.g.:
options "SEMMNI=40"
options "SEMMNS=240"
-options "SEMUME=40"
-options "SEMMNU=120"
-
+
+
option singular.)
- You might also want to use the <application>sysctl> setting to
+ You might also want to use the <command>sysctl> setting to
lock shared memory into RAM and prevent it from being paged out
to swap, e.g. kern.ipc.shm_use_phys>.
- Alternatively, you can use
- sysctl
- 8, if available, to
- control these parameters. Look for a file called
- /etc/sysctl.conf and add lines like the
- following to it:
+ Alternatively, you can use sysctl, if
+ available, to control these parameters. Look for a file
+ called /etc/sysctl.conf and add lines
+ like the following to it:
kernel.shmall = 134217728
kernel.shmmax = 134217728
This file is usually processed at boot time, but
- <application>sysctl> can also be called
+ <command>sysctl> can also be called
explicitly later.
sysctl -w kern.sysv.shmseg
sysctl -w kern.sysv.shmall
- These values have the same meanings on
MacOS> X
- as those listed for previous operating systems.
In the default configuration, only 512 kB of shared memory per
segment is allowed, which is about enough for
- 12>. To increase the setting, first change directory to
+ 12>. To increase the setting, first change to the directory
/etc/conf/cf.d>. To display the current value of
- SHMMAX>, in bytes, run
+ SHMMAX>, run
./configure -y SHMMAX
- To set a new value for SHMMAX>, run:
+ To set a new value for SHMMAX>, run
./configure SHMMAX=value>
where value> is the new value you want to use
- (in bytes). After setting SHMMAX>, rebuild the kernel
+ (in bytes). After setting SHMMAX>, rebuild the kernel:
./link_unix
/etc/conf/bin/idtune -g SHMMAX
which displays the current, default, minimum, and maximum
- values, in bytes. To set a new value for SHMMAX>,
- run:
+ values. To set a new value for SHMMAX>,
+ run
/etc/conf/bin/idtune SHMMAX value>
where value> is the new value you want to use
(in bytes). After setting SHMMAX>, rebuild the
- kernel
+ kernel:
/etc/conf/bin/idbuild -B
-
(Bourne shells) or
limit (
csh>) is
used to control the resource limits from the command line. On
BSD-derived systems the file /etc/login.conf
- controls the various resource limits set during login. See
- login.conf
- 5 for details. The relevant
+ controls the various resource limits set during login. See the
+ operating system documentation for details. The relevant
parameters are maxproc,
openfiles, and datasize. For
example:
SIGTERM
- After receiving SIGTERM, the postmaster
- disallows new connections, but lets existing backends end their
- work normally. It shuts down only after all of the backends
- terminate normally. This is Smart
+ After receiving SIGTERM, the server
+ disallows new connections, but lets existing sessions end their
+ work normally. It shuts down only after all of the sessions
+ terminate normally. This is the Smart
Shutdown.
SIGINT
- The postmaster disallows new connections and sends all existing
- backends SIGTERM, which will cause them
+ The server disallows new connections and sends all existing
+ server processes SIGTERM, which will cause them
to abort their current transactions and exit promptly. It then
- waits for the backends to exit and finally shuts down. This is
+ waits for the server processes to exit and finally shuts down. This is the
Fast Shutdown.
SIGQUIT
-
This is Immediate Shutdown, which
- will cause the postmaster to send a
- SIGQUIT to all backends and exit
- immediately (without properly shutting itself down). The backends
+ This is the Immediate Shutdown, which
+ will cause the postmaster process to send a
+ SIGQUIT to all child processes and exit
+ immediately (without properly shutting itself down). The child processes
likewise exit immediately upon receiving
SIGQUIT. This will lead to recovery (by
replaying the WAL log) upon next start-up. This is recommended
+
It is best not to use SIGKILL to shut down
- the postmaster. This will prevent the postmaster from releasing
+ the server. This will prevent the server from releasing
shared memory and semaphores, which may then have to be done by
manually.
- The
PID> of the postmaster process can be found using the
-
ps program, or from the file
+ The
PID> of the postmaster process can be found using the
+ ps program, or from the file
postmaster.pid in the data directory. So for
example, to do a fast shutdown:
- The program <application>pg_ctl> is a shell script
+ The program <command>pg_ctl> is a shell script
that provides a more convenient interface for shutting down the
- postmaster.
+ server.
PostgreSQL> has native support for using
SSL> connections to encrypt client/server communications
- for increased security. This requires
-
OpenSSL be installed on both client and
- server systems and support enabled at build time (see
- linkend="installation">).
+ for increased security. This requires that
+
OpenSSL is installed on both client and
+ server systems and
that support in PostgreSQL> is
+ enabled at build time (see linkend="installation">).
With
SSL> support compiled in, the
PostgreSQL> server can be started with
-
SSL> support by setting the parameter
+
SSL> enabled by setting the parameter
ssl to on in postgresql.conf>. When
starting in
SSL> mode, the server will look for the
files server.key> and server.crt> in the
- data directory. These files should contain the server private key
- and certificate respectively. These files must be set up correctly
+ data directory, which should contain the server private key
+ and certificate, respectively. These files must be set up correctly
before an
SSL>-enabled server can start. If the private key is
protected with a passphrase, the server will prompt for the
passphrase and will not start until it has been entered.
The server will listen for both standard and
SSL>
- connections on the same TCP/IP port, and will negotiate with any
+ connections on the same TCP port, and will negotiate with any
connecting client on whether to use
SSL>. See
linkend="client-authentication"> about how to force the server to
require use of
SSL> for certain connections.
openssl req -new -text -out server.req
Fill out the information that openssl> asks for. Make sure
- that you enter the local host name as Common Name; the challenge
- password can be left blank. The script will generate a key that is
+ that you enter the local host name as Common Name>
; the challenge
+ password can be left blank. The programm will generate a key that is
passphrase protected; it will not accept a passphrase that is less
than four characters long. To remove the passphrase (as you must if
you want automatic start-up of the server), run the commands
-
-
Acknowledgement
- Idea taken from an email by Gene Selkov, Jr.
- to a question from Eric Marsden.
-
-
-
One can use
SSH to encrypt the network
connection between clients and a
First make sure that an
SSH server is
- running properly on the same machine as
-
PostgreSQL and that you can log in using
+ running properly on the same machine as the
+
PostgreSQL server and that you can log in using
ssh as some user. Then you can establish a secure
tunnel with a command like this from the client machine:
The first number in the argument, 3333, is the
port number of your end of the tunnel; it can be chosen freely. The
- second number, 5432, is the remote end of the tunnel -- the port
+ second number, 5432, is the remote end of the tunnel: the port
number your server is using. The name or the address in between
the port numbers is the host with the database server you are going
to connect to. In order to connect to the database server using
authentication procedure was set up for this user. In order for the
tunnel setup to succeed you must be allowed to connect via
as if you had attempted to use ssh to set up a
terminal session.
-
+
Write-Ahead Logging (WAL)
-
-
Author
- Vadim Mikheev and Oliver Elphick
-
-
-
-
-
General Description
-
-
Write Ahead Logging (
WAL)
+
Write-Ahead Logging (
WAL)
is a standard approach to transaction logging. Its detailed
description may be found in most (if not all) books about
transaction processing. Briefly,
WAL's central
concept is that changes to data files (where tables and indexes
- reside) must be written only after those changes have been logged -
+ reside) must be written only after those changes have been logged,
that is, when log records have been flushed to permanent
- storage. When we follow this procedure, we do not need to flush
+ storage. If we follow this procedure, we do not need to flush
data pages to disk on every transaction commit, because we know
that in the event of a crash we will be able to recover the
database using the log: any changes that have not been applied to
the data pages will first be redone from the log records (this is
roll-forward recovery, also known as REDO) and then changes made by
uncommitted transactions will be removed from the data pages
- (roll-backward recovery - UNDO).
+ (roll-backward recovery, UNDO).
- 2 id="wal-benefits-now">
-
Immediate Benefits of WAL
+ 1 id="wal-benefits-now">
The first obvious benefit of using
WAL is a
- index tuples pointing to nonexistent table rows
+ index rows pointing to nonexistent table rows
- index tuples lost in split operations
+ index rows lost in split operations
page content in the log if that is required to ensure page
consistency for after-crash recovery.
- 2>
+ 1>
- 2 id="wal-benefits-later">
+ 1 id="wal-benefits-later">
Future Benefits
- UNDO operation is not implemented. This means that changes
+ The UNDO operation is not implemented. This means that changes
made by aborted transactions will still occupy disk space and that
- we still need a permanent pg_clog file to hold
- the status of transactions, since we are not able to re-use
- transaction identifiers. Once UNDO is implemented,
+ a permanent pg_clog file to hold
+ the status of transactions is still needed, since
+ transaction identifiers cannot be reused. Once UNDO is implemented,
pg_clog will no longer be required to be
permanent; it will be possible to remove
pg_clog at shutdown. (However, the urgency of
this concern has decreased greatly with the adoption of a segmented
- storage method for pg_clog --- it is no longer
+ storage method for pg_clog: it is no longer
necessary to keep old pg_clog entries around
forever.)
A difficulty standing in the way of realizing these benefits is that
they require saving
WAL entries for considerable
- periods of time (eg, as long as the longest possible transaction if
+ periods of time (e.g., as long as the longest possible transaction if
transaction UNDO is wanted). The present
WAL
format is extremely bulky since it includes many disk page
snapshots. This is not a serious concern at present, since the
but to achieve these future benefits some sort of compressed
WAL format will be needed.
-
-
-
-
-
Implementation
-
-
WAL is automatically enabled from release 7.1
- onwards. No action is required from the administrator with the
- exception of ensuring that the additional disk-space requirements
- of the
WAL logs are met, and that any necessary
- tuning is done (see ).
-
-
-
WAL logs are stored in the directory
- $PGDATA/pg_xlog, as
- a set of segment files, each 16 MB in size. Each segment is
- divided into 8 kB pages. The log record headers are described in
- access/xlog.h; record content is dependent on
- the type of event that is being logged. Segment files are given
- ever-increasing numbers as names, starting at
- 0000000000000000. The numbers do not wrap, at
- present, but it should take a very long time to exhaust the
- available stock of numbers.
-
-
- The
WAL buffers and control structure are in
- shared memory, and are handled by the backends; they are protected
- by lightweight locks. The demand on shared memory is dependent on the
- number of buffers. The default size of the
WAL
- buffers is 8 buffers of 8 kB each, or 64 kB total.
-
-
- It is of advantage if the log is located on another disk than the
- main database files. This may be achieved by moving the directory,
- pg_xlog, to another location (while the
- postmaster is shut down, of course) and creating a symbolic link
- from the original location in $PGDATA to
- the new location.
-
-
- The aim of
WAL, to ensure that the log is
- written before database records are altered, may be subverted by
- disk drives that falsely report a successful write to the kernel,
- when, in fact, they have only cached the data and not yet stored it
- on the disk. A power failure in such a situation may still lead to
- irrecoverable data corruption. Administrators should try to ensure
- that disks holding
PostgreSQL's
- log files do not make such false reports.
-
-
-
-
Database Recovery with WAL
-
- After a checkpoint has been made and the log flushed, the
- checkpoint's position is saved in the file
- pg_control. Therefore, when recovery is to be
- done, the backend first reads pg_control and
- then the checkpoint record; then it performs the REDO operation by
- scanning forward from the log position indicated in the checkpoint
- record.
- Because the entire content of data pages is saved in the log on the
- first page modification after a checkpoint, all pages changed since
- the checkpoint will be restored to a consistent state.
-
-
- Using pg_control to get the checkpoint
- position speeds up the recovery process, but to handle possible
- corruption of pg_control, we should actually
- implement the reading of existing log segments in reverse order --
- newest to oldest -- in order to find the last checkpoint. This has
- not been implemented, yet.
-
-
-
+
- There are several
WAL-related parameters that
+ There are several
WAL-related
configuration parameters that
affect database performance. This section explains their use.
Consult for details about setting
configuration parameters.
log (known as the redo record) it should start the REDO operation,
since any changes made to data files before that record are already
on disk. After a checkpoint has been made, any log segments written
- before the undo records are no longer needed and can be recycled or
+ before the redo records are no longer needed and can be recycled or
removed. (When
WAL-based
BAR is
implemented, the log segments would be archived before being recycled
or removed.)
- The postmaster spawns a special backend process every so often
+ The server spawns a special process every so often
to create the next checkpoint. A checkpoint is created every
- CHECKPOINT_SEGMENTS log segments, or every
- CHECKPOINT_TIMEOUT seconds, whichever comes first.
+ checkpoint_segments log segments, or every
+ checkpoint_timeout seconds, whichever comes first.
The default settings are 3 segments and 300 seconds respectively.
It is also possible to force a checkpoint by using the SQL command
CHECKPOINT.
- Reducing CHECKPOINT_SEGMENTS and/or
- CHECKPOINT_TIMEOUT causes checkpoints to be done
+ Reducing checkpoint_segments and/or
+ checkpoint_timeout causes checkpoints to be done
more often. This allows faster after-crash recovery (since less work
will need to be redone). However, one must balance this against the
increased cost of flushing dirty data pages more often. In addition,
- There will be at least one 16MB segment file, and will normally
- not be more than 2 * CHECKPOINT_SEGMENTS
- + 1 files. You can use this to estimate space requirements for
- WAL. Ordinarily, when old log segment files are no longer needed,
- they are recycled (renamed to become the next sequential future
- segments). If, due to a short-term peak of log output rate, there
- are more than 2 * CHECKPOINT_SEGMENTS + 1 segment files,
- the unneeded segment files will be deleted instead of recycled until the
- system gets back under this limit.
+ There will be at least one 16 MB segment file, and will normally
+ not be more than 2 * checkpoint_segments + 1
+ files. You can use this to estimate space requirements for WAL.
+ Ordinarily, when old log segment files are no longer needed, they
+ are recycled (renamed to become the next segments in the numbered
+ sequence). If, due to a short-term peak of log output rate, there
+ are more than 2 * checkpoint_segments + 1
+ segment files, the unneeded segment files will be deleted instead
+ of recycled until the system gets back under this limit.
to write (move to kernel cache) a few filled
WAL
buffers. This is undesirable because LogInsert
is used on every database low level modification (for example,
- tuple insertion) at a time when an exclusive lock is held on
+ row insertion) at a time when an exclusive lock is held on
affected data pages, so the operation needs to be as fast as
possible. What is worse, writing
WAL buffers may
also force the creation of a new log segment, which takes even more
not occur often enough to prevent
WAL buffers
being written by LogInsert. On such systems
one should increase the number of
WAL buffers by
- modifying the postgresql.conf
-
WAL_BUFFERS parameter. The default number of
+ modifying the configuration parameter wal_buffers.
WAL buffers is 8. Increasing this value will
correspondingly increase shared memory usage.
buffers to disk using the operating system sync()> call.
Busy servers may fill checkpoint segment files too quickly,
causing excessive checkpointing. If such forced checkpoints happen
- more frequently than CHECKPOINT_WARNING seconds,
+ more frequently than checkpoint_warning seconds,
a message, will be output to the server logs recommending increasing
- CHECKPOINT_SEGMENTS.
+ checkpoint_segments.
- The COMMIT_DELAY parameter defines for how many
- microseconds the backend will sleep after writing a commit
+ The commit_delay parameter defines for how many
+ microseconds the server process will sleep after writing a commit
record to the log with LogInsert but before
performing a LogFlush. This delay allows other
- backends to add their commit records to the log so as to have all
+ server processes to add their commit records to the log so as to have all
of them flushed with a single log sync. No sleep will occur if fsync
- is not enabled or if fewer than COMMIT_SIBLINGS
- other backends are not currently in active transactions; this avoids
- sleeping when it's unlikely that any other backend will commit soon.
+ is not enabled or if fewer than commit_siblings
+ other sessons are currently in active transactions; this avoids
+ sleeping when it's unlikely that any other session will commit soon.
Note that on most platforms, the resolution of a sleep request is
- ten milliseconds, so that any nonzero COMMIT_DELAY
- setting between 1 and 10000 microseconds will have the same effect.
+ ten milliseconds, so that any nonzero commit_delay
+ setting between 1 and 10000 microseconds would have the same effect.
Good values for these parameters are not yet clear; experimentation
is encouraged.
- The WAL_SYNC_METHOD parameter determines how
+ The wal_sync_method parameter determines how
PostgreSQL will ask the kernel to force
WAL updates out to disk.
All the options should be the same as far as reliability goes,
but it's quite platform-specific which one will be the fastest.
- Note that this parameter is irrelevant if FSYNC
+ Note that this parameter is irrelevant if fsync
has been turned off.
- Setting the WAL_DEBUG parameter to any nonzero
+ Setting the wal_debug parameter to any nonzero
value will result in each LogInsert and
- logged to standard error. At present, it makes no difference what
+ logged to the server log. At present, it makes no difference what
the nonzero value is. This option may be replaced by a more
general mechanism in the future.
+
+
+
Internals
+
+
WAL is automatically enabled; no action is
+ required from the administrator except ensuring that the additional
+ disk-space requirements of the
WAL logs are met,
+ and that any necessary tuning is done (see
+ linkend="wal-configuration">).
+
+
+
WAL logs are stored in the directory
+ pg_xlog under the data directory, as a set of
+ segment files, each 16 MB in size. Each segment is divided into 8
+ kB pages. The log record headers are described in
+ access/xlog.h; the record content is dependent
+ on the type of event that is being logged. Segment files are given
+ ever-increasing numbers as names, starting at
+ 0000000000000000. The numbers do not wrap, at
+ present, but it should take a very long time to exhaust the
+ available stock of numbers.
+
+
+ The
WAL buffers and control structure are in
+ shared memory and are handled by the server child processes; they
+ are protected by lightweight locks. The demand on shared memory is
+ dependent on the number of buffers. The default size of the
+
WAL buffers is 8 buffers of 8 kB each, or 64 kB
+ total.
+
+
+ It is of advantage if the log is located on another disk than the
+ main database files. This may be achieved by moving the directory
+ pg_xlog to another location (while the server
+ is shut down, of course) and creating a symbolic link from the
+ original location in the main data directory to the new location.
+
+
+ The aim of
WAL, to ensure that the log is
+ written before database records are altered, may be subverted by
+ disk drives that falsely report a successful write to the kernel,
+ when, in fact, they have only cached the data and not yet stored it
+ on the disk. A power failure in such a situation may still lead to
+ irrecoverable data corruption. Administrators should try to ensure
+ that disks holding
PostgreSQL's
+
WAL log files do not make such false reports.
+
+
+ After a checkpoint has been made and the log flushed, the
+ checkpoint's position is saved in the file
+ pg_control. Therefore, when recovery is to be
+ done, the server first reads pg_control and
+ then the checkpoint record; then it performs the REDO operation by
+ scanning forward from the log position indicated in the checkpoint
+ record. Because the entire content of data pages is saved in the
+ log on the first page modification after a checkpoint, all pages
+ changed since the checkpoint will be restored to a consistent
+ state.
+
+
+ Using pg_control to get the checkpoint
+ position speeds up the recovery process, but to handle possible
+ corruption of pg_control, we should actually
+ implement the reading of existing log segments in reverse order --
+ newest to oldest -- in order to find the last checkpoint. This has
+ not been implemented, yet.
+
+