-
Regression Test
+ id="regress-title">Regression Test
operations as well as the extended capabilities of PostgreSQL.
+ There are two different ways in which the regression tests can be run:
+ the "sequential" method and the "parallel" method. The sequential method
+ runs each test script in turn, whereas the parallel method starts up
+ multiple server processes to run groups of tests in parallel. Parallel
+ testing gives confidence that interprocess communication and locking
+ are working correctly. Another key difference is that the sequential
+ test procedure uses an already-installed postmaster, whereas the
+ parallel test procedure tests a system that has been built but not yet
+ installed. (The parallel test script actually does an installation into
+ a temporary directory and fires up a private postmaster therein.)
+
+
+ Some properly installed and fully functional PostgreSQL installations
+ can "fail" some of these regression tests due to artifacts of floating point
+ representation and time zone support. The tests are currently evaluated
+ using a simple
diff comparison against the
+ outputs generated on a reference system, so the results are sensitive to
+ small system differences.
+ When a test is reported as "failed", always examine the differences
+ between expected and actual results; you may well find that the differences
+ are not significant.
+
+
The regression tests were originally developed by Jolly Chen and Andrew Yu,
and were extensively revised/repackaged by Marc Fournier and Thomas Lockhart.
the regression tests are current for every official release.
- Some properly installed and fully functional PostgreSQL installations
- can fail some of these regression tests due to artifacts of floating point
- representation and time zone support. The current tests are evaluated
- using a simple "diff" algorithm, and are sensitive to small system
- differences. For apparently failed tests, examining the differences
- may reveal that the differences are not significant.
-
+
+
Regression Environment
The regression testing notes below assume the following (except where noted):
-
-
Regression Environment
-
- To prepare for regression testing, do make all in the regression test
- directory. This compiles a
C program with PostgreSQL extension functions
- into a shared library. Localized SQL scripts and output-comparison
- files are also created for the tests that need them. The localization
- replaces macros in the source files with absolute pathnames and user names.
-
-
Normally, the regression tests should be run as the postgres user since
the 'src/test/regress' directory and sub-directories are owned by the
postgres user. If you run the regression test as another user the
- 'src/test/regress' directory tree must be writeable to that user.
+ 'src/test/regress' directory tree must be writeable by that user.
results/ .. .out files that contain what the results *actually* look
like. Also used as temporary storage for table copy testing.
+
+ tmp_check/ temporary installation created by parallel testing script.
like ps and tar vary wildly on what options you should use on each
platform. Use common sense before typing in these commands.
-
- For a fresh install or upgrading from previous releases of
-
-
<ProductName>Postgres</ProductName> Regression <span class="marked">Configuration</span>
-
+
<ProductName>Postgres</ProductName> Regression <span class="marked">Test</span>
+
- The file /usr/src/pgsql/src/test/regress/README has detailed
- instructions for running and interpreting the regression tests.
- A short version follows here:
+ Prepare the files needed for the regression test with:
+ cd /usr/src/pgsql/src/test/regress
+ gmake clean
+ gmake all
+
+ You can skip "gmake clean" if this is the first time you
+ are running the tests.
+
+ program with PostgreSQL extension functions into a shared library.
+ Localized SQL scripts and output-comparison files are also created
+ for the tests that need them. The localization replaces macros in
+ the source files with absolute pathnames and user names.
-
+
+
+
- If the postmaster is not already running, start the postmaster on an
- available window by typing
+ If you intend to use the "sequential" test procedure, which tests
+ an already-installed postmaster, be sure that the postmaster
+ is running. If it isn't already running,
+ start the postmaster in an available window by typing
postmaster
-
or start the postmaster daemon running in the background by typing
cd
nohup postmaster > regress.log 2>&1 &
-
-
- Run postmaster from your
Postgres super user account (typically
- account postgres).
+ The latter is probably preferable, since the regression test log
+ will be quite lengthy (60K or so, in
+
Postgres 7.0) and you might want to
+ review it for clues if things go wrong.
-
- If you have previously invoked the regression test, clean up the
- working directory with:
-
- cd /usr/src/pgsql/src/test/regress
- gmake clean
-
-
- You do not need to type "gmake clean" if this is the first time you
- are running the tests.
-
-
-
-
- Build the regression test. Type
+ Run the regression tests. For a sequential test, type
cd /usr/src/pgsql/src/test/regress
- gmake all
+ gmake runtest
-
-
-
-
- Run the regression tests. Type
+ For a parallel test, type
cd /usr/src/pgsql/src/test/regress
- gmake runtest
+ gmake runcheck
+ The sequential test just runs the test scripts using your
+ already-running postmaster.
+ The parallel test will perform a complete installation of
+
Postgres into a temporary directory,
+ start a private postmaster therein, and then run the test scripts.
+ Finally it will kill the private postmaster (but the temporary
+ directory isn't removed automatically).
-
You should get on the screen (and also written to file ./regress.out)
a series of statements stating which tests passed and which tests
failed. Please note that it can be normal for some of the tests to
- "fail". For the failed tests, use diff to compare the files in
- directories ./results and ./expected. If float8 failed, type
- something like:
- cd /usr/src/pgsql/src/test/regress
- diff -w expected/float8.out results
-
+ "fail" due to platform-specific variations. See the next section
+ for details on determining whether a "failure" is significant.
+
+ Some of the tests, notably "numeric", can take a while, especially
+ on slower platforms. Have patience.
After running the tests and examining the results, type
- dropdb regression
cd /usr/src/pgsql/src/test/regress
gmake clean
to recover the temporary disk space used by the tests.
+ If you ran a sequential test, also type
+ dropdb regression
+
Regression Analysis
- The results are in files in the ./results directory. These results
- can be compared with results in the ./expected directory using 'diff'.
- (The test script does this for you, and leaves the differences
- in ./regression.diffs.)
+ The actual outputs of the regression tests are in files in the
+ ./results directory. The test script
+ uses
diff to compare each output file
+ against the reference outputs stored in the
+ ./expected directory. Any differences are
+ saved for your inspection in
+ ./regression.diffs. (Or you can run
+
diff yourself, if you prefer.)
When comparing the results from different platforms, differences occur
in the 2nd or 3rd place to the right of the decimal point. The SQL
- statements where these problems occur are the folowing:
+ statements where these problems occur are the following:
QUERY: SELECT * from street;
Random differences
- There is at least one test case in random.out which is intended to produce
- random results. This causes random to fail the regression testing
- once in a while.
+ There is at least one case in the "random" test script that is
+ intended to produce
+ random results. This causes random to fail the regression test
+ once in a while (perhaps once in every five to ten trials).
Typing
diff results/random.out expected/random.out
-
- should produce only
- one or a few lines of differences for this reason, but other floating
- point differences on dissimilar architectures might cause many more
- differences. See the release notes below.
+ should produce only one or a few lines of differences. You need
+ not worry unless the random test always fails in repeated attempts.
+ (On the other hand, if the random test is never
+ reported to fail even in many trials of the regress tests, you
+ probably should worry.)
system using the postgres5-1.02a5.tar.gz source tree. It was compared
with a file created on an I386 Solaris 2.4 system and the differences
were only in the floating point polygons in the 3rd digit to the right
- of the decimal point. (see below)
+ of the decimal point.
The original sample.regress.out file was from the postgres-1.01 release
- constructed by Jolly Chen and is included here for reference. It may
+ constructed by Jolly Chen. It may
have been created on a DEC ALPHA machine as the Makefile.global
in the postgres-1.01 release has PORTNAME=alpha.
projects / postgresql.git / commitdiff