Parallel queries (Robert Haas, Amit Kapila, David Rowley, many others)

With 9.6, PostgreSQL introduces initial support for parallel execution of large queries. Only strictly read-only queries where the driving table is accessed via a sequential scan can be parallelized. Hash joins and nested loops can be performed in parallel, as can aggregation (for supported aggregates). Much remains to be done, but this is already a useful set of features.

Use of parallel query execution can be controlled through the new configuration parametersmax_parallel_degree, force_parallel_mode, parallel_setup_cost, and parallel_tuple_cost.

Provide infrastructure for marking functions as parallel-safe or not (Robert Haas, Amit Kapila)

Allow GIN index builds to make effective use of maintenance_work_mem settings larger than 1GB (Robert Abraham, Teodor Sigaev)

Add pages deleted from a GIN index's pending list to the free space map immediately, to reduce bloat if the table isn't vacuumed often (Jeff Janes, Teodor Sigaev)

Add gin_clean_pending_list() function to allow manual invocation of pending-list cleanup for a GIN index, separately from vacuuming or analyzing the parent table (Jeff Janes)

Improve handling of dead index tuples in GiST indexes (Anastasia Lubennikova)

Dead index tuples are now marked as such when an index scan notices that the corresponding heap tuple is dead; and when inserting tuples, marked-dead tuples will be removed if needed to make space on the page.

Add an SP-GiST operator class for type box (Alexander Lebedev)

Avoid re-vacuuming pages containing only frozen tuples (Masahiko Sawada, Robert Haas)

Formerly, an anti-wraparound vacuum had to visit every page of a table whether or not there was anything to do there. Now, pages containing only already-frozen tuples are identified in the table's visibility map, and can be skipped by vacuum even when it's doing transaction wraparound prevention. This should greatly reduce the cost of maintaining large tables containing mostly-unchanging data.

Avoid useless heap-truncation attempts during VACUUM (Jeff Janes, Tom Lane)

This change avoids taking an exclusive table lock in some cases where no truncation is really possible. The main benefit comes from avoiding unnecessary query cancellations on standby servers.

Reduce interlocking on standby servers during replay of btree index vacuuming operations (Simon Riggs)

This change avoids substantial replication delays that sometimes occurred while replaying such operations.

Drop entries from GROUP BY if they are functionally dependent on other entries (David Rowley)

If a GROUP BY clause includes all columns of a non-deferred primary key, as well as other columns of the same relation, those other columns are redundant and can be dropped from the grouping. This saves computation in many common cases.

When appropriate, postpone evaluation of SELECT output expressions till after ORDER BY sorting (Konstantin Knizhnik)

This change ensures that volatile or expensive functions in the output list are executed in the order suggested by ORDER BY, and that they are not evaluated more times than required when there's a LIMIT. Previously, these properties held if the ordering was performed by an indexscan or pre-mergejoin sort, but not if it was performed by a top-level sort step.

Where feasible, trigger kernel writeback after a configurable number of writes, to prevent accumulation of dirty data in kernel disk buffers (Fabien Coelho, Andres Freund)

PostgreSQL writes data to the kernel's disk cache, from where it should be flushed to physical storage in due time. Many operating systems are not very smart about managing this, and will allow large amounts of dirty data to accumulate then decide to flush it all at once, leading to long delays for new I/O requests. This change attempts to alleviate this problem by explicitly requesting data flushes after a configurable interval.

On Linux, sync_file_range() is used for this purpose, and the feature is on by default because that function has few downsides. The feature is also available on other platforms that have msync() or posix_fadvise(), but those interfaces have some undesirable side-effects so the feature is not enabled by default on other platforms.

The new configuration parameters backend_flush_after, bgwriter_flush_after, checkpoint_flush_after, andwal_writer_flush_after control this behavior.

Perform checkpoint writes in sorted order (Fabien Coelho, Andres Freund)

Previously, checkpoints wrote out dirty pages in whatever order they happen to appear in within shared buffers, which usually is nearly random. That performs poorly, especially on rotating media. This change causes checkpoint-driven writes to be done in order by file and block number, and to be balanced across tablespaces.

Allow old MVCC snapshots to be invalidated after a configurable timeout (Kevin Grittner)

Normally, deleted tuples cannot be physically removed by vacuuming until the last transaction that could"see" them is gone. A transaction that stays open for a long time can thus cause considerable table bloat because space cannot be recycled. This feature allows setting a time-based limit, via the new configuration parameter old_snapshot_threshold, on how long an MVCC snapshot is guaranteed valid. After that, dead tuples are candidates for removal. A transaction using an outdated snapshot will get an error, but only if it attempts to read a page that's been modified recently enough that it might have contained such data.

Allow using an index-only scan with a partial index when the index's predicate involves column(s) not stored in the index (Tomas Vondra, Kyotaro Horiguchi)

An index-only scan is now allowed if the query mentions such columns only in WHERE clauses that match the index predicate.

Use foreign key relationships to infer selectivity for multi-column join predicates (Tomas Vondra, David Rowley)

If a table t has a multi-column foreign key restriction, say (a,b) REFERENCES r (x,y), then a WHERE condition such as t.a = r.x AND t.b = r.y cannot select more than one r row per t row. The planner formerly considered the AND'ed conditions to be independent and would often drastically misestimate the selectivity as a result. Now it compares the WHERE conditions to applicable foreign key constraints and arrives at a better estimate.

Improve aggregate-function performance by sharing calculations across multiple aggregates if they have the same arguments and transition functions (David Rowley)

Speed up visibility tests for recently-created tuples by checking our transaction snapshot, not pg_clog, to decide if the source transaction should be considered committed (Jeff Janes, Tom Lane)

Allow tuple hint bits to be set sooner than before (Andres Freund)

Improve performance of short-lived prepared transactions (Stas Kelvich, Simon Riggs, Pavan Deolasee)

Two-phase commit information is now written only to WAL during PREPARE TRANSACTION, and read back from there during COMMIT PREPARED. A separate state file is created only if the pending transaction does not get committed or aborted by the time of the next checkpoint.

Improve performance of memory context destruction (Jan Wieck)

Improve performance of ResourceOwners with many tracked objects (Aleksander Alekseev)

Improve speed of the output functions for timestamps, times, and dates (David Rowley, Andres Freund)

Avoid some unnecessary cancellations of hot-standby queries during replay of actions that take AccessExclusiveLocks (Jeff Janes)

Improve ANALYZE's estimates for columns with many nulls (Tomas Vondra, Alex Shulgin)

Previously the code tended to underestimate the number of non-null distinct values in a column with many nulls, and it also might make poor decisions about what is a most-common value.

Improve planner's estimate of the number of distinct values in a query result (Tomas Vondra)

Extend relations multiple blocks at a time, when there is contention for the relation's extension lock (Dilip Kumar)

This improves scalability by decreasing contention.

Improve sorting performance by using quicksort, not replacement selection, within steps of an external sort (Peter Geoghegan)

The new approach makes better use of CPU cache for typical cache sizes and data volumes. Where necessary, the behavior can be adjusted via the new configuration parameter replacement_sort_tuples, which see for further details.

Speed up text sorts where the same strings occur multiple times (Peter Geoghegan)

Speed up sorting of uuid, bytea, and char(n) fields by using "abbreviated" keys (Peter Geoghegan)

Support for abbreviated keys has also been added to the non-default operator classes text_pattern_ops,varchar_pattern_ops, and bpchar_pattern_ops. Processing of ordered-set aggregates can also now exploit abbreviated keys.

Speed up CREATE INDEX CONCURRENTLY by treating TIDs as 64-bit integers during the sort phase (Peter Geoghegan)

Increase the number of clog buffers for better scalability (Amit Kapila, Andres Freund)

Reduce contention for the ProcArrayLock (Amit Kapila, Robert Haas)

Improve performance by moving buffer content locks into the buffer descriptors (Andres Freund, Simon Riggs)

Replace shared-buffer header spinlocks with atomic operations to improve scalability (Alexander Korotkov, Andres Freund)

Use atomic operations, rather than a spinlock, to protect an LWLock's wait queue (Andres Freund)

Partition the freelist for shared hash tables, to reduce contention on many-CPU servers (Aleksander Alekseev)

Speed up expression evaluation in PL/pgSQL by keeping ParamListInfo entries for simple variables valid at all times (Tom Lane)

Avoid reducing the SO_SNDBUF setting below its default on recent Windows versions (Chen Huajun)

Improve the pg_stat_activity view's information about what a process is waiting for (Amit Kapila, Ildus Kurbangaliev)

Historically a process has only been shown as waiting if it was waiting for a heavyweight lock. Now waits for lightweight locks and buffer pins are also shown in pg_stat_activity. Also, the type of lock being waited for is now visible. These changes replace the waiting column with wait_event_type and wait_event.

Add pg_stat_progress_vacuum system view to provide progress reporting for VACUUM operations (Amit Langote, Robert Haas, Vinayak Pokale, Rahila Syed)

Add pg_config system view (Joe Conway)

This view exposes the same information available from the pg_config utility, namely assorted compile-time configuration information for PostgreSQL.

Add a confirmed_flush_lsn column to the pg_replication_slots system view (Marko Tiikkaja)

Add pg_stat_wal_receiver system view to provide information about the state of a hot-standby server's WAL receiver process (Michael Paquier)

Add pg_blocking_pids() function to reliably identify which sessions block which others (Tom Lane)

This function returns an array of the process IDs of any sessions that are blocking the session with the given process ID. Historically users have obtained such information using a self-join on the pg_locks view; but it's unreasonably tedious to do it that way with any modicum of correctness, and the addition of parallel queries has made the approach entirely impractical, since locks might be held or awaited by child worker processes rather than the session's main process.

Add pg_control_system(), pg_control_checkpoint(), pg_control_recovery(), and pg_control_init() functions to expose fields of pg_control to SQL (Joe Conway, Michael Paquier)

Add function pg_current_xlog_flush_location() to expose the current transaction log flush location (Tomas Vondra)

Add function pg_notification_queue_usage() to report how full the NOTIFY queue is (Brendan Jurd)

Limit the verbosity of memory context statistics dumps (Tom Lane)

The memory usage dump printed to the postmaster log during an out-of-memory failure now summarizes statistics when there are a large number of memory contexts, rather than possibly printing a very large report. There's also a "grand total" summary line now.

Change display format for NextXID in pg_controldata and related places (Joe Conway, Bruce Momjian)

Display epoch-and-transaction-ID values in the format number:number. The previous format number/numberwas confusingly similar to that used for LSNs.

Create a bsd authentication method to allow use of the BSD Authentication service for PostgreSQL client authentication (Marisa Emerson)

BSD Authentication is currently only available on OpenBSD.

When using PAM authentication, provide the client IP address or host name to PAM modules via thePAM_RHOST item (Grzegorz Sampolski)

Provide detail in the postmaster log for more password authentication failures (Tom Lane)

All ordinarily-reachable password authentication failure cases should now provide specific DETAIL fields in the log.

Support RADIUS passwords up to 128 characters long (Marko Tiikkaja)

Add new SSPI authentication parameters compat_realm and upn_username, to control whether NetBIOS or Kerberos realm names and user names are used during SSPI authentication (Christian Ullrich)

Add --with-systemd configure switch to enable calling sd_notify() at server start and stop (Peter Eisentraut)

This allows the use of systemd service units of type notify, which greatly simplifies management ofPostgreSQL under systemd.

Allow effective_io_concurrency to be set as a tablespace parameter, to support cases where different tablespaces have different I/O characteristics (Julien Rouhaud)

Allow sessions to be terminated automatically if they sit too long in an idle-in-transaction state (Vik Fearing)

This behavior is enabled and controlled by the new configuration parameteridle_in_transaction_session_timeout. It can be useful to prevent forgotten transactions from holding onto locks or preventing vacuum cleanup for very long periods.

Add log_line_prefix option %n to print the time as a Unix epoch, with milliseconds (Tomas Vondra, Jeff Davis)

Add syslog_sequence_numbers and syslog_split_messages configuration parameters to provide more control over message format when logging to syslog (Peter Eisentraut)

Merge the archive and hot_standby values of the wal_level configuration parameter into a single valuereplica (Peter Eisentraut)

Making a distinction between these settings no longer appears to be a good idea, and it's in the way of planned future simplification of replication setup. The old names are still accepted but are converted internally.

Allow the server's SSL key file to have group read access if owned by root (Christoph Berg)

Formerly we insisted on the key file being owned by the user running the PostgreSQL server, but that is inconvenient for some systems (such as Debian) that wish to manage certificates centrally. So also allow the case where the key file is owned by root and has group read access. It's up to the root admin to ensure that such a group doesn't include any untrusted users.

Force backends to exit if the postmaster dies (Rajeev Rastogi, Robert Haas)

Under normal circumstances the postmaster should always outlive its child processes. If for some reason it dies, cause backend sessions to exit with an error. Formerly, existing backends would continue to run until their client disconnects; but that is unsafe and inefficient, and furthermore it prevents a new postmaster from being started until the last old backend is gone. Backends will detect postmaster death when waiting for client I/O, so the exit will not be instantaneous, but in most circumstances it should happen no later than the end of the current query.

Check for serializability conflicts before reporting constraint-violation failures (Thomas Munro)

When using serializable transaction isolation, it is desirable that any error due to a concurrent transaction should manifest as a serialization failure, thereby cueing the application that a retry might succeed. Unfortunately, this doesn't reliably happen for duplicate-key failures caused by concurrent insertions. This change ensures that such an error will be reported as a serialization error, if the application explicitly checked for the presence of a conflicting key (and didn't find it) earlier in the transaction.

Ensure that invalidation messages are recorded in WAL even when issued by a transaction that has no XID assigned (Andres Freund)

This fixes some corner cases in which transactions on standby servers failed to notice changes such as new indexes.

Prevent multiple processes from trying to clean a GIN index's pending list concurrently (Teodor Sigaev, Jeff Janes)

This had been intentionally allowed, but it causes race conditions that can result in vacuum missing index entries it needs to delete.