8

My PostgreSQL 9.5 server on Debian 8 (Jessie) keeps exiting directly after being started via service postgresql start:

# service postgresql status
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
   Active: active (exited) since Fr 2016-12-02 11:02:51 CET; 11min ago
  Process: 2360 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 2360 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service

Dez 05 16:29:24 dev systemd[1]: Starting PostgreSQL RDBMS...
Dez 05 16:29:24 dev systemd[1]: Started PostgreSQL RDBMS.

Note the active (exited) state. The server is down, e.g. I cannot connect to it via TCP or domain socket, and I can't find any associated processes.

However, when I start PostgreSQL manually it works:

# sudo -u postgres /usr/lib/postgresql/9.5/bin/postgres -d 3 -D /var/lib/postgresql/9.5/main/ -c config_file=/etc/postgresql/9.5/main/postgresql.conf 

2016-12-05 16:34:32 CET [1593-1] DEBUG:  postgres: PostmasterMain: initial environment dump:
2016-12-05 16:34:32 CET [1593-2] DEBUG:  -----------------------------------------
2016-12-05 16:34:32 CET [1593-3] DEBUG:     LC_PAPER=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-4] DEBUG:     LC_ADDRESS=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-5] DEBUG:     LC_MONETARY=C
2016-12-05 16:34:32 CET [1593-6] DEBUG:     TERM=xterm-256color
2016-12-05 16:34:32 CET [1593-7] DEBUG:     LC_NUMERIC=C
2016-12-05 16:34:32 CET [1593-8] DEBUG:     LC_TELEPHONE=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-9] DEBUG:     LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
2016-12-05 16:34:32 CET [1593-10] DEBUG:    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2016-12-05 16:34:32 CET [1593-11] DEBUG:    LC_IDENTIFICATION=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-12] DEBUG:    LANG=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-13] DEBUG:    LC_MEASUREMENT=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-14] DEBUG:    LC_TIME=C
2016-12-05 16:34:32 CET [1593-15] DEBUG:    LC_NAME=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-16] DEBUG:    SHELL=/bin/bash
2016-12-05 16:34:32 CET [1593-17] DEBUG:    MAIL=/var/mail/postgres
2016-12-05 16:34:32 CET [1593-18] DEBUG:    LOGNAME=postgres
2016-12-05 16:34:32 CET [1593-19] DEBUG:    USER=postgres
2016-12-05 16:34:32 CET [1593-20] DEBUG:    USERNAME=postgres
2016-12-05 16:34:32 CET [1593-21] DEBUG:    HOME=/var/lib/postgresql
2016-12-05 16:34:32 CET [1593-22] DEBUG:    SUDO_COMMAND=/usr/lib/postgresql/9.5/bin/postgres -d 3 -D /var/lib/postgresql/9.5/main/ -c config_file=/etc/postgresql/9.5/main/postgresql.conf
2016-12-05 16:34:32 CET [1593-23] DEBUG:    SUDO_USER=root
2016-12-05 16:34:32 CET [1593-24] DEBUG:    SUDO_UID=0
2016-12-05 16:34:32 CET [1593-25] DEBUG:    SUDO_GID=0
2016-12-05 16:34:32 CET [1593-26] DEBUG:    PGLOCALEDIR=/usr/share/locale
2016-12-05 16:34:32 CET [1593-27] DEBUG:    PGSYSCONFDIR=/etc/postgresql-common
2016-12-05 16:34:32 CET [1593-28] DEBUG:    LC_COLLATE=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-29] DEBUG:    LC_CTYPE=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-30] DEBUG:    LC_MESSAGES=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-31] DEBUG:  -----------------------------------------
2016-12-05 16:34:32 CET [1593-32] DEBUG:  invoking IpcMemoryCreate(size=23887872)
2016-12-05 16:34:32 CET [1593-33] DEBUG:  mmap(25165824) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory
2016-12-05 16:34:32 CET [1593-34] DEBUG:  SlruScanDirectory invoking callback on pg_notify/0000
2016-12-05 16:34:32 CET [1593-35] DEBUG:  removing file "pg_notify/0000"
2016-12-05 16:34:32 CET [1593-36] DEBUG:  dynamic shared memory system will support 288 segments
2016-12-05 16:34:32 CET [1593-37] DEBUG:  created dynamic shared memory control segment 1173813643 (2316 bytes)
2016-12-05 16:34:32 CET [1593-38] DEBUG:  max_safe_fds = 983, usable_fds = 1000, already_open = 7
2016-12-05 16:34:32 CET [1593-39] LOG:  redirecting log output to logging collector process
2016-12-05 16:34:32 CET [1593-40] HINT:  Future log output will appear in directory "/var/log/postgresql".

(server is now up and running and can be connected to)

The logs in /var/log/postgresql don't contain anything suspicious. Here's the output for a working manual run:

2016-12-05 16:48:07 CET [1700-1] LOG:  database system was shut down at 2016-12-05 16:36:04 CET
2016-12-05 16:48:07 CET [1700-2] DEBUG:  checkpoint record is at 0/13823548
2016-12-05 16:48:07 CET [1700-3] DEBUG:  redo record is at 0/13823548; shutdown TRUE
2016-12-05 16:48:07 CET [1700-4] DEBUG:  next transaction ID: 0/14019; next OID: 35693
2016-12-05 16:48:07 CET [1700-5] DEBUG:  next MultiXactId: 1; next MultiXactOffset: 0
2016-12-05 16:48:07 CET [1700-6] DEBUG:  oldest unfrozen transaction ID: 617, in database 1
2016-12-05 16:48:07 CET [1700-7] DEBUG:  oldest MultiXactId: 1, in database 1
2016-12-05 16:48:07 CET [1700-8] DEBUG:  commit timestamp Xid oldest/newest: 0/0
2016-12-05 16:48:07 CET [1700-9] DEBUG:  transaction ID wrap limit is 2147484264, limited by database with OID 1
2016-12-05 16:48:07 CET [1700-10] DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 1
2016-12-05 16:48:07 CET [1700-11] DEBUG:  starting up replication slots
2016-12-05 16:48:07 CET [1700-12] DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 1
2016-12-05 16:48:07 CET [1700-13] LOG:  MultiXact member wraparound protections are now enabled
2016-12-05 16:48:07 CET [1700-14] DEBUG:  MultiXact member stop limit is now 4294914944 based on MultiXact 1
2016-12-05 16:48:07 CET [1700-15] DEBUG:  shmem_exit(0): 1 before_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-16] DEBUG:  shmem_exit(0): 3 on_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-17] DEBUG:  proc_exit(0): 2 callbacks to make
2016-12-05 16:48:07 CET [1700-18] DEBUG:  exit(0)
2016-12-05 16:48:07 CET [1700-19] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-20] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-21] DEBUG:  proc_exit(-1): 0 callbacks to make
2016-12-05 16:48:07 CET [1698-41] LOG:  database system is ready to accept connections
2016-12-05 16:48:07 CET [1705-1] DEBUG:  removing permanent stats file "pg_stat/db_12381.stat"
2016-12-05 16:48:07 CET [1704-1] LOG:  autovacuum launcher started
2016-12-05 16:48:07 CET [1704-2] DEBUG:  InitPostgres
2016-12-05 16:48:07 CET [1705-2] DEBUG:  removing permanent stats file "pg_stat/db_0.stat"
2016-12-05 16:48:07 CET [1704-3] DEBUG:  StartTransaction
2016-12-05 16:48:07 CET [1705-3] DEBUG:  removing permanent stats file "pg_stat/global.stat"
2016-12-05 16:48:07 CET [1704-4] DEBUG:  name: unnamed; blockState:       DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children: 
2016-12-05 16:48:07 CET [1704-5] DEBUG:  CommitTransaction
2016-12-05 16:48:07 CET [1704-6] DEBUG:  name: unnamed; blockState:       STARTED; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children: 
2016-12-05 16:48:07 CET [1705-4] DEBUG:  received inquiry for database 0
2016-12-05 16:48:07 CET [1705-5] DEBUG:  writing stats file "/var/run/postgresql/9.5-main.pg_stat_tmp/global.stat"
2016-12-05 16:48:07 CET [1705-6] DEBUG:  writing stats file "/var/run/postgresql/9.5-main.pg_stat_tmp/db_0.stat"
2016-12-05 16:48:07 CET [1701-1] DEBUG:  checkpointer updated shared memory configuration values

In case of an unsuccessful run via service postgresql start, no logs are written at all to /var/log/postgresql. Similarly, I can't find any PostgreSQL-related messages in /var/log/messages.

I've installed PostgreSQL via the official repository:

# apt-cache policy postgresql-9.5
postgresql-9.5:
  Installed: 9.5.5-1.pgdg80+1
  Candidate: 9.5.5-1.pgdg80+1
  Version table:
 *** 9.5.5-1.pgdg80+1 0
        500 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main amd64 Packages
        100 /var/lib/dpkg/status

All of this is on a VirtualBox VM.

Update 1

The systemd unit seems to belong to the postgresql-common package:

# find /lib/systemd -iname "*postgres*"
/lib/systemd/system/postgresql.service
/lib/systemd/system/postgresql@.service
/lib/systemd/system-generators/postgresql-generator

# dpkg -S postgresql.service
postgresql-common: /lib/systemd/system/postgresql.service

# dpkg -S postgresql@.service
postgresql-common: /lib/systemd/system/postgresql@.service

# dpkg -S postgresql-generator
postgresql-common: /lib/systemd/system-generators/postgresql-generator

Here's their content:

/lib/systemd/system/postgresql.service

# systemd service for managing all PostgreSQL clusters on the system. This
# service is actually a systemd target, but we are using a service since
# targets cannot be reloaded.

[Unit]
Description=PostgreSQL RDBMS

[Service]
Type=oneshot
ExecStart=/bin/true
ExecReload=/bin/true
RemainAfterExit=on

[Install]
WantedBy=multi-user.target

/lib/systemd/system/postgresql@.service

# systemd service template for PostgreSQL clusters. The actual instances will
# be called "postgresql@version-cluster", e.g. "postgresql@9.3-main". The
# variable %i expands to "version-cluster", %I expands to "version/cluster".
# (%I breaks for cluster names containing dashes.)

[Unit]
Description=PostgreSQL Cluster %i
ConditionPathExists=/etc/postgresql/%I/postgresql.conf
PartOf=postgresql.service
ReloadPropagatedFrom=postgresql.service
Before=postgresql.service

[Service]
Type=forking
# @: use "postgresql@%i" as process name
ExecStart=@/usr/bin/pg_ctlcluster postgresql@%i --skip-systemctl-redirect %i start
ExecStop=/usr/bin/pg_ctlcluster --skip-systemctl-redirect -m fast %i stop
ExecReload=/usr/bin/pg_ctlcluster --skip-systemctl-redirect %i reload
PIDFile=/var/run/postgresql/%i.pid
SyslogIdentifier=postgresql@%i
# prevent OOM killer from choosing the postmaster (individual backends will
# reset the score to 0)
OOMScoreAdjust=-900
# restarting automatically will prevent "pg_ctlcluster ... stop" from working,
# so we disable it here. Also, the postmaster will restart by itself on most
# problems anyway, so it is questionable if one wants to enable external
# automatic restarts.
#Restart=on-failure
# (This should make pg_ctlcluster stop work, but doesn't:)
#RestartPreventExitStatus=SIGINT SIGTERM

[Install]
WantedBy=multi-user.target

/lib/systemd/system-generators/postgresql-generator

#!/bin/sh

# This systemd generator creates dependency symlinks that make all PostgreSQL
# clusters with "auto" in their start.conf file be started/stopped/reloaded
# when postgresql.service is started/stopped/reloaded.

set -eu

gendir="$1"
wantdir="$1/postgresql.service.wants"
pgservice="/lib/systemd/system/postgresql@.service"

mkdir -p "$wantdir"

for conf in /etc/postgresql/*/*/postgresql.conf; do
    test -e "$conf" || continue
    dir="${conf%/*}"

    # evaluate start.conf
    if [ -e "$dir/start.conf" ]; then
        start=$(sed 's/#.*$//; /^[[:space:]]*$/d; s/^\s*//; s/\s*$//' "$dir/start.conf")
    else
        start=auto
    fi
    [ "$start" = "auto" ] || continue

    verdir="${dir%/*}"
    version="${verdir##*/}"
    cluster="${dir##*/}"
    ln -s "$pgservice" "$wantdir/postgresql@$version-$cluster.service"
done

exit 0

postgresql-common was installed from the official PostgreSQL repository as a dependency of the postgresql-9.5 package:

# apt-cache policy postgresql-common
postgresql-common:
  Installed: 177.pgdg80+1
  Candidate: 177.pgdg80+1
  Version table:
 *** 177.pgdg80+1 0
        500 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main amd64 Packages
        100 /var/lib/dpkg/status
     165+deb8u1 0
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

Update 2

As requested, here's the output of strace -f service postgresql start on Pastebin (due to its size).

  • Do netstat -punta after starting it manually, is postgres listening on any port? – sysfiend Dec 5 '16 at 16:17
  • @Alex When I start PostgreSQL manually it listens and accepts connections via TCP (port 5432) and domain socket (/var/run/postgresql/.s.PGSQL.5432). When I start it via service none of that works. – Florian Brucker Dec 5 '16 at 16:21
  • You could give strace a try: – Oliver Rahner Dec 12 '16 at 8:51
  • strace -f service postgresql start – Oliver Rahner Dec 12 '16 at 8:58
  • @OliverRahner: Thanks for the idea, i've updated my answer with the output of strace. – Florian Brucker Dec 12 '16 at 10:02
6
+50

Theres a couple of things going on here. If you call a systemd service with an @ in it, its a signal to start an instance rather than main service. Putting @9.5-main is telling it the instance parameters to use.

The postgres service file you are seeing is nothing more than a placeholder. There is a directory on the system (its normally somewhere in /run) called postgresql.wants.d which the systemd will look for when postgresql.service is started.

Anything in that directory will be started along with it as postgresql.service is told it wants it.

That directory is filled through a shell script generator file as everything in /run is a tmpfs. Thats the shell script you quoted above.

If I had to hazard a guess whats going on there via the logic of the shell script, you probably have got the file /etc/postgresql/9.5/main/start.conf set to manual.

This means the generator skips over it and the postgresql.wants.d never gets added with that instance name.

According to the script logic simply removing it has the right affect of making it auto.

You can also set it to auto and do a daemon-reload to regenerate the necessary config.

  • Thanks, this has already clarified a lot! However, my /etc/postgresql/9.5./main/start.conf already says auto. If I manually execute /lib/systemd/system-generators/postgresql-generator 9.5-main from within /run/systemd/generator/postgresql.service.wants then the necessary link gets created and everything works after I systemctl restart postgresl (although systemctl status postgresql still says active (exited)). However, after a reboot the wants link is not created automatically, and systemctl daemon-reload doesn't re-create it either. Any ideas? – Florian Brucker Dec 19 '16 at 9:38
  • You could possibly try copying the generator to /etc/systemd/system-generators/ and seeing if it works from that path on reboot. Something might be wrong with the shellscript at boot time (maybe a filesystem isn't mounted?). You'd have to alter the shellscript to get it to be more verbose and store the data somewhere to see what gets executed from within it. – Matthew Ife Dec 19 '16 at 10:09
  • systemctl daemon-reload does run the generator, I must have missed that in my previous comment. However I can't get the generator to run at boot-time, so as a hacky workaround I'm now running systemctl daemon-reload and systemctl restart postgresql after booting via a Vagrant provision script. Thanks a lot for your help! – Florian Brucker Dec 19 '16 at 13:42
1

As of 26 May 2017, the issue still exists on Debian 8 (as far as I can see): PostgreSQL fails to start on boot.

The solution for the time being is to issue:

systemctl daemon-reload
systemctl restart postgresql

manually on boot and everything should be all right. Thanks to Florian Brucker for pointing this out above.

The bug is apparently in systemd and, hopefully, should be rectified soon.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.