Gitlab docker error on database migration after update (restart loop)

Question

I just updated my gitlab docker instance, running using docker-compose on a host machine. I am experiencing the following error:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
    STDOUT: rake aborted!
    PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:db:configure
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
    Ran "bash"  "/tmp/chef-script20200707-23-1sorsih" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     55: bash "migrate gitlab-rails database" do
     56:   code <<-EOH
     57:     set -e
     58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     59:     umask 077
     60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     61:     STATUS=${PIPESTATUS[0]}
     62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
     63:     echo $STATUS > #{db_migrate_status_file}
     64:     exit $STATUS
     65:   EOH
     66:   environment env_variables unless env_variables.empty?
     67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     68:   dependent_services.each do |svc|
     69:     notifies :restart, svc, :immediately
     70:   end
     71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     73: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f\n    exit $STATUS\n"
      domain nil
      user nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=14.14.29
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
Ran "bash"  "/tmp/chef-script20200707-23-1sorsih" returned 1

Running handlers complete
Chef Client failed. 9 resources updated in 18 seconds
Thank you for using GitLab Docker Image!
Current version: gitlab-ce=13.1.3-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab vim /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

I have seen this question but neither answer solved my issue.

If any more information is needed, feel free to comment.

UPDATE:

Following @fuero s answer, I have come across a few problems:

1. I had a tty error when trying to run the docker command (probably unrelated but I found a workaround anyway). I had to detach the container first.

2. /assets/wrapper crashed the machine so I ignored this step since it ran fine using docker-compose up.

3. The postgresql packages did not exist in the repositories so I had to add them manually.

4. I got a permission error when checking because I was at /. Moving to /tmp fixed the issue.

5. I am now still at --check with another error: Old cluster data and binary directories are from different major versions. I cannot find a way to remedy this as of yet.

Answer 1

Same thing here. Looking at the Postgresql Log it contains:

The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.

Seems that the docker version is unable to properly upgrade its database.

I've reverted to an older version (12.7) for now - 13.x seems broken without a manual DB upgrade. Here's how someone did this, including the content as a quote to preserve:

Upgrade PostgreSQL 10 to 11 within a gitlab-ce 13.1.2 container

Using docker-compose to start our container will start gitlab in it. Because of the database failure, the container restart every minute. We don´t want this for our migration work. So we start it manually and access the bash in it.

Please replace (at least) the hostname and the other options with your custom configuration!

docker run --rm \
--hostname gitlab.example.org \
--publish 8143:443 --publish 8180:80 \
--env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.org'; letsencrypt['enabled'] = false" \
--name gitlab_custom \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
-it gitlab/gitlab-ce:latest /bin/bash

Now we are in the docker container.

First of all run the default entry script to check that it really doesn`t work!

/assets/wrapper

If there is a failure in the Recipe: gitlab::database_migrations section and at least the error message Error executing action 'run' on resource 'bash[migrate gitlab-rails database]', continue to migrate the database.

We need to install the default (old and new) PostgreSQL versions.

apt update
apt install postgresql-10 postgresql-11

Create the destination folder for the migrated database and update the directory owner to gitlab-psql. We assume, the 'old' (version 10) database data is located at /var/opt/gitlab/postgresql/data.

mkdir -p /tmp/11/data
chown -R gitlab-psql /tmp/11 /var/opt/gitlab/postgresql/data

Now we need to initialize the new database structure.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/initdb -D /tmp/11/data

Take a look at /var/opt/gitlab/postgresql/data/postgresql.conf for the right port (and set it with -p).

Before running the real migration, we should check the access and schema of the database. In case of errors, this command can be run multiple times.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432 --check

If there was an error during the check, you may need to update some other directory permissions.

If the check was successful, do the database migration with the following command. This can take some time, depending of the size of your gitlab instance.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432

Stop all gitlab services, backup the old database and move the migrated data to the original path.

gitlab-ctl stop
rm -f /var/opt/gitlab/postgresql/.s.PGSQL.5432*
mv /var/opt/gitlab/postgresql/data /var/opt/gitlab/postgresql/data_old_10
mv /tmp/11/data /var/opt/gitlab/postgresql/data

Now run the default entry script to finish the built-in gitlab migration.

/assets/wrapper

If it works (some log output appears after loading the gitlab parts and of course the website works), you can press Ctrl+C, and enter exit to leave and remove the current container.

If there were still database errors, ... oops.

If there is an error like this,

PG::ConnectionBad: could not connect to server: No such file or directory
     |                Is the server running locally and accepting
     |                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

make sure no postgresql process is running, remove the socket file and try again.

After that, leave the container.

Answer 2

Following @fuero 's answer yields a working installation however the gitlab.rb must be updated manually to stop the database migration occuring when no migration is needed.

The Old cluster data and binary directories are from different major versions error from following the steps was due to the database already being half migrated. The fix is to use version 11 for both the old and new binaries.