Setup: GitLab installed with Omnibus package on CentOS 6.4

After I've made a change in the GitLab configuration file /etc/gitlab/gitlab.rb to configure Bitbucket as OAuth2 provider, I've run the reconfiguration:

sudo gitlab-ctl reconfigure

After I've launched this command I've seen some error, maybe about a monitoring service, but the SSH session drop and I haven't seen what happened. After restoring the session GitLab was not accessible, I was able to see the GitLab logo with a 502 error: "GitLab is taking too much time to respond" or something similar.

So I've restored the configuration file, thinking something was wrong in it...

Running tail I've got the below errors (Prometheus is failing):

2018-01-30_22:19:19.28021 time="2018-01-30T23:19:19+01:00" level=info msg="Starting prometheus (version=, branch=, revision=)" source="main.go:87"
2018-01-30_22:19:19.28027 time="2018-01-30T23:19:19+01:00" level=info msg="Build context (go=go1.9.2, user=, date=)" source="main.go:88"
2018-01-30_22:19:19.28030 time="2018-01-30T23:19:19+01:00" level=info msg="Host details (Linux 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016 x86_64 lnx01.brainsen.com (none))" source="main.go:89"
2018-01-30_22:19:19.29486 time="2018-01-30T23:19:19+01:00" level=info msg="Loading configuration file /var/opt/gitlab/prometheus/prometheus.yml" source="main.go:254"
2018-01-30_22:19:19.29779 time="2018-01-30T23:19:19+01:00" level=error msg="Could not open the fingerprint-to-metric index for archived series. Please try a 3rd party tool to repair LevelDB in directory "/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric". If unsuccessful or undesired, delete the whole directory and restart Prometheus for crash recovery. You will lose all archived time series." source="persistence.go:213"
2018-01-30_22:19:19.29781 time="2018-01-30T23:19:19+01:00" level=error msg="Error opening memory series storage: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000923]" source="main.go:192"

I've run the command

sudo -u gitlab-prometheus python -c "import leveldb; leveldb.RepairDB('/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric')"

but I get the message:

Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named leveldb

Now GitLab is working fine but running tail I still see the above error about Prometheus every 2 seconds and I'd like to fix it without breaking my GitLab installation.

1 Answer 1


The issue is gitlab runs Prometheus in a pyvenv setup which I can't find where it is.

You can always

pip install leveldb

And then run the command.. it should work

  • Hi Mike, I'll try this as soon as I can touch the machine. Now I need it 100% working for a code review to win a tender... and if I broke it... I'll keep you update if it works
    – Giox
    Commented Feb 3, 2018 at 17:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .