Where I work, servers are managed by an external company, so when you want a new server you get the root password for a while to set it up, and then they change it and you are supposed to make do with a non-root user.

So a while ago I set up my server, it runs Red Hat Enterprise Linux server 5.4, Apache 2.2.3, Apache Tomcat 5.5.23, and PostgreSQL 8.1.22. I created a user, a database, populated it with data, no problem. They changed the root password and everything seemed to work okay.

A couple of months ago, the server was shutdown for electrical maintenance in the building. I don't know if they shut it down cleanly or not.
As the server is not much used, I only found out last week that the PostgreSQL server didn't start again at reboot. I had to ask them to start it up and when they did, surprise, everything is gone: the user I created, the database.... The Tomcat application itself is still there and working but without a database connection, it doesn't do much.

I managed to get a root password for the time being, so I can look around.
Is there a way to find out what deleted everything? How can I prevent it from happening again?

Everything in /var/lib/pgsql/ is dated the day they restarted the server, in fact everything looks like that day was the first time it was ever started.
Is it possible there are multiple PostgreSQL servers or locations on the machine and the old database is still there somewhere? Or did something wipe out PostgreSQL?

I have a backup so the data itself is not lost, but I would very much like to learn what happened.

  • Without knowing what this other company did, it's hard to say what happened or what to do about it. One guess is that the database wasn't shut down cleanly, and that postgres was refusing to start until it was repaired, and they forced it to start when you told them to start it. Since you have root now, it would be best to see if you can find the logs from when the system powered back on after reboot, and when they turned postgresql on, to see if there are any meaningful messages – DerfK Oct 12 '11 at 13:06
  • I think your support company restore server from its own backup, witch was made when they changed root password. Check last system logs before system was damaged. – Mikhail Khirgiy Nov 21 '16 at 16:34

Anything special about the disk/lvm configuration on this server?

Was the /var/lib/pgsql/data directory a separate filesystem, maybe on a SAN or just a separate set of disks? If someone had manually mounted data/ without also putting it in fstab you would have a big surprise after your first reboot.


I think on older version of RH, the init scripts would always look in /var/lib/pgsql/data and init a new cluster if one wasn't found, so a situation like @iPaulo mentions, or any non-default might end up like you have described.

I'd be inclined to do a server wide search for postgres specific files like postgresql.conf or pg_hba.conf and see if I could find another copy of the database, and also check for any logs in /var/log or similar places that might show evidence of the old cluster; there might be a PGDATA lying around some place.

side note, if all is lost, you could use this opportunity to upgrade; 8.1 was EOL'd years ago. Get 9.1 instead :-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.