Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Where I work, servers are managed by an external company, so when you want a new server you get the root password for a while to set it up, and then they change it and you are supposed to make do with a non-root user.

So a while ago I set up my server, it runs Red Hat Enterprise Linux server 5.4, Apache 2.2.3, Apache Tomcat 5.5.23, and PostgreSQL 8.1.22. I created a user, a database, populated it with data, no problem. They changed the root password and everything seemed to work okay.

A couple of months ago, the server was shutdown for electrical maintenance in the building. I don't know if they shut it down cleanly or not.
As the server is not much used, I only found out last week that the PostgreSQL server didn't start again at reboot. I had to ask them to start it up and when they did, surprise, everything is gone: the user I created, the database.... The Tomcat application itself is still there and working but without a database connection, it doesn't do much.

I managed to get a root password for the time being, so I can look around.
Is there a way to find out what deleted everything? How can I prevent it from happening again?

Everything in /var/lib/pgsql/ is dated the day they restarted the server, in fact everything looks like that day was the first time it was ever started.
Is it possible there are multiple PostgreSQL servers or locations on the machine and the old database is still there somewhere? Or did something wipe out PostgreSQL?

I have a backup so the data itself is not lost, but I would very much like to learn what happened.

share|improve this question
    
Without knowing what this other company did, it's hard to say what happened or what to do about it. One guess is that the database wasn't shut down cleanly, and that postgres was refusing to start until it was repaired, and they forced it to start when you told them to start it. Since you have root now, it would be best to see if you can find the logs from when the system powered back on after reboot, and when they turned postgresql on, to see if there are any meaningful messages –  DerfK Oct 12 '11 at 13:06

2 Answers 2

Anything special about the disk/lvm configuration on this server?

Was the /var/lib/pgsql/data directory a separate filesystem, maybe on a SAN or just a separate set of disks? If someone had manually mounted data/ without also putting it in fstab you would have a big surprise after your first reboot.

share|improve this answer

I think on older version of RH, the init scripts would always look in /var/lib/pgsql/data and init a new cluster if one wasn't found, so a situation like @iPaulo mentions, or any non-default might end up like you have described.

I'd be inclined to do a server wide search for postgres specific files like postgresql.conf or pg_hba.conf and see if I could find another copy of the database, and also check for any logs in /var/log or similar places that might show evidence of the old cluster; there might be a PGDATA lying around some place.

side note, if all is lost, you could use this opportunity to upgrade; 8.1 was EOL'd years ago. Get 9.1 instead :-)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.