Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

After resolving some problems with the network drive on which our Postgres database was hosted, we encountered this error:

OperationalError: FATAL: cache lookup failed for index 2662

That particular error was from a Python command line, but other attempts at connecting to the DB yielded the same thing, either ordinary querying or attempting a dump of the database.

How do you go about fixing this error and recovering the data?

share|improve this question
    
When was your last backup? I've found this: archives.postgresql.org/pgsql-bugs/2010-02/msg00187.php –  user3914 Jun 30 '10 at 6:44
    
That's not the same thing though - that's a user index. The question about the backup is very valid, though.. –  Magnus Hagander Jun 30 '10 at 9:14

1 Answer 1

up vote 0 down vote accepted

Sounds like you have data corruption. When you say "network drive", that's a red flag right there. It's very hard to configure NFS to get data reliability with PostgreSQL. It's AFAIK impossible to configure SMB to get it. So most likely, you got data corruption in the crash, because the network mount was not safe.

2662 is the oid index on pg_class - which is basically the index on the table that lists which tables and indexes exist in your database. When this is corrupt, well, you're in serious trouble.

Your best bet at this point is trying to start a standalone backend (with the main one shut down of course) with "-P" which turns off usage of system indexes. Then attempt to REINDEX pg_class, and possibly the rest of the system indexes. If this worked, do a pg_dump backup right away, and re-initialize the database. There is likely more corruption in the system.

If this does not work, try posting to the pgsql-general mailinglist, you're likely to get more and quicker responses there. But most likely, it's back to your backups - or hire a data recovery expert if your data is really important and you don't have working backups.

share|improve this answer
    
Thought this had actually worked, however I could only access the Postgres backend without specifying the data directory. Even with the -P flag, specifying the correct data directory yielded the same cache lookup error. Given that we've got a backup (not as recent as we'd like) can we restore from that point on using the transaction logs? –  bennylope Jun 30 '10 at 11:46
    
Posted a follow up question on how to reindex, given that the -P flag doesn't seem to be working: serverfault.com/questions/156284/… –  bennylope Jun 30 '10 at 12:59
    
If you have your system set up with PITR backups (base backup using pg_start_backup/pg_stop backup and log archiving) then yes, you can restore from that point up until before the corruption occurred. If you are using pg_dump backups then, no, they are logical backups and not physical ones, and thus have no relation to the transaction log. –  Magnus Hagander Jun 30 '10 at 14:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.