Over the few weeks our Exchange server has had problems with its backup jobs failing to complete which has caused our normally ~empty log drives to fill up to the point where Exchange has dismounted databases and logged various errors regarding replaying log files. Lamentably, nobody in the backup team did their job properly, so over the weekend we had a failure situation which dismounted ~40 databases due to there being ~100GB of logs, which generally sit at ~3GB. This caused everyone working the weekend to not look at the history of the issue, and rather than reach out to contact anyone else, enable circular logging after everyone on the team was instructed not to, remount all the databases and call it a day.
We've not heard of any data loss from any users yet, but the fact that all databases encountered this, and that there are complaints about logging failures, replay failures, and unexpected dismounts I'm concerned that there may be some.
Aside from firing the backup team, the weekend monitors and the admin that decided to enable circular logging after a significant amount of time between backups without saving the logs anywhere in case of a need to restore from backup and get whatever we could back, what is my best course of action to determine if we lost anything?
Are there particular events which may be buried in the 3,000,000 log entries that span the six hour long section while this was going on? Is performing an integrity check recommended? Defrag, on or offline?
On the Exchange server the following is typically what happened, I've stripped event source and ID because everything seems to be generic and of little assistance in determining if things actually went super south, or just ruined my Monday:
At 'TIME' the Microsoft Exchange Information Store Database 'DATABASE' copy on this server experienced a serious error which caused it to terminate its functional activity. The error returned by the remount attempt was "There is only one copy of this mailbox database (DATABASE). Automatic recovery is not available.". Consult the event log on the server for other storage and "ExchangeStoreDb" events for more specific information about the failures.
Information Store - DATABASE (9564) DATABASE: An attempt to write to the file "F:\Logs\DATABASE\E0Etmp.log" at offset 1048576 (0x0000000000100000) for 0 (0x00000000) bytes failed after 0.000 seconds with system error 112 (0x00000070): "There is not enough space on the disk. ". The write operation will fail with error -1808 (0xfffff8f0). If this error persists then the file may be damaged and may need to be restored from a previous backup.
Information Store - DATABASE (9564) DATABASE: Unable to create a new logfile because the database cannot write to the log drive. The drive may be read-only, out of disk space, misconfigured, or corrupted. Error -529.
Information Store - DATABASE (9564) DATABASE: The logfile sequence in "F:\Logs\DATABASE\" has been halted due to a fatal error. No further updates are possible for the databases that use this logfile sequence. Please correct the problem and restart or restore from backup.
Information Store - DATABASE (9564) DATABASE: Database recovery/restore failed with unexpected error -510.
The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database. Database: DATABASE Error: MapiExceptionMdbOffline: Unable to open message store. (hr=0x80004005, ec=1142) Diagnostic context:
At 'TIME', the copy of database 'DATABASE' on this server encountered an error during the mount operation. For more information, consult the Event log on the server for "ExchangeStoreDb" or "MSExchangeRepl" events. The mount operation will be tried again automatically.
This is a standalone server, so the only one copy errors seem to be expected. There are also numerous client access errors logged during the time that this was happening, which I've omitted.