It appears some fat-fingering of Subversion configuration may have lead to a Subversion mirror being corrupted. Details: We have a set of Subversion repositories hosted on a master server and then mirrored to two off-site servers using snvsync. All transactions are done using HTTPS, and so Apache is configured on all three servers to allow HTTPS access to the masters and mirrors. Last Friday I set up a new repository on the master server, but in the Apache configuration for the new repo, I set the SVNPath to the same as an existing repository. A few hours later I fixed that error. I'm not entirely sure the errors I'm seeing now are due to that typo, or are just coincidental, but whatever the cause, the repository that actually owns that SVNPath reported errors during svnsync attempts, and now one of the two mirrors of that repository has at least one missing file. Here is the message I get when trying to perform an svnsync on that mirror now:
svnsync: '/Repo-proxy-sync/path/to/missing/file/MissingFile.pm' path not found
A check-out of the mirror's latest revision confirms that file is not present. The missing file was first committed in revision #16166 of the master repository. Looking back at the svnsync logs, it appears there that revision was committed cleanly to the mirror (prior to the creation of the new repository):
Fri Aug 21 06:03:20 CDT 2015
Committed revision 16166.
Copied properties for revision 16166.
A little after 8 AM was when errors began happening. I also suspect my restarts of Apache to apply the new configuration could be a factor. Among the errors I saw through-out the day was a message specifically about revision #16166.
Fri Aug 21 08:01:58 CDT 2015
svnsync: Server sent unexpected return value (500 Internal Server Error) in response to OPTIONS request for 'http://master-ip/Repo'
Fri Aug 21 09:00:02 CDT 2015
svnsync: Server sent unexpected return value (500 Internal Server Error) in response to PROPFIND request for '/Repo/!svn/bln/16166'
Fri Aug 21 10:00:01 CDT 2015
svnsync: Server sent unexpected return value (500 Internal Server Error) in response to PROPFIND request for '/Repo'
Fri Aug 21 14:00:01 CDT 2015
svnsync: Server sent unexpected return value (500 Internal Server Error) in response to OPTIONS request for 'http://master-ip/Repo'
Despite the errors, the master repository appears to be fine; I ran an "svnadmin verify" on the latest revisions, including several revisions committed before creating the other repository and after fixing its configuraiton. All were clean. The second off-site mirror did not run into any problems syncing the affected repository, and currently has the latest revisions with no missing files.
Right now the bad mirror is sitting at revision #16168. Is it possible to force the mirror back several revisions, perhaps to #16165, and then allow the syncs to progress again? Or am I going to have to rebuild the mirror from scratch?
