Team Foundation Server (TFS) 2017 Failing to create backup

Question

So we've had backups running for a few years now, and in the last few days it started backing up extra times. I went to reconfigure the backups to verify the settings and got an error saying it can't backup the dummy database.

[Info   @12:03:50.669] +-+-+-+-+-| Running Confirming Permissions: Confirming Permissions |+-+-+-+-+-
[Info   @12:03:50.669] 
[Info   @12:03:50.669] +-+-+-+-+-| Confirming Permissions |+-+-+-+-+-
[Info   @12:03:50.669] Starting Node: CONTAINER
[Info   @12:03:50.669] NodePath : Container/Progress
[Info   @12:03:50.669] Node returned: Ignore
[Info   @12:03:50.669] Completed Confirming Permissions: Ignore
[Info   @12:03:50.669] -----------------------------------------------------
[Info   @12:03:50.669] 
[Info   @12:03:50.669] +-+-+-+-+-| Running VerifyCanBackupDatabase: Verifying backing up an empty database succeeds |+-+-+-+-+-
[Info   @12:03:50.669] 
[Info   @12:03:50.669] +-+-+-+-+-| Verifying backing up an empty database succeeds |+-+-+-+-+-
[Info   @12:03:50.669] Starting Node: DUMMYDATABASE
[Info   @12:03:50.669] NodePath : Container/Progress/DUMMYDATABASE
[Info   @12:03:50.669] Creating dummy database to test backup.
[Info   @12:03:51.200] Queue verify backup job
[Info   @12:03:51.263] Queued job [GUID]
[Info   @12:03:51.263] Starting backup job to verify necessary permissions
[Info   @12:03:56.295] Waiting for backup job to finish
[Error  @12:04:01.311] Backup job Failed: TF400797: Job extension had an unhandled error: System.Data.SqlClient.SqlException (0x80131904): The server principal "[DOMAIN\SERVERNAME]$" is not able to access the database "Tfs_Temp[GUID]" under the current security context.
BACKUP DATABASE is terminating abnormally.
   at Microsoft.VisualStudio.Services.Configuration.SqlDatabaseBackupRestoreComponent.Execute(String sqlStatement)
   at Microsoft.VisualStudio.Services.Configuration.SqlDatabaseBackup.Backup(ISqlConnectionInfo connectionInfo)
   at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.TakeFullBackupOfSingleDatabase(IVssRequestContext requestContext, String dataSource, String databaseName, String fullFileName, ITFLogger logger)
   at Microsoft.TeamFoundation.Admin.Jobs.VerifyBackupPermissionsJobExtension.Run(IVssRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime jobQueueTime, String& resultMessage)
   at Microsoft.TeamFoundation.Framework.Server.JobRunner.ExecuteJob(IVssRequestContext requestContext)
ClientConnectionId:00000000-0000-0000-0000-000000000000
Error Number:916,State:1,Class:14
[Info   @12:04:04.405] Node returned: Error
[Error  @12:04:04.405] TF401002: The SQL Server Database Engine failed to save the database backup to path \\[MACHINE]\TFSBackups.  Please grant SQL service account read/write access to that folder.
[Info   @12:04:04.405] Completed VerifyCanBackupDatabase: Error
[Info   @12:04:04.405] -----------------------------------------------------

The permissions on the folder are fine with both our service and the machine having full write permissions

Is there a way to bypass this temp database? The normal backup was working, but now that it has failed to backup this test one, it removed the nightly scheduled backups

Answer 1

A TFS server backup can only be restored by restoring all databases that are part of the TFS instance. performing a backup that is skipping one of the databases puts you in an unrecoverable state.

The first error message is likely the cause of the issue, the backup is being executed by the LocalSystem account of the TFS Data Tier server (e.g. [DOMAIN\SERVERNAME]$ and because that user doesn't have the permission to create backups of the temp tfs_temp database, it is unable to complete the full backup set.

The solution is to either drop the temp project collection from the TFS admin console or to fix the security settings of the user performing the backup, so that it can perform a complete server backup.