DFSR Failure After VM Restore (DFSR Error 2104)
Reading Time: 3 minutesI have an environment that heavily leverages DFS, and recently one of the replication member servers had to be restored from a VEEAM backup. Typically VEEAM is great and doesn’t cause any issues, in this case though DFS completely broke. I got a TON of SCOM alerts, and the event log was littered with them as well.
The DFS Replication service failed to recover from an internal database error on volume D:. Replication has been stopped for all replicated folders on this volume. Additional Information: Error: 9214 (Internal database error (-1605)) Volume: D: xxxxxx Database: D:\System Volume Information\DFSR
Event 2212, DFSR
The DFS Replication service has detected an unexpected shutdown on volume D:. This can occur if the service terminated abnormally (due to a power loss, for example) or an error occurred on the volume. The service has automatically initiated a recovery process. The service will rebuild the database if it determines it cannot reliably recover. No user action is required.
Additional Information:
Volume: D:
GUID: xxxxxxxxxxxxxxx
Error 2104, DFSR
The DFS Replication service failed to recover from an internal database error on volume D:. Replication has been stopped for all replicated folders on this volume.
Additional Information:
Error: 9214 (Internal database error (-1605))
Volume: xxxxxxxxxxxxxxxxxxxxxxxx
Database: D:\System Volume Information\DFSR
The important error here is 2104, noting the database issue. There are multiple topics out there that talk about this, but they all end up linking back to this support article.
In the end, essentially the database that is used by DFS replication becomes corrupted. It is a system-generated database so all you need to do is disable the replication service, delete the database, and start the replication service back up. Easy? No. There are a myriad of issues with doing this, mostly because the database is hosted in “System Volume Information” on the volume that hosts the DFS Root folder, or wherever you’ve placed the replication targets. Luckily for you, I hit my head against a wall for hours on end and figured out the solution.
Step 1: Stop DFSR service (stop-service DFSR)
Step 2: Grant yourself visibility to the “System Volume Information” folder. This entails flipping the radio button in explorer to “view hidden files”, as well as unchecking the box for “hide all system protected folders”.
Step 3: Grant yourself proper permissions to the “System Volume Information” folder. Go to the root of the volume that holds the replication targets eg. D:\. You will now see a grayed-out folder with a lock on it called “System Volume Information”. Go through the normal rigamaroo to grant “Administrators” full control over the folder. You should then be able to open it up, before it would have said “Access Denied”.
Step 4: Delete or rename the “DFSR” folder inside “System Volume Information”. Unfortunately, that’s not easy. Based on what I saw, it was because the file names in the database folder exceeded the limitations of explorer ( https://hansencloud.com/2014/04/22/varying-file-name-too-long-issues ). Here the easiest thing to use is the wonderful Robocopy /MIR! Create an empty folder in the root of the drive and copy it into the DFSR folder using the /mir flag in robocopy. This will “mirror” the source folder into the destination folder.
Now the DFSR folder should be completely empty.
Step 5: Start the DFS Replication service (start-service DFSR)
Step 6: Check for validating event logs.
Event 4102, DFSR
The DFS Replication service initialized the replicated folder at local path D:\xxxxxx and is waiting to perform initial replication. The replicated folder will remain in this state until it has received replicated data, directly or indirectly, from the designated primary member.
Additional Information:
Replicated Folder Name: XXXXXXX
Replicated Folder ID: XXXXXXXXXXXXXXXXXXXX
Replication Group Name: XXXXX\XXXX
Replication Group ID: XXXXXXXXXX
Member ID: XXXXXXXXXXXXX
Event 4412, DFSR
The DFS Replication service detected that a file was changed on multiple servers. A conflict resolution algorithm was used to determine the winning file. The losing file was moved to the Conflict and Deleted folder.
Additional Information:
Original File Path: D:\XXXXXXX
New Name in Conflict Folder: XXXXXXXXXXX
Replicated Folder Root: D:\XXXXXXXX
File ID: XXXXXXXXXXXXXXXX
Replicated Folder Name: XXXXXXXXXXXX
Replicated Folder ID: XXXXXXXXXXXXXXX
Replication Group Name: XXXXXXXXXXXXXX
Replication Group ID: XXXXXXXXXXXXXXXXX
Member ID: XXXXXXXXXXXXXXXXXXXX
There you go! You’ve done it! Microsoft said you had to contact their support to fix it, but you crafty devil – you’ve gone and done it yourself.
I hope I’ve made your day at least a little bit easier.
December 20, 2017 at 5:54 am
Hi.
In the Robocopy i suppose it should be “System Volume Information\DFSR” and not DFS?
Thanks for a good article.
December 20, 2017 at 6:36 am
You are correct. I apparently typed incorrectly when taking the screenshot.
December 20, 2017 at 6:42 am
I have updated the screenshot with the correct string, thanks.
December 20, 2017 at 8:43 am
No problem, i wanted to be sure that i delete the correct folder.
Is it enough to leave the DFRS folder and empty the files and folders in the DFRS directory or should i remove the complete DFSR folder?.
Can i do this when there are clients logged on and mabye using files in the DFRS directory?
December 20, 2017 at 10:31 am
I believe deleting all the files in the folder would accomplish the same thing, but at that point you may as well be safe and delete the folder as well. If clients are logged on and using DFS there is a chance (if the client has a persistant connection) they will maintain access to the share. Though I would just assume that anyone using DFS will be impacted until the service rebuilds the database.
April 1, 2019 at 2:19 pm
This just saved me a ton of time. Thank you.
March 11, 2020 at 3:58 pm
P2V’d a couple servers that were DFSRing and DFSR broke hard. Nuked the DFSR folder with your robocopy command, started the service and voila resync is happening. Thanks mucho!
October 28, 2020 at 2:12 pm
Instead of trying to give Administrators access to System Volume Information, I used “psexec -i -s cmd.exe” and used the command line to nuke the DFSR folder.
October 29, 2020 at 4:11 am
Thanks a lot for this!
August 17, 2021 at 10:26 pm
SUCCESS!!! Thank you!
February 2, 2022 at 4:36 am
The DFS Replication service did not find a valid machine configuration file at the location specified. It has created a new machine configuration file with default settings. If you made custom changes to the configuration file, such as setting a static RPC port for replication, these changes have been lost. You must recreate these changes. Event ID: 6702
Received above error in DFS diagnostics report. I hope its normal. When this error will vanish?
BTW, wonderful article to fix the corrupt file issue.