Wednesday November 25 2009
Information Technology Central Services at the University of Michigan

ITCS Service Status

Incident Type: Service Degradation Service Degradation
Start Time: Monday, November 2, 2009 2:00PM
Service Restored: Wednesday, November 4, 2009 4:42PM
Service(s) Affected: Windows (Campus Forest)
Server Name: UMROOT Windows Infrastructure
Symptoms: A number of GPOs (Group Policy Objects) were accidently deleted from UMROOT domain today. We are working to restore them. End user symptoms can be widely ranging from mild to serious depending on the details of each GPO. Please contact windows.support@umich.edu if you suspect a missing GPO in your OU.
Comments: 11/3 12:30am TLW: GPOs have been restored from backup. We are not sure whether they will function properly. Please contact windows.support@umich.edu if you see problems tomorrow.

11/3 9:00am TLW: We are continuing to troubleshoot this problem and work with units that are having GPO problems. Please contact us if you are having problems.

11/3 12:08 PM EET: We have restored most of the deleted GPOs and are in the process of fixing their permissions. If you are still experiencing problems, please contact us at windows.support@umich.edu. If at all possible. please include the name of the problem GPO(s) and its GUID. You may find the GUID in error messages in the Windows Event Logs. Thank you for your continued patience.

11/3 3:00 PM EET: Many GPO problems have been fixed, but many still remain. We have a highest-priority case open with Microsoft and are working with them on a solution. More updates as work progresses.

11/3 4:30 PM: At 5:00 PM we are going to perform a new restore of the AD GPO objects. We will be taking itcs-dc01 off-line in order to perform the authoritative restore. We expect that this will disconnect Outlook users. We extending our estimated fix time to midnight tonight.

11/3 5:00 PM: We have changed the target to itcs-dc03. itcs-dc01 will not go offline.

11/4 1:30am TLW: All Group Policy Objects were successfully restored to their state on Sunday night at midnight. We think everything is working properly. Please test your GPOs and contact us if you have any problems.

We will be keeping this incident open until midday Wednesday or until we are sure there are no lingering problems. Thanks very much for your patience.

11/4 11:20am TLW: The majority of GPOs are working. Some restored GPOs are no longer linked to the OUs. You should be able to find any missing GPOs in the ???Group Policy Objects??? folder within the Group Policy Management Console and re-link them to your OU. Some GPOS are still being reported as missing.

Please continue to report any other errors or GPO problems that you are running into. If you contact us with a problem, please send us as many details as possible. The name of the GPO, which OU it should be linked to, and the GUID if you know it.

11/4 16:42 EET: We have received only one additional report of a possible GPO issue, and we will be working with the department in question as they gather data for us. No other reports have been received. At this time, we believe there are no remaining problems with UMROOT GPOs.

Please continue to report any other errors or GPO problems that you are running into. If you contact us with a problem, please send us as many details as possible. The name of the GPO, which OU it should be linked to, and the GUID if you know it.
Contact: windows.support@umich.edu

« Go Back