Hi all, I'm working for a client that has a failing UPS incremental sync. The scheduled sync was running fine until a couple of weeks ago when it suddenly stopped working. Coincidently, this was also at the same time that they noticed the incremental search crawl and UPS audience compilation timer jobs failing. A timer service stop/start fixed the search/audience compilation problems, but they’re still having issues with the UPS incremental sync. I see the following errors in the ULS when the sync fails (nothing logged in FIM operations window):
08/15/2013 09:19:05.73 OWSTIMER.EXE (0x1E40) 0x1C78 SharePoint Foundation Timer 6398 Critical The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob (ID ad907731-0878-411e-9184-2bd8eea74501) threw an exception. More information is included below. Access to the requested resource(s) is denied 36285c84-5805-4af3-9df5-88254e67ba9f?
…followed by
08/15/2013 09:19:05.73 OWSTIMER.EXE (0x1E40) 0x1C78 SharePoint Foundation Timer 72ae Unexpected Exception stack trace: at Microsoft.ResourceManagement.WebServices.Client.ResourceTemplate.GetResource(UniqueIdentifier identifier, String[] attributeNames, Nullable`1 resourceTime) at Microsoft.ResourceManagement.WebServices.ResourceManager..ctor(UniqueIdentifier resourceIdentifier, String typeName, String[] attributeNames, CultureInfo locale, Boolean includePermissionHints, TimeZoneInfo localTimeZone) at Microsoft.ResourceManagement.WebServices.ResourceManager..ctor(UniqueIdentifier resourceIdentifier, String typeName, String[] attributeNames, CultureInfo locale, Boolean includePermissionHints) at Microsoft.Office.Server.UserProfiles.Synchronization.MAConfiguration..ctor(Guid resourceIdentifier) at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.CreateSteps() at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.Execute() at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result) 36285c84-5805-4af3-9df5-88254e67ba9f
I’ve confirmed the following:
- Were any changes made to farm servers around the time issue first appeared (group policy etc.) -No
- Farm (SP2010 Standard) version number - 14.0.6106.5000 (June 2011 CU) / 14.0.6029.1000 (SP1)
- Are both FIM services started – Yes
- Does the farm account have local admin rights – Yes
- Does the farm account have log-in locally rights – Yes
- Does the farm account have access to relevant FIM binaries - Yes
- Does the network service account have access to relevant FIM binaries – Yes (Network service has full control to this location (D:\Program Files\Microsoft Office Servers\14.0\Service) and this location (D:\Program Files\Microsoft Office Servers\14.0 directory)
- Does the farm account have dbo on all UPS databases – Yes
Steps tried to fix so far:
- Restarted timer service
- Restarted User Profile Synchronization service in Central Administration
- Restarted 'Forefront Identity Manager Service' and 'Forefront Identity Manager Sync Service' (server service)
- Rebooted all farm servers including SQL
- Moved service instance to another server (and back). Errors reported on other server after move:
System.Data.DataException: This request has an invalid target: B87AD086-D1CB-4008-BE44-706FF05D6378. The invalid target identifier is F2CC710F-99E1-45A2-8F9F-D6A5D1A3CAC4. Microsoft.ResourceManagement: System.Data.DataException: This request has an invalid target: B87AD086-D1CB-4008-BE44-706FF05D6378. The invalid target identifier is F2CC710F-99E1-45A2-8F9F-D6A5D1A3CAC4. at Microsoft.ResourceManagement.Utilities.ExceptionManager.ThrowException(Exception exception) at Microsoft.ResourceManagement.Data.DataAccess.DoRequestCreation(RequestType request, Guid cause, Boolean doEvaluation) at Microsoft.ResourceManagement.Data.DataAccess.CreateRequest(RequestType request, Guid cause, Boolean doEvaluation) at Microsoft.ResourceManagement.WebServices.RequestDispatcher.CreateRequest(UniqueIdentifier requestor, UniqueIdentifier targetIdentifier, OperationType operation, String businessJustification, List`1 requestParameters, CultureInfo locale, Boolean isChildRequest, Guid cause, Boolean doEvaluation) at Microsoft.ResourceManagement.WebServices.ResourceManagementService.Put(Message request) ?
Other issues
When creating a new sync connection in CA, the following message is displayed when the connection is saved:
“Unable to process create message”
I also see this error message in the logs:
A update on the configuration of a MA or MV failed to replicate to a target connector directory that is capable of storing MA/MV configurations. As a result, the MA/MV configuration data in this connector directory is not up to date. Please correct the condition that causes the error, and triggers a resync by updating the password information of the target MA. Additional information: Error Code: 0x80230709 Error Message: (The extension operation aborted due to an internal error in FIM Synchronization Service.) Operation: Create MA Name of the MA to replicate: MOSSAD-BLAH Guid of the MA to replicate: {BF18A3FA-190D-49F8-878B-D469D545B2A8} Name of the target MA: ILMMA Guid of the target MA: {F2CC710F-99E1-45A2-8F9F-D6A5D1A3CAC4}
The management agent appears in FIM, but isn’t displayed in SharePoint CA (UPS SA). At present we have a few test MA’s that appear in FIM but not in SharePoint CA (UPS SA).
I’m at a loss as to why this incremental sync has stopped working. It’s also strange that test sync connections are appearing in FIM despite error displayed in CA when provisioning them. Stranger sill, the test sync connections don’t appear in CA, but appear in FIM.
Potential next steps include resetting the profile synchronization DB as detailed here http://technet.microsoft.com/en-us/library/ff681014(v=office.14).aspx#resetSync. If anyone has had any other suggestions, I’d be very grateful to hear about them.
Many thanks