Error Opening Permissions Page of a Document Set

I have a library full of document sets and on some of them I can not open the Permissions (User.aspx) page. I get a error in the ULS of: 

System.InvalidOperationException: Operation is not valid due to the current state of the object.   at Microsoft.SharePoint.ApplicationPages.CBaseAclPage.ParseAclObj(String str)    at Microsoft.SharePoint.ApplicationPages.CBaseAclPage.OnLoad(EventArgs e)    at System.Web.UI.Control.LoadRecursive()    at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Any idea whats causing this or how to correct it? I've tried all admin accounts possible.