In some of our SP2010 environments, we're finding the Sharepoint crawler is only able to crawl a small set of content, but not does not completely crawl all of the lists/libraries that we really need it to. With what appear to be identical settings, we have this working on other servers, we are just unable to determine why on some servers this does not work.
We've tried many things from checking rules to using alternate access mappings, however the issue remains unresolved. There are no related errors show in event viewer or sharepoint logs.
We noticed a few things in fiddler that are interesting:
On servers where everthing works, there are a large amount of requests to the web service (sitedata.asmx) - which is how sharepoint is getting the lists/library data and URLs
On the servers not working correctly, no requests are made to sitedata.asmx, but we do see a single request to spdisco.aspx, and even stranger we see random requests to "<site path>/pages/none" for each page library in the site hierarch (not blocked
by rules).
Anyone ever see something like this?
It seems that sharepoint can't or doesn't recognize our content source url as being a sharepoint site collection?