So we’re setting up a crawl of a TWiki site as one source in a suite of content sources.

So far so good, once the authentication was sorted we noticed a problem, only the root Url of the site was getting crawled.

Various ideas were thrown around about nofollow and noindex attributes but we couldn’t find anything wrong with our configuration and nothing seemed to fit the problem.

I noticed that this particular TWiki installation was case sensitive to Urls by accident (thought those days were gone, configurable apparently) and that got me thinking.

By kicking a crawl off i noticed that SharePoint was requesting lower case urls from the Site for every link on the home page getting a 404 and stopping.

Ok penny drops but why is SharePoint sending a lower case url, well… this is by design as part of the crawler’s normalization of urls (http://blogs.msdn.com/b/enterprisesearch/archive/2010/07/09/crawling-case-sensitive-repositories-using-sharepoint-server-2010.aspx)

In 2010 if you’re setting up a crawl rule that checkbox you’ve ignored called Match Case (badly named surely Preserve Url Casing would get the point across better) just needs to be set and viola the crawler will preserve the case of Urls it requests.

You know how it is, you start developing a project and then 6 months later you look back and realise you have to document everything you've produced.

I've just gone through that process and need a quick list of all the features ids scattered around various subdirectories of a large project.

sample:

          <Feature
Id="886f12cf-97ca-4789-baf8-6f13f9f2cedf" Title="PGPSO
Contract Management Project Upgrade" Description="Feature
that upgrades Project Sites for Contract Management." Version="12.0.0.0" Hidden="FALSE" Scope="Web" DefaultResourceFile="core" ReceiverAssembly="PGPSO_CM_Project_Upgrade,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=aa0408b86137366a" ReceiverClass="PGPSO_CM_Project_Upgrade.PGPSO_CM_Project_Upgrade" xmlns="http://schemas.microsoft.com/sharepoint/">
        

Firing up Powershell navigating to the root of the solution folders and running this command gets me the list

          gci
-recurse -filter feature.xml |% {
$contents=get-content $_.fullname; $x=[XML]$contents; "{0}
{1}" -f $x.Feature.Id, $x.feature.title }
        

result:

 

gci is an alias for get-childitem which allow you to recurse subfolders and provide a filter parameter. Then use get-content to open the file, convert to an XML object and then directly reference the Id and title of the feature.xml file.

 

 

 

Recently I went through the process of indexing a subversion source code repository with SharePoint. I thought I'd share those steps as OOTB SharePoint won't index ps1, cs or vb files.

Setting up search to index these files works either if the files themselves live in a document library or are external to SharePoint. The process to index files from other source control systems will vary depending on how you can get access to the source files. If you need to index SourceSafe you can set up what's called a mirror directory that automatically save the files from your repositories on disk and I suspect you can index Team Foundation Server via its Web Access URL's although I've not tried that.

The subversion side of things is pretty easy, pick the repository you want and export the latest version using the svn client i.e. svn export svn://devhosting/svn/webparts d:\SVNExport\webparts. Script the export of each repository and then schedule it.

On the SharePoint side you set up a new content source to crawl the directories.

In this case the Indexing is on a separate machine so we enter the UNC path. Make sure the content access account has read rights to the share. If needed you can setup separate credentials for this source.

In the SSP on the Search Setting page, click New Content Source under Content source and crawl schedules

The problem now is if you start a full crawl typically only the .txt files are indexed as the SharePoint indexers have no idea what to do with file extensions it doesn't recognise.

There are a couple of steps to getting new file extensions indexed. This assumes you are a Search Service administrator.

First add the extension to File Types

1. On the Search Administration page, click File Types under Crawling.

2. On the Manage File Types page, click New File Type.

3. On the Add File Type page, type the file name extension in the File extension box for the file type that you want to add.
To search for PowerShell files, type ps1
Do not include the period (.) character in front of the file name extension.

4.Click OK.

5. Rinse and repeat for each file type that you want to add.

The second step in getting the file extensions recognised is to add it to the registry entries the SharePoint Server Search service reads when it starts up. This key is located at

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Setup\ContentIndexCommon\Filters\Extension

Add a new key, enter the extension including the dot i.e. .ps1.

Save and set its default value to be {4A3DD7AB-0A6B-43B0-8A90-0D8B0CC36AAB}. This means use the text parser Ifilter tquery.dll for this extension.

And a new key for each file extension you want indexed in this case cs,ps1 and aspx but you can add vb vbs or whatever other text files you need indexed.

Stop and start the search service with these commands

net stop osearch

net start osearch

Now do a full crawl of your content type and your files should have been full text indexed. The crawl log is useful in seeing if the filtering barfed on your files.

 

Now you can go to the Search Center enter your keyword and get a list of code files back.

Here I've set up a custom scope, search page, and added a custom search tab so separate the code results on its own. I won't go into it here but there is a good post here that shows how you do this.

Even better with SharePoint Search if you know you want PowerShell files only you can enter the fileextension keyword and search will filter out everything but PowerShell files.

 

Searching your entire code repository with subsecond query times is now pretty easy.