Uncategorized

A while back i encountered the following events at a customers SharePoint Farm: Event ID10038 Query server removed from rotation.

The full event is shown below:

 Rule ID: Microsoft.Office.Sharepoint.Server.2007.Query_server_removed_from_rotation

Description: This alert indicates that a query server has been removed from the load-balanced rotation because IIS has encountered errors while communicating with the query server.

 Rule Category: Event Collection
Rule Target: Microsoft.Office.Sharepoint.Server.2007.MOSS.Application
Alert Type: Error
Event ID: 10038
Event Source: Office SharePoint Server Search

This error can occur if the architecture of the farm has the functionality of the Office Search Service split up between the index and dedicated Query servers. It however does not mention what is the cause of this problem, nor does it even classify the issue as a problem as the Event is shown as a warning.

However when a query server is removed from the rotation, the propogation of the index will not occur on that server and SharePoint will divert user queries to other Servers running Office Search Service with the Query role. When this happens often and on multiple servers the performance of the Office Search Service

IIS WAMREG admin service

In this case the actual problem was a missing “Local Activation” permission setting on the DCOM Service “IIS WAMREG admin Service”. This service is a part of the IIS “Web Application Manager” (WAM). The WAM provides the inter-process communication (IPC) mechanism between IIS-hosted processes and those that are not hosted by IIS. This was indicated by an error in the eventlog similar to the one below:

Type: Error
Source: DCOM
Event ID: 10016
Description:
The application-specific permissions settings do not grant Local Activation permission for the COM Server application with CLSID {CLSID} to the user DomainNameUserName SID {SID}. This security permission can be modified using the Component Services administration tool.

Fixing this issue is rather simple on Windows 2003 and 2008, you just start the  dcomcnfg.exe utility (with elevated privileges) and navigate to the “IIS WAMREG admin service”. In the security tab click Edit in the “Launch and Activation Settings” field. Add the account or group and select “Local Activation” for each of the accounts or groups added then click Ok.

 In Windows 2008 R2 however you will first need to grant your administration account access to be able to change these permissions. By default Windows 2008 R2 has limited access to this dialog through an ACL on a key in the registry which the Trusted Installer is owner of and all other accounts only have read permissions to.

The actual key used by the IIS WAMREG admin service Security dialog is:

HKEY_CLASSES_ROOTAppID{61738644-F196-11D0-9953-00C04FD919C1}

After granting your administration account access to this key you can then start the dcomcnfg.exe and follow the above procedure to add the missing accounts or groups and grant those the “Local Activation” permission.

After changing the permissions for the accounts you must perform an IISRESET on the server.

Remember to prevent this error from occurring again on Servers Hosting SharePoint you will have to add all the Application Pool accounts including the accounts used for the Central Administration (only if central administration is running on this server) and Shared Service Provider Administration web applications.

Firewall

The event 10038 might also occur if a network issue is preventing the Search (Index) server to connect to the Query servers. Since Windows 2008 the Firewall is enabled out of the box, remember to allow all inter-farm and windows server/domain traffic between the servers hosting SharePoint Server.

In the SharePoint business administrating the SharePoint farm often involves also taking care of the databases SharePoint runs on. Even if your organisation employs dedicated DBAs often the limitations of what you can do with the databases and the procedures for backup and restore will have the SharePoint administrators involved at least partly with the creation of procedures for backup and restore and regular maintenance jobs.

For testing purposes, for example performance tests or granular restores with the SQL server backup and restore functions sometimes will involve restoring the SharePoint databases to another SQL Server instance or a different server even. In these scenario’s when you have restored the database, the database users will be orphaned, even if the SQL login names on the new server are exactly the same they will still become orphaned. This is caused due to the fact that the database users are identified with SID’s and when you restore a database to another server or instance these SID’s no longer match with the SID’s of the SQL Server logins in the Master Database.

When you try to reset the permissions for the SQL Server Logins in the database you will recieve an error message and the permissions will not be reset.

Error 15023: User or role <username> already exists in the current database.

This problem can be fixed however by useing  Transact-SQL scripts that can help you identify and fix the problem. To identify what users are orphaned you can use the following script, it will return a list of all the database users that are orphaned.

USE <Database Name>
GO

sp_change_users_login @Action=’Report’
GO

When these users are identified you can match them in the restored database to SQL Server logins with the following script. You can choose any SQL Server Login to match the database user with this method.

USE <Database Name>
GO

sp_change_users_login @Action=’update_one’,
@UserNamePattern='<Database User Name>’,
@LoginName='<SQL Server Login Name>’
GO

As an alternative you can also use the following script if the database users in the restored database are named exactly like the SQL Server Logins you want to link.

EXEC sp_change_users_login ‘Auto_Fix’, ‘<User Name>’
GO

 This will solve the problem of orphaned users and when adding the database to SharePoint will prevent any Access Denied messages in your Web Application from showing up.

 

When working with SharePoint Databases oftentimes the only way SharePoint identifies the database is via a GUID. Sometimes this can be frustrating because you want to know exactly what database you are working with. As far as i know there is no stsadm command available to easily translate GUID to Database name.

Word of warning, the following script directly queries a SharePoint database, this is never a good idea. Therefor i reccomend you restore the database to a restore or test server where you can execute the query as to prevent anypotential  issues with your production environment.

Luckily there is a way to get this information from SQL Server, to do this you must run a query against the Configuration database (See warning above).

First you need to open Management Studio

  1. Select New Query.
  2. Select the Configuration Database from the Available Databases dropdown list.
  3. Copy the script below and paste it into the Query window and press Execute.

Select ID, Name from objects
where properties like
‘%Microsoft.SharePoint.Administration.SPContentDatabase%m_nWarningSiteCount%’

This will output the Database GUID and Database Name in a nice list, giving you the data you need to identify what database SharePoint is talking about when it is talking GUID’s to you.

As stated There is some risk involved in running queries against SharePoint databases, Microsoft advises not to run any query against a SharePoint database as it might interfere with any SharePoint processes running against the databases. When this happens the database will lose support status and if you need to contact Microsoft for support with issues in the future you might be asked to replace the database with a fresh copy. I usually restore a recent backup of the configuration database to a restore server or a test server and run the query there. This allows me to get the data but still comply with the “No query for you!” rule that SharePoint databases have.

 

Not too long ago I posted about how to remove (detatch) content databases safely from the SharePoint Farm. to enable you to move/replace databases between Web Applications and or Database Servers. One of the reasons why it is important to use the preparetomove  command is to prevent problems when you reattach these databases to the farm.

Usually these problems show themselves via the event viewer in the form of event ID 5555 and 7888, to solve these sync issues that will prevent profile information inside the content databases to be updated you can follow these steps.

 First we want to find all databases that are not synced up correctly, to do this we use the command:

stsadm -o sync -listolddatabases <number>

This will give you a  list of database GUID’s and the date/time they were last synchronized. We need the GUID’s to get the databases in sync again. The following command will change all GUID’s in the database as old and the next time the indexer runs it will generate new GUID’s for that content database.

 stsadm -o preparetomove -contentdb <databse server name>:<content database name> -oldcontentdb <GUID>

 where <GUID> is a guid from the content database in the list generated with the listolddatabases command above. Repeat this for every database in the list you want to ‘recover’

Once your next full crawl completes you can run the stsadm -o sync -listolddatabases <number> command again.  And anything still on the list can likely be removed at this point.  You can remove these by running the following command.

stsadm -o sync -deleteolddatabases <number>

This will delete all GUID entries in the SSP for anything that is out of sync for more than <number> of  days.  After running these commands all events in the eventlog should stop to appear in the eventlog.

You can test this by decreasing the sync timing job to 5 minutes and check the eventlog to see if these events are gone, you can do this by running the following command.

stsadm -o sync -synctiming “m:5”

Then reset the sync timing back to default value of 1 hour by running the following command.

stsadm -o sync -synctiming “h:1”

This procedure will solve your out of sync issues you may have in your farm and will ensure that the relation between content database and SSP is optimal. This also means profile information residing inside the content database will be updated again and the crawl should show less errors as well. As i have stated in the previous post you can prevent this problem by using stsadm -o preparetomove before detaching a database.

Last week I posted about the problem with local loopback connections, and showed how to solve them with a registry setting. So i was positively surprised when i checked Gary Lapointe’s blog today and saw he had created a new extension for stsadm.exe. This new extention configures the SharePoint site URL’s allowing local loopback connections to thiose URL’s, it takes a lot of work out of your hands by configuring all URL’s in the backconnectionhostnames key to solve the connection problems.

A small recap of the command from his blog:

C:>stsadm -help gl-setbackconnectionhostnames

stsadm -o gl-setbackconnectionhostnames
Sets the BackConnectionHostNames registry key with the URLs associated with each web application.
Parameters:
        [-updatefarm (update all servers in the farm)]
        [-username <DOMAINuser (must have rights to update the registry on each server)>]
        [-password <password>]

His post can be found here as well as links to his extensions and scripts.

When you encounter errors when you try to access the SharePoint Sites from a Front End locally. This problem occurs because since Windows Server 2003 SP1, Windows includes a new security feature named loopback check functionality. By default, loopback check functionality is turned on in Windows Server 2003 SP1 and Windows 2008.

Ever since that time i have ever only heared people advise me to disable the loopback check entirely by using a registry setting called DisableLoopbackCheck. You can configure this at the following location.

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlLsa

Create a DWORD called DisableLoopbackCheck and set it’s value to 1

However, this does disable the entire function and thus reduces the security of the affected system. In some cases this might still be a valid way to circumvent the problem but Microsoft is offering a better solution that enable you to set exclusions for hostnames used on the server called BackConnectionHostNames.

The following registry entry is used to configure this

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlLsaMSV1_0

Create a  multi-sting value BackConnectionHostNames that contains a list of hostnames.

For SharePoint this means that you need to enter the SharePoint sites host headers in this list, alternatively you may want to investigate configuring this via a script and or via Group Policy for manageability.

Note: This solution also solves the problem of connecting SQL Server management Studio to the local Database Server for management.

Sometimes you need to remove a content database from the Farm. Either to move it to another web application or database server or when dealing with an elaborate restore scenario. Prior to running the deletecontentdb command you need to run preparetomove on the database you are going to detach from SharePoint.

This ensures that the relationship between the database and the SSP will be severed and ensures that you can cleanly reattatch it. Failing to run the preparetomove command may result in the deletion of user membership metadata including the selections made in the Privacy and Grouping section on the Edit Profile page for My Site memberships.

STSADM –o preparetomove –contentdb <database server name:database name> -site <site url>

You can also undo the last preparetomove action by using the command undo, this will revert the changes and connect the database to the SSP again.

STSADM –o preparetomove –contentdb <database server name:database name> -site <site url> -undo

Note that the deletecontentdb does not actually delete the database nor does it detach the database in SQL Server, it only detaches the database from the SharePoint Farm Configuration.

STSADM -o deletecontentdb –url <site url> -databasename <database name> -databaseserver <database server name>

Then after you have moved the databse in/from SQL Server you can add it to the Web Application or a new Web Application by using the following command.

STSADM –o addcontentdb –url <site url> -databasename <database name> -databaseserver <database server name>

When moving databases in SQL always detach them from SharePoint with the deletecontentdb command and always use the preparetomove command prior to using the deletecontentb command (or shut down the entire farm). 

It has been a while since i have been posting on here. This was mainliy due to a busy schedule and vacation time hitting. However something curious has happened when I was installing Windows 2008 Server on a system for a client.

The server had given the client some problems and it was reinstalled a few times prior to my involvement. And when i tried to install the 64bit version of Server 2008 it would not recognize any of the disks attached to the system. At first i thought it was a problem with the drivers for the controller and downloaded the latest (still old mind you) drivers for it from the vendor site.

These however could not be loaded at all, the system gave a message about an invalid certificate. And i suspect that the certificate used to sign those drivers has expired and an update had not been posted to the vendor’s site. When testing the 32bit version of the OS it would run the installer flawlessly. I guess due to the ability to install unsigned drivers and would thus even accept the expired certificate.

Somewhere between calling the vendor outright (on a saturday?) and demanding a new driver, and opening up the WinPE environment on the setup disk and disabeling the requirement for signed drivers when installing 64bit OSses i got a peek at the disk configuration. And saw that for some reason there was a software mirror definded on the disks (the system had a good RAID controller so why this would be configured in Windows is beyond me but there it was). 

When the mirror configuration and partition information was removed the Windows 2008 Setup PE environment was able to see the disks as normal and would install without problems and without the need for the driver. So for next time when installing a system, i would reccomend to check for software RAID configurations on the disks and remove those when this problem pops up.

We had a Windows 2008 Server Network Load balancing setup  consisting of two Windows 2008 Servers wich had two NIC’s each. The Load Balancing was configured to use the one of the two Interfaces for dedicated Load balancing, the other interfaces were configured for normal network traffic as shown in the picture below.

NLB hardware Configuration

The virtual cluster IP address responded if you send it a ping from another host on the same subnet that the cluster was installed on. However, if you were on another segment of the network  it didn’t respond.

Originally we were send off track due to an issue with the vendor’s NIC Teaming solution that was enabled by default and had proved difficult to remove.

This however proved not be the cause of our problem with the NLB configuration not responding to off-subnet network locations.

What did prove to be the case is that in Windows 2008 Server NLB by default IP Forwarding is not enabled. This is the feature of Windows networking that, in the context of NLB, allows responses to requests sent to one NIC to be routed out the other. It can be enabled by using a netsh.exe command.

“netsh interface ipv4 set int “[name of the NIC]” forwarding=enabled”

The command will respond with an “Ok.” and no reboot is required for the configuration to apply. This command has to be run on both machines and only for the interface configured for normal network traffic.

In this case the NLB became active immediately  after running the commands on both machines. This little command is something that is good to remember when installing Windows Server 2008 NLB solutions and I for one will add it to my checklist.