When installing a Windows 2008 Failover Cluster and for whatever reason the creation of the Virtual Computer Objects, Cluster Name Object and the dns records fail you will have to create these manually.

For example you have Server-A and Server-B and create a two node Failover Cluster with these named Cluster-A and you create a Virtual Computer Object named Application-A.

In Active Directory you will need to create a Computer Account for the Cluster-A CNO (Cluster Name Object) and another for the Virtual Computer Object Application-A. Furthermore you will have to give the CNO (Cluster-A$) Full Control permissions on the ACL on the Application-A VCO.

The same has to be done for the dns entries, where the CNO (Computer Account in Active Directory) must have permissions on the dns entries. It might seem strange since this account is a Computer Object but without these permissions and records the cluster will not start the Service. 

Since Microsoft has deprecated the “RequireDNS” option, this would otherwise prevent you from serving your clustered application. Note that the private property “RequireDNS” still exists on a Windows Server 2008 Failover Cluster for backward compatibility, however changing it’s setting has no effect.  In Windows Server 2008 R2 the property has been removed completely.

Some things change between versions, especially with Microsoft products the new management features often are accompanied by a new way of ‘getting things done’. One of these is the disappearance of the cluster.log logfile in Windows 2008 Server. When troubleshooting a Failover Cluster this file used to be the best source of information on the cluster and what would be ailing the configuration. In Windows 2008 Server Failover Cluster you are however unlikely to find this useful file on the system.

The cluster.log has been replaced by a more sophisticated event based tracing system, based of the event model used in Windows 2008 Server and Vista. This system has also replaced the old Eventlog system used in previous versions of the OS.

So how does one get to the cluster.log file in Windows 2008 Server? .

Open a Command Prompt (cmd.exe) and type the following command:

Cluster /Cluster:yourclustername log /gen /copy “C:tmp”

The files containing the log information will be stored in c:tmp or any other directory you choose to create these in, they are in clear text and can be opened via notepad for easy access. To me this beats the GUI by a long shot as personally i dislike the slow to respond MMC 3.0 Event Tracing GUI, and prefer to get my information quick and dirty from the Command Line.

I hope this information will help some of you out tehre who might have wondered what happened to the cluster.log logfile.

The other day the normal routine was disturbed by several users reporting that they were unable to log on to their Terminal Server Environment.  The following event was recorded on the Windows 2003 Terminal Server.

Event Type: Error
Event Source: Winlogon
Event ID: 1219

Logon rejected for DOMAINUsername. Unable to obtain Terminal Server User Configuration. Error: Access is denied.

There were thirty five users who were generating this event when log-in into the server, on the user end the error was displayed as : Access is Denied. and they were unable to log-in to the Terminal Server via RDP.

Usualy this event is logged when DNS or Domain connection problems prevent communication between the Server and a Domain controller, however in this case that is not the problem. Furthermore a small group of users was able to log in and run applications on the server, this seemed to be random.

To investigate the problem further and see what would happen if a user would be able to log in despite the existing problem we changed the IgnoreRegUserConfigErrors setting in the Registry to 1.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlTerminal Server]

After a reboot we had a test user log in and checked the status of the account, what happened was that the log-in was permitted but the properties of the object in Active Directory were being ignored. The home drive was not mapped, and the application that was supposed to start after the user had logged in as specified in the Environment tab on the Active Directory Object never ran.

The test user was also unable to log on to any other Terminal Server in the domain, and the same event was logged on those servers as well. It was not long after this that we got word that a project group had been migrating Exchange Mailboxes and we soon identified that all accounts that were having problems were the accounts that were migrated. Lucky for us this was limited to just thirty five accounts. The group of people that was able to work on this server had not been migrated due to their accounts residing in a different Organisational Unit in the Active Directory.

Investigation into the account Object in Active Directory revealed that the migration process had changed properties on the account Object, something that was not supposed to happen. At first these looked to be very innocent an additional x.500 address for example would not have caused the failure to log-on. But we also found that the security settings had been changed and ‘SELF’ was removed all permissions on the migrated account’s.

Changing these back resolved the issue of the users who were unable to log-in to the Terminal Server. This problem and the resulting event description are generated because the Winlogon process runs under the user’s credentials when logging on and passing the account properties from the Active Directory Object to the Terminal Server Service. The absence of  the ‘SELF’ permissions generated an Access Denied on the query for these properties resulting in the event and error shown to the user.

The second lesson that one can learn from stories like these is; Test before even thinking of running your script’s and or assumed safe to install and use applications on any production environment.  Make it part of your procedures and best practises to always test every line of code that runs against your production environment.  Communicate what and when it will be implemented and preferably test it on a copy of the production environment.

Using DTAP (OTAP for us Dutchies) and enforcing the testing of  code before releasing it ohe next level goes a long way, and in these times where the buzz is all about vitalization there really no longer is an excuse not to test your code/applications before they hit production and perhaps wreak havoc on your existing infrastructure. Costing you time and money in the process.

As the saying goes:  “A failure to plan, usually leads to a plan to failure”.
A little bit of information about Windows Terminal Server 2003 and Winlogon in this document: http://technet.microsoft.com/en-us/library/cc755399.aspx

Logon and logoff information is found in the Event IDs 528 and 540 for a successful logon and  event ID 538 for a logoff, there are more event’s where this information can be seen but these will give you a general idea where the accounts log on and off. If you are troubleshooting why an account is failing or for example who or what is locking out that user that call’s in every day to have his account unlocked you may want to look at the following Events.
Logon Types in Event ID’s 528, 540 and 538:

Logon Type 2 – Interactive
This type indicates that the user was either logged on either physical or via some sort of KVM (iLo for HP Blades for example) solution.

Logon Type 3 – Network
Typically indicated that the user was logged in via a SMB connection (File Share, Printers), IIS Windows Integrated or Kerberos authentication also logged the logontype as type 3.

Logon Type 4 – Batch
This indicates that the account was logged in from a scheduled job. Also happens when using at.exe to run a job immediately.

Logon Type 5 – Service
This indicates the account that logged in is configured in a Service.

Logon Type 7 – Unlock
When a user unlocks the system after the desktop was locked type 7 is logged in the eventlog.

Logon Type 8 – NetworkCleartext
Typicaly seen when using basic authentication in IIS, unless encryption is being ofered at a lower level (SSL for example) this would be very bad.

Logon Type 9 – NewCredentials
This indicates that a user used the runas /netonly to authenticate to a remote source on the network.

Logon Type 10 – RemoteInteractive
This indicates that the user logged in interactively via RDP (Terminal Server, Citrix Desktop). Users and Administrators logging in via the RDP protocol will

show up as type 10 in the event viewer.

Logon Type 11 – CachedInteractive
This is usualy seen on laptops where an user logges in on a notebook that is not connected to the network, Windows caches the cridentials if the user was previously logged on successfully on the network and this enabeles people to still be able to use their notebooks when not directly connected to the network. This may also indicate a problem in the LAN/WAN connections if this is logged frequently within desktops in the domain.

The indicator that a logon attempt was made when the account was logged off is the registration of Event ID 539. This event will not give a reason why the account was locked, it serves solely as an indicator to the fact that it is locked. Event ID’s 675 and 681 give a lot of information on what went wrong with the logon process. The following list shows the failiure codes in binary and hex and their descriptions for Event ID 675:

1 0x1 Client’s entry in database has expired  
2 0x2 Server’s entry in database has expired  
3 0x3 Requested protocol version # not supported  
4 0x4 Client’s key encrypted in old master key  
5 0x5 Server’s key encrypted in old master key  
6 0x6 Client not found in Kerberos database Bad user name, or new account has not replicated to DC yet
7 0x7 Server not found in Kerberos database  New computer account has not replicated yet or computer is pre-w2k
8 0x8 Multiple principal entries in database  
9 0x9 The client or server has a null key  administrator should reset the password on the account
10 0xA Ticket not eligible for postdating  
11 0xB Requested start time is later than end time  
12 0xC KDC policy rejects request Workstation/logon time restriction
13 0xD KDC cannot accommodate requested option  
14 0xE KDC has no support for encryption type  
15 0xF KDC has no support for checksum type  
16 0x10 KDC has no support for padata type  
17 0x11 KDC has no support for transited type  
18 0x12 Clients credentials have been revoked Account disabled, expired, or locked out.
19 0x13 Credentials for server have been revoked  
20 0x14 TGT has been revoked  
21 0x15 Client not yet valid – try again later  
22 0x16 Server not yet valid – try again later  
23 0x17 Password has expired The user’s password has expired.
24 0x18 Pre-authentication information was invalid Usually means bad password
25 0x19 Additional pre-authentication required*  
31 0x1F Integrity check on decrypted field failed  
32 0x20 Ticket expired Frequently logged by computer accounts
33 0x21 Ticket not yet valid  
33 0x21 Ticket not yet valid  
34 0x22 Request is a replay  
35 0x23 The ticket isn’t for us  
36 0x24 Ticket and authenticator don’t match  
37 0x25 Clock skew too great Workstation’s clock too far out of sync with the DC’s
38 0x26 Incorrect net address  IP address change?
39 0x27 Protocol version mismatch  
40 0x28 Invalid msg type  
41 0x29 Message stream modified  
42 0x2A Message out of order  
44 0x2C Specified version of key is not available  
45 0x2D Service key not available  
46 0x2E Mutual authentication failed  may be a memory allocation failure
47 0x2F Incorrect message direction  
48 0x30 Alternative authentication method required*  
49 0x31 Incorrect sequence number in message  
50 0x32 Inappropriate type of checksum in message  
60 0x3C Generic error (description in e-text)  
61 0x3D Field is too long for this implementation  

The list of Event ID 681 error codes:

3221225572 The user name doesn’t exist.
3221225578 The user name is correct, but the password is wrong.
3221226036 The user is currently locked out.
3221225586 The account is currently disabled.
3221225583 The user tried to log on outside the user’s time-of-day restrictions.
3221225584 The user tried to log on outside the user’s workstation restrictions.
3221225875 The user account has expired.
3221225585 The user tried to log on with an expired password.
3221226020 The user tried to log on with an account on which the administrator has selected the User must change password at next logon option.

The only bad thing with this type of investigation is that you definitely need to collect the events from all Domain Controllers, and in some cases even from the Workstations and servers where the account is logging in.

Last Friday i had the pleasure to attend the “Meet the Experts” seminar sponsored by Quest Software.  The planned speakers were Joel Oleson, Robin Meure, Mike Watson and Daniel McPherson. 

It was a great day and the sessions were educational and fun to attend and there were a lot of great SharePoint people who took the opportunity to show up and “Meet the Experts”.

The First session was presented By Joel Oleson, who gave his presentation of 10 steps to SharePoint Deployment Success. The slides from this presentation are available on his website at http://www.sharepointjoel.com/Presentations/. A great session and presented with great enthusiasm by Joel.  One which i might add may not be solely for just SharePoint but also other deployments might benefit from his tips.

The seccond Session was presented by Daniel McPherson, he presented the social media aspect of SharePoint through customised applications and features he developed. Think facebook and twitter for the business, one of his points was not to name it “social” but “business” to emphasis the added  business value in these applications.

Joel co-presented another session together with Robin Meure a well known SharePoint expert from The Netherlands. The presentation: SharePoint Logical and Physical Infrastructure Fundamentals witch was very educational even if some of the information was a repeat on what we already (should) know.

The last session before the Experts Q&A was Backup Demystified by Mike Watson. He has some surprising things to say about backing up SharePoint and I am still trying to wrap my head around some of the issues he raised to come up with solutions both working and supported that is 😉

The Q&A was awesome, the audience had some good deep technical questions and even though not all of them could be answered immediately the Experts panel was able to refer them to other experts in the field or give some basic direction to the troubleshooting process.

I also attended the SharePint in the evening, and had a blast there a twitterfeed was put up and Joel stole Robin’s phone to take some pictures of the event. You can find the tweets via by searching for #sp020  on twitter.

All in all i had a great day, returned home way too late and dead tired, but had learned a great deal and had even more to ponder and investigate further.

On another note, Friday i cleared the 070-648 exam. Granting me two cleared exams towards my MCTIP:Enterprise Administrator cridentials. (640 and 642)

I allready had the MCTS cridential from when i took the SharePoint Exam in December 2008, so this did not give me any new abreviations to put on my businesscards or anything. But its always a nice step to check yet another box on the endless list of exams us IT people have to get through.

Personaly i am usualy a little behind on the actual certs and try to be ahead in experience, but i am going to mark some more exams towards the MCTIP certification this year (in the summer). After that I will probably update some of the non-Microsoft checkmarks like ITIL3 and other technology certifications.

For MCTIP: Enterprise Administrator the count is now 2 down, 3 to go.

When you need information on installed or running drivers on a Windows 2008 Server you can build a WMI query to get the information you want or you can use the build in tool driverquery.exe. This utility provides information about drivers and their status.

Just typing driverquery.exe will give you a list of all drivers the OS has installed. A more useful view will be provided if you add the argument /v (verbose). This will among other things provide you with the Driver Type, State, Status and  also with the path and file name of the driver.

This information could be usefull in various ways, providing you with forensic information on malicious installed drivers (that would be bad) or just making sure that the right drivers are installed and started.

And best of all, the utility can be used remotely and even has an argument for providing a domainuser name and password. The report may be formatted by comma or tab separated columns or just as a list.

The difference of this list with the one from pnputil is that it also shows the non 3rd party drivers even more the verbose list will provide you with detailed info on the validity and state of the driver and to me that was very useful.

 The command line options of this utility are :

DRIVERQUERY [/S system [/U username [/P [password]]]]
              [/FO format] [/NH] [/SI] [/V]
    Enables an administrator to display a list of
    installed device drivers.

Parameter List:
      /S     system           Specifies the remote system to connect to.

      /U     [domain]user    Specifies the user context
                              under which the command should execute.

      /P     [password]       Specify the password for the given
                              user context.

      /FO    format           Specifies the type of output to display.
                              Valid values to be passed with the
                              switch are “TABLE”, “LIST”, “CSV”.

      /NH                     Specifies that the “Column Header”
                              should not be displayed. Valid for
                              “TABLE” and “CSV” format only.

      /SI                     Provides information about signed drivers.

      /V                      Displays verbose output. Not valid
                              for signed drivers.

      /?                      Displays this help message.

    DRIVERQUERY /S ipaddress /U user /V
    DRIVERQUERY /S system /U domainuser /P password /FO LIST

Again, an Exchange server is being looked at, in the course of solving a problem with mysterious NDR’s a lot of changes to the configuration of the server were tried to see if the behaviour changed in any way. This was to pinpoint the possible culprit causing the NDR’s. I know , changing configuration to a server like this is not the best idea i could think of either, but Microsoft support engineers working on the case with us were unable to locate the issue through less intrusive means.

One of these tests included disabeling the groupware antivirus solution installed on top of Exchange (not the OS scanner).  During the talk on how dangerous it was going to be to the organisation – as now viruses sure would level the network in mere minutes – i generated a report from the virus scanner.

Messages scanned in the course of 128 days: 245731 , files blocked: 4452, messages infected: 0.

Zero, nil, nothing at all, the virus scanner either was missing all viruses, Trojans and other nefarious applications it had patterns for – or not – or it was nose picking due to these malicious things never reaching the server anyway.

When talking to a member of the Network team it became clear that there was a Barracuda security appliance in the network to scan mails for SPAM and viruses, a quick report on the virus totals confirmed what we believed to be true. Around 47 thousand viruses were reported to be detected and dropped.

While i agree that having multiple layers of security is a good thing, the reports were very clear on this one. The added value of the groupware solution was only one, namely the blocking of unwanted file types. This however does provide the Sysadmins with the daily task of recovering these files to their intended recipients causing a considerable workload. 

Suddenly this application was a very expensive content blocker that it never should be from the beginning, this specific product only blocks files based on extension and does not scan the contents as vigorously as the more advanced solutions would.

I am not saying that Anti Virus is dead like some are doing at the moment, it has it’s uses and does migitate threats to your network, one should however always look at how one would position the product. In this case the inbound and outbound mail always routes through the Barracuda appliance. Outbound mail also are scanned on the desktop when users are creating their mails, attachments are scanned on the disk (network and local) and in Outlook and again on their way out on the edge of the network. The extra Scanner in the middle never received any virus or i should say caught any virus due to the fact that all surrounding Anti Virus solutions were detecting the exact same malicious applications as it was.

Licencing and maintenance costs of this product can be considerably lowered by replacing it with a product that will do a better job of its – currently only – function; blocking attachments.

Recently i had to find a way to create a backup of a special printer, this printer was installed by sysadmins who no longer were employed by the customer and was running a special driver that could parse data from one of their business applications to create a .doc file.

This ‘special’ printer was installed on a virtualized workstation (XP), and the total lack of documentation first had us on a wild goose chase through the network to even find the darn thing.  And after finding it no one really was able to understand how it was installed. Let alone reconfigure it if it ever went down. With enough time one could reverse engineer the application hooks into the printer driver and derive how one could recreate the printer. Time was short and so i decided we needed to backup the printer settings to eventually enable us to recreate the printer (on the same or another machine).

The easiest way to go about it is to either snapshot or pull an image of the disk of the virtualized PC, but this would not enable us to migrate the printer to another machine if  that was needed. After having bad luck with the printer management scripts (that are default in XP). I found a little gem of an application called the “Microsoft Printer Migration Wizard 3.1”. This util allows you to backup the whole printer, its settings and even copies the driver to the .cab file you specify.

Starting it at the command line will pop up a new console window, the following options are available to you for possible automation of backing up printers.

Usage: PRINTMIG [options] [server]
       [-?] Display this message
       [-b] Backup – followed by CAB file name
       [-r] Restore – followed by CAB file name
       [-l] Attempt to convert LPR Ports to SPM
       [-i] Suppress warning popups.  Info still written to log file.

 If server is not specified then the local machine is implied.

 Example command line to restore an existing config to a server:
  printmig -r d:printps1.cab

 Example command line to backup a server to the specified file:
  printmig -b “
\filesrvstoreprint server 2.cab” \prt-srvr2

The process is pretty straightforward from here on, backup the settings to a .cab or restore settings from a .cab file. A few recommendations when using this utility are to be considered however.

For one Microsoft does not support the backup of printers via this migration wizard. Or at least not on the new version for Windows 2008 as it states so in that article.

Secondly this specific utility is for Windows 2003, XP and prior OSses. Windows 2008 and Vista have a new way of dealing with this specific issue.

And lastly, always when saving settings, type the full file name including the .cab part, otherwise the creation of the file may fail.

Link to the Printmig.exe utility: http://www.microsoft.com/WindowsServer2003/techinfo/overview/printmigrator3.1.mspx

Vista and Windows 2008 Printer backup via the command line:

This utlity requires and elevated command prompt.

To perform a (remote) print backup, type the following command in the “%WINDIR%System32SpoolTools” folder at the command prompt:  Printbrm -s \<sourceserverrname> -b -f <filename>.printerExport

To restore the (printers), type the following command in the %WINDIR%System32SpoolTools folder at the command prompt:  Printbrm -s \<destinationservername> -r -f <filename>.printerExport

On windows 2008 it may be required to install the role “Print Services” to be able to access this utility.

Recently i have been working with Server 2008 Core quite a bit, and have been revisiting the good old command line scripting and utilities that most people tend to forget these day’s.

I was thinking i would document some of these commands and utilities on the blog here, who knows there could be a small series in it.

I will start off with a very usefull util, that might not have been getting as mutch attention it should have been getting in my opinion. pnputil.exe

To add a driver to the driver store use:  pnputil.exe -a a:driverdriver.inf

To add all drivers in a folder use: pnputil.exe -a c:drivers*.inf

To add and then install a driver use:  pnputil.exe -i -a a:driverdriver.inf

To get a list of available third party drivers use: pnputil.exe -e

To delete a specific driver use: pnputil.exe -d driver.inf

To force the deletion of a specific driver use: pnputil.exe -f -d driver.inf

Remember that when you are adding or installing  a driver you first have to copy the files to the server. This utility is quite usefull when dealing with either a bulk driver install or replacement, or when installing third party hardware with Server 2008 Core. Another thing to remember when installing a driver is that you have to know what inf file you have to use as the util will not search for drivers like the Device manager GUI. And adding to many or the wrong driver to the driverstore or even installing them is something i would want to avoid.