Backup Exec 2014 – Jobs failing on HP Proliant DL360 G6 with an HP MSL2024 LTO-6 Tape Library

 Backup, HP, Storage, Windows Server  14 Responses »
Mar 262016
 

An issue that’s been making me rip my hair apart for some time… And a fix for you experiencing the same.

 

Equipment:

HP Proliant DL360 G6 Server (with a P800 Controller) running Server 2012 R2 and Backup Exec 2014

HP MSL-2024 Tape Library with a single HP SAS LTO-6 Tape Drive

 

Symptoms:

-After a clean restart, a backup job completes successfully. Subsequent jobs fail until server or services restarted.

-While the initial backup does complete, errors/warnings can be seen in the adamm.log and the Event Viewer even when successful.

-Subsequent backups failing report that the device is offline. The Windows Device Manager reports everything is fine.

-Windows Server itself does not report any device errors whatsoever.

 

Observations:

[5648] 03/05/16 07:50:46 Adamm Mover Error: DeviceIo: 03:07:00:00 – Device error 1167 on “\\.\Tape0”, SCSI cmd 0a, 1 total errors
[5648] 03/05/16 07:55:46 Adamm Mover Error: DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 00, new handle 214, error 0
[5648] 03/05/16 07:55:46 Adamm Mover Error: DeviceIo: 00:00:00:00 – Retry Logic: Retry logic was engaged on device: HP       Ultrium 6-SCSI
[5648] 03/05/16 07:55:46 Adamm Mover Error: DeviceIo: 00:00:00:00 – Retry Logic: Original settings restored on device: HP       Ultrium 6-SCSI

Event ID 58053
Backup Exec Alert: Storage Error
(Server: “WhatsMySRVRname”) The device state has been set to offline because the device attached to the computer is not responding.

Ensure that the drive hardware is turned on and is properly cabled. After you correct the problem, right-click the device, and then click Offline to clear the check mark and bring the device online.

[09968] 03/05/16 01:42:08.426 DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 17, new handle ffffffff, error 32
[09968] 03/05/16 01:42:08.426 DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 1a, new handle ffffffff, error 32
[09968] 03/05/16 01:22:07.867 PvlSession::DismountMedia( 0, 0, 0 )
Job = {JOBHEXNUMBERZZZZZZ} “ServerBackup-Full”
Changer    = {CHANGERZZZZ} “Robotic library 0001”
Drive      = {MYBACKUPDRVXZZZZZ} “Tape drive 0001”
Slot       = 13
Media      = {MEDIAZIDZZZZ} “BARCODEID”
ERROR = 0xE0008114 (E_PVL_CHANGER_NOT_AVAILABLE)

[19812] 03/05/16 01:42:12.613 DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 1a, new handle ffffffff, error 32
[19812] 03/05/16 01:42:13.129 DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 00, new handle ffffffff, error 32
[19812] 03/05/16 01:42:13.645 PvlDrive::DisableAccess() – ReserveDevice failed, offline device
Drive = 1007 “Tape drive 0001”
ERROR = 0x0000001F (ERROR_GEN_FAILURE)

[19812] 03/05/16 01:42:13.645 PvlDrive::UpdateOnlineState()
Drive = 1007 “Tape drive 0001”
ERROR = The device is offline!

[19812] 03/05/16 01:42:12.613 DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 1a, new handle ffffffff, error 32
[19812] 03/05/16 01:42:13.129 DeviceIo: 03:07:00:00 – Refresh handle on “\\.\Tape0”, SCSI cmd 00, new handle ffffffff, error 32
[19812] 03/05/16 01:42:13.645 PvlDrive::DisableAccess() – ReserveDevice failed, offline device
Drive = 1007 “Tape drive 0001”
ERROR = 0x0000001F (ERROR_GEN_FAILURE)

[19812] 03/05/16 01:42:13.645 PvlDrive::UpdateOnlineState()
Drive = 1007 “Tape drive 0001”
ERROR = The device is offline!

Event ID 1000
Faulting application name: wmiprvse.exe, version: 6.3.9600.17415, time stamp: 0x54505614
Faulting module name: MSVCR110.dll, version: 11.0.51106.1, time stamp: 0x5098826e

 

Research:

I spent a ton of time researching this… Old support threads were pointing me in all different directions, most of the threads being old, mentioning drivers, etc… Initially I thought it was hardware related, until through testing I got the gut feeling it was software related. There was absolutely no articles covering Backup Exec 2014 running on Windows Server 2012 R2 with this specific issue.

Tried a bunch of stuff, including swapping the P800 controller, for another HP P212. While it didn’t fix the issue, I gained some backup speed! 🙂

Updating the HP software (agents, providers, HP SMH, WBEM) had no effect.

Disabling the HP providers, and disabling the HP Monitoring, Insight, Management services had no effect whatsoever. Tried different firmware versions, also tried different drivers for the Library and Tape drive, no effect. Tried factory resets, no effect. Tried Library and Tape tools, all tests passed.

Disabled other monitoring software we have in place to monitor software/hardware on clients servers, no effect.

 

Resolution:

-Uninstalled the HP WBEM Providers and Agents.

-Added a “BusyRetryCount” 32-bit DWORD value of 250 (decimal) to the “Storport” key under “Device Parameters” in all the Tape Library and Tape Drive Registry entries. Example:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\Changer&Ven_HP&Prod_MSL_G3_Series\5&334e8424&0&000500\Device Parameters\Storport]
“BusyRetryCount”=dword:000000fa

This needs to be added to ONLY and ALL the tape device entries (under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\) for the Tape Library and Tape Drives. You probably will have to create “Storport” key under the devices “Device Parameters” key.

After doing this, the backups run consecutively with absolutely no issues. The event log is CLEAN, and Adamm.log is clean, and the “Faulting application name: wmiprvse.exe” errors in the event log no longer occur.

Fixed!

 

Additional Notes:

-Both “fixes” were applied at the same time. I believe the WBEM providers/agent caused the Event ID 1000 errors on WMIPRVSE.exe. While the registry keys alone may have possibly resolved the backup issues, I believe there still would have been an underlying issue with WMIPRVSE.exe faulting that could have other consequences.

-I do not believe the original installation of the HP WBEM providers caused the issue, I have a feeling a subsequent Windows Update, Backup Exec update, other module update, or an update to the HP software may have caused the issue to occur at a later time than original install. I do remember we didn’t have an issue with the backups for months, until one day it started occurring.

-I will be re-installing the HP providers and agents at a later time. I will be uninstalling all of them, and re-installing from scratch the latest versions. I will post an update with my results.

-There is a chance the registry key is needed for the HP software to co-exist with Backup Exec backups for this configuration.

-There is a chance that the registry key isn’t needed if you never load the HP software.

Microsoft Small Business Server Post-Migration Issues with Microsoft Exchange

 Exchange Server, Small Business Server, Windows Server  1 Response »
Mar 052016
 

Just wanted to write about a couple issues that I’ve seen occur after migrating customers from Microsoft Small Business Server to Microsoft Server 2012 R2 (with Essentials Experience role), with Microsoft Exchange 2013 On-Premise.

Migration documents that were available were used at the time of migration. We still observed these issues after following. Please note that since these issues occurred, migration documents may have been updated.

Just an FYI: I provide Small Business Server Migration and consulting services. For more information, click here!

Windows SBS Company Web Connector ServerName

After the migration was complete we started seeing event logs pertaining to a “Windows SBS Company Web Connector ComputerName”, often mentioning it’s referencing an object in the Deleted Items container, also referencing the connector is not being activated due to no routes available.

Event ID: 5016

Microsoft Exchange could not discover any route to connector CN=Windows SBS Company Web Connector SERVERNAME,CN=Connections,CN=Exchange Routing Group (XXXXXXXXXXXXXXXXX),CN=Routing Groups,CN=Exchange Administrative Group (XXXXXXXXXXXXXXXXX),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domainname,DC=local in the routing tables with the timestamp 3/5/2016 1:55:34 PM. This connector will not be used.  Total source server count: 1; unknown source server count: 1; unrouted source server count: 0; non-active source server count: 0.

What is happening is that a “Foreign Connector” is still present in the Active Directory and Exchange Configuration for the SBS environments SharePoint e-mail to web feature. In my client’s environments SharePoint is no longer used, so it is safe for us to delete this connector. Only delete this connector if you know you’re not using it (it is used for SharePoint e-mail to web feature).

To list and get information on the orphaned connector, open Exchange Powershell and run:

Get-ForeignConnector | Format-List

To delete the orphaned connector, enter the following command in Exchange Powershell and update the connector name to match the name shown in the command above:

Remove-ForeignConnector “Windows SBS Company Web Connector SERVERNAME”

This will remove the orphaned connector and clean up these errors from occurring. You can also remove the connector using ADSIEDIT, however I prefer to use ADSIEDIT as a last resort, and find this method not only easier, but cleaner.

SMTP rejected a (P1) mail from ‘[email protected]’

Initially post-migration we started observing this event on the server. Mail flow was not affected and everything was functioning properly.

Event ID: 1025

SMTP rejected a (P1) mail from ‘[email protected]’ with ‘Client Proxy EXCHSRVR’ connector and the user authenticated as ‘HealthMailboxXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX’. The Active Directory lookup for the sender address returned validation errors. Microsoft.Exchange.Data.ProviderError

Additionally, on our corporate firewall (that provides anti-spam), we would observe numerous undeliverable bouncebacks on outgoing messages to the e-mail address “[email protected]” with the subject “Inbound proxy probe”. These messages occur on exact 5 minute intervals continuously.

Using Exchange powershell to view the active Health Mailboxes, we see that each of these bounce backs are being stored on a particular health mailbox. Essentially the mailbox will continue to grow. Due to the growth, this issue needs to be resolved so the mailbox doesn’t continue to grow in size.

Numerous things can cause this, however in our case looking at transport logs, it is seen that a HealthMailbox is sending e-mail to another HealthMailbox but using an incorrect e-mail address. The Health Mailboxes on the Exchange server have “domain.com” e-mail addresses, while according to the transport logs, the e-mails are being sent to “domain.local”.

Something got mixed up, either with provisioning the Exchange E-Mail address policies, or the domain configured as “default domain”. Either way, Exchange is configured and running, so I wanted to correct this in a manor that would have minimal consequences or changes to the system.

To correct this issue, we need to go in to ADSI edit and modify the “ProxyAddresses” value for the HealthMailbox. Note that any type of mailbox can have numerous aliases and a single default alias. Inside of ADSIEdit for “ProxyAddresses” the value/format is case-sensitive, and uppercase SMTP configures default e-mail address, while lowercase smtp configures alternative aliases. An example value: “SMTP:[email protected]” for default, or “smtp:[email protected]” for an alternative alias.

Identifying the account from the event log (note the XXXXXXXXXXXXXXXX in the example), we found the account in the Monitoring Mailboxes container inside of ADSIEdit. We right-clicked on the specific HealthMailbox account, went to properties, and found the “ProxyAddresses” value. We then proceeded to create a new alias by clicking edit, using lowercase smtp and created “smtp:[email protected]” and added it to the list, we did not modify or delete any existing values. All we did is create an alternative alias.

So now the Health Mailbox is receiving e-mail for both “@domain.com”, and “@domain.local”. Immediately the bounce-backs stopped, and event logs disappeared.

PLEASE NOTE: For this fix to work, you MUST confirm that the issue is due to the domain .com and .local mismatch. I’m not quite sure, but this issue may also occur after changing the default domain, or default e-mail address policies, in which case you still could use this technique to resolve the issue.

Hope this helps some of you, cheers!

Industrial Windows CE embedded systems connecting to network shares

 Windows CE  5 Responses »
Nov 172015
 

Decided to whip up a post about an issue that I have been running in to more and more as of late.

Typically, situation goes as follows: Customer has an environment where there are industrial machines running Windows CE Embedded computers as controllers. These systems typically are configured to either host files, or grab files off a network. These systems are typically dated, and IT staff is unable to get the Windows CE based machines to connect to network shares on Windows Servers running SMB version 2 or later (ie. Windows Server 2008 and later).

 

This issue is due to authentication issues with protocols and incompatibles. Over the years, Windows File Sharing has come a long way (SMB to be precise). Numerous security enhancements have been made, authentication mechanisms, etc…

In all cases, I’ve noticed companies usually either give up, or hire someone who is able to resolve it, but the resolution is never documented.

 

The solution I have come to could be considered somewhat controversial (due to the fact that Windows XP has reached it’s EOF), but I’ve found a way.

To provide file sharing solutions, in my experiences I have been able to accomplish this by implementing a Windows XP based “proxy” machine (calling it a proxy by name, not by actual usage). Configuring a Windows XP machine, enabling the “guest” account on it, and configuring file shares, will allow users on the network to dump files on these “proxy” network shares, in turn which will be browsable and accessible to the Windows CE machine. This Windows XP machine can be joined to the domain, to allow seamless authentication with other network users/computers, and also contains it’s own local user database.

The guest account needs to be enabled as the Windows CE machines typically browse and do initial file sharing handshakes as “guest”. You’ll also need a local user account configured on the Windows XP machine, which is the account that the actual Windows CE machine will use to connect/authenticate against the share and it’s access.

Please note, you may also have to go in to the “Local Security” policy, and allow guest access to file shares and browsing on the Windows XP machine.

 

As always, since Windows XP has reached it’s end of life, no more security updates are available. You want to make sure you have other security measures in place to mitigate any security concerns that could arise from having an active XP OS running on the network. If anyone else has a better solution or can comment further on this, please do! I’ve had to deal with this issue multiple times for CNC machines with older CE based controllers, as well as handheld Windows CE devices that require network share access.

NVIDIA GeForce Drivers for Windows 10 – Dual Displays not working

 NVIDIA, Windows 10  3 Responses »
Nov 162015
 

After upgrading to Windows 10, I immediately noticed that my 3 display setup no longer worked. It was powered by two NVidia graphics cards (GeForce GT 640, and a GeForce GTX 550 Ti).

For some time, I couldn’t find anything on the internet explaining as to why I lost my dual display setup. Finally I came across a forum that pointed to this NVidia Support KB article: http://nvidia.custhelp.com/app/answers/detail/a_id/3707/~/windows-10-will-not-load-the-nvidia-display-driver-for-my-older-graphics-card

Essentially Fermi based GPUs utilize WDDM 1.3 mode, whereas the newer architectures of Maxwell and Kepler support WDDM 2.0. In Windows 10, it is not able to load multiple display drivers using different WDDM versions.

For a really long time I waited and no updates enabled the functionality until September when I performed an update, and out of nowhere they started to work. I assumed they fixed the issue permanently, however after updating once again, I lost the capabilities. In this case I reverted to the last driver.

I’m not sure if they updated the Fermi driver to support WDDM 2.0, but I just know it started working. And then after a short while, with another driver update stopped working again. Again, the driver rollback fixed the issue.

 

I recently upgraded to the latest build of Windows 10, and completely lost the ability once again, and lost the ability to rollback drivers.

It was time to find out exactly what driver version WORKS with both Kepler, Fermi, and Maxwell architectures.

After playing around, I found the WORKING NVidia driver version to be: 358.50

Load this version up, and you’ll be good to go! Hope it saves you some time!

Event ID: 513 – The system cannot find the file specified.

 Windows Server  6 Responses »
Sep 302014
 

Recently, a new type of error I haven’t seen showed up on one of the servers I maintain and manage.

 

Event ID: 513

Source: CAPI2

Event:

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:
AddLegacyDriverFiles: Unable to back up image of binary EraserUtilRebootDrv.

System Error:
The system cannot find the file specified.
.

 

Also, after further investigation I also noticed that when Windows Server Backup was running, sometimes snapshots on the C: volume wouldn’t “grow in time” so were automatically deleting.

It was difficult to find anything on the internet regarding this as in my case it was reporting “The system cannot find the file specified”, whereas all other cases were due to security permissions. On the bright side, I was able to identify the software that this file belonged to: Symantec Endpoint Protection.

Ultimately I found a fix. PLEASE ONLY attempt this, if you are receiving the “The system cannot find the file specified”. If you are seeing any “Access Denied” messages under System Error, your issue is related to something else.

 

To fix:

1) Uninstall Symantec Endpoint protection.

2) Restart Server

3) Disable VSS snapshots for C: volume (NOTE: This will delete all existing snapshots for the drive.).

4) Re-install Symantec Endpoint protection.

5) Re-enable VSS snapshots for C: volume.

 

When this issue occurred, I was seeing the event many times every hour. It’s been 4 days since I applied this fix and it has completely disappeared, back to a 100% clean event log!

 Older Entries  Newer Entries