Mar 052016
 

Just wanted to write about a couple issues that I’ve seen occur after migrating customers from Microsoft Small Business Server to Microsoft Server 2012 R2 (with Essentials Experience role), with Microsoft Exchange 2013 On-Premise.

Migration documents that were available were used at the time of migration. We still observed these issues after following. Please note that since these issues occurred, migration documents may have been updated.

Just an FYI: I provide Small Business Server Migration and consulting services. For more information, click here!

Windows SBS Company Web Connector ServerName

After the migration was complete we started seeing event logs pertaining to a “Windows SBS Company Web Connector ComputerName”, often mentioning it’s referencing an object in the Deleted Items container, also referencing the connector is not being activated due to no routes available.

Event ID: 5016

Microsoft Exchange could not discover any route to connector CN=Windows SBS Company Web Connector SERVERNAME,CN=Connections,CN=Exchange Routing Group (XXXXXXXXXXXXXXXXX),CN=Routing Groups,CN=Exchange Administrative Group (XXXXXXXXXXXXXXXXX),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domainname,DC=local in the routing tables with the timestamp 3/5/2016 1:55:34 PM. This connector will not be used.  Total source server count: 1; unknown source server count: 1; unrouted source server count: 0; non-active source server count: 0.

What is happening is that a “Foreign Connector” is still present in the Active Directory and Exchange Configuration for the SBS environments SharePoint e-mail to web feature. In my client’s environments SharePoint is no longer used, so it is safe for us to delete this connector. Only delete this connector if you know you’re not using it (it is used for SharePoint e-mail to web feature).

To list and get information on the orphaned connector, open Exchange Powershell and run:

Get-ForeignConnector | Format-List

To delete the orphaned connector, enter the following command in Exchange Powershell and update the connector name to match the name shown in the command above:

Remove-ForeignConnector “Windows SBS Company Web Connector SERVERNAME”

This will remove the orphaned connector and clean up these errors from occurring. You can also remove the connector using ADSIEDIT, however I prefer to use ADSIEDIT as a last resort, and find this method not only easier, but cleaner.

SMTP rejected a (P1) mail from ‘[email protected]

Initially post-migration we started observing this event on the server. Mail flow was not affected and everything was functioning properly.

Event ID: 1025

SMTP rejected a (P1) mail from ‘[email protected]’ with ‘Client Proxy EXCHSRVR’ connector and the user authenticated as ‘HealthMailboxXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX’. The Active Directory lookup for the sender address returned validation errors. Microsoft.Exchange.Data.ProviderError

Additionally, on our corporate firewall (that provides anti-spam), we would observe numerous undeliverable bouncebacks on outgoing messages to the e-mail address “[email protected]” with the subject “Inbound proxy probe”. These messages occur on exact 5 minute intervals continuously.

Using Exchange powershell to view the active Health Mailboxes, we see that each of these bounce backs are being stored on a particular health mailbox. Essentially the mailbox will continue to grow. Due to the growth, this issue needs to be resolved so the mailbox doesn’t continue to grow in size.

Numerous things can cause this, however in our case looking at transport logs, it is seen that a HealthMailbox is sending e-mail to another HealthMailbox but using an incorrect e-mail address. The Health Mailboxes on the Exchange server have “domain.com” e-mail addresses, while according to the transport logs, the e-mails are being sent to “domain.local”.

Something got mixed up, either with provisioning the Exchange E-Mail address policies, or the domain configured as “default domain”. Either way, Exchange is configured and running, so I wanted to correct this in a manor that would have minimal consequences or changes to the system.

To correct this issue, we need to go in to ADSI edit and modify the “ProxyAddresses” value for the HealthMailbox. Note that any type of mailbox can have numerous aliases and a single default alias. Inside of ADSIEdit for “ProxyAddresses” the value/format is case-sensitive, and uppercase SMTP configures default e-mail address, while lowercase smtp configures alternative aliases. An example value: “SMTP:[email protected]” for default, or “smtp:[email protected]” for an alternative alias.

Identifying the account from the event log (note the XXXXXXXXXXXXXXXX in the example), we found the account in the Monitoring Mailboxes container inside of ADSIEdit. We right-clicked on the specific HealthMailbox account, went to properties, and found the “ProxyAddresses” value. We then proceeded to create a new alias by clicking edit, using lowercase smtp and created “smtp:[email protected]” and added it to the list, we did not modify or delete any existing values. All we did is create an alternative alias.

So now the Health Mailbox is receiving e-mail for both “@domain.com”, and “@domain.local”. Immediately the bounce-backs stopped, and event logs disappeared.

PLEASE NOTE: For this fix to work, you MUST confirm that the issue is due to the domain .com and .local mismatch. I’m not quite sure, but this issue may also occur after changing the default domain, or default e-mail address policies, in which case you still could use this technique to resolve the issue.

Hope this helps some of you, cheers!

Feb 082016
 

For some time now, I’ve been having issues using HP Intelligent Provisioning to update the firmware of my HP Proliant DL360p Gen8 servers. Typically, when configured and when running the firmware update option, it times out saying it cannot connect to HP’s servers.

With the recent separation of HP and HPE (HP Enterprise), I had a feeling this had to do with separation of their FTP server storage which houses all the updates.

Other behaviors when experiencing the issue, messages shown:

Retrieving data, this may take a few minutes…
Looking for updates, this may take a few minutes…
This process is taking longer than expected – Wait or Quit

Keep in mind that version 2.x of HP’s Intelligent Provision package is only for Gen9 (Generation 9) servers. For Gen8 (Generation 8) servers, you need the latest version of 1.x which at this time is 1.62(b) dated August 5th, 2015.

Download this version for your Gen8 servers from HPE’s Support website. After downloading the HPIP162b.2015_0730.43.iso file, burn it to a DVD, or mount it to a virtual media on your iLo connection, and update the software on the server during boot.

After doing this, you will be able to connect and check for firmware. I’m assuming you probably need the updated 2.x image if you’re running a Gen9 server.

HP Intelligent Provisioning Website

HP Intelligent Provisioning Recovery Media (Update Tool) Version 1.62 (b)

Cheers!

Nov 212015
 
HP MSA2040 Dual Controller SAN with 10Gb DAC SFP+ cables

I’d say 50% of all e-mails/comments I receive from the blog in the last 12 months or so, have been from viewers requesting pictures or proof of the HPE MSA 2040 Dual Controller SAN being connection to servers via 10Gb DAC Cables. This should also apply to the newer generation HPE MSA 2050 Dual Controller SAN.

Decided to finally publicly post the pics! Let me know if you have any questions. In the pictures you’ll see the SAN connected to 2 X HPE Proliant DL360p Gen8 servers via 4 X HPE 10Gb DAC (Direct Attach Cable) Cables.

Connection of SAN from Servers

Connection of SAN from Servers

Connection of DAC Cables from SAN to Servers

Connection of DAC Cables from SAN to Servers

See below for a video with host connectivity:

Nov 172015
 

Decided to whip up a post about an issue that I have been running in to more and more as of late.

Typically, situation goes as follows: Customer has an environment where there are industrial machines running Windows CE Embedded computers as controllers. These systems typically are configured to either host files, or grab files off a network. These systems are typically dated, and IT staff is unable to get the Windows CE based machines to connect to network shares on Windows Servers running SMB version 2 or later (ie. Windows Server 2008 and later).

 

This issue is due to authentication issues with protocols and incompatibles. Over the years, Windows File Sharing has come a long way (SMB to be precise). Numerous security enhancements have been made, authentication mechanisms, etc…

In all cases, I’ve noticed companies usually either give up, or hire someone who is able to resolve it, but the resolution is never documented.

 

The solution I have come to could be considered somewhat controversial (due to the fact that Windows XP has reached it’s EOF), but I’ve found a way.

To provide file sharing solutions, in my experiences I have been able to accomplish this by implementing a Windows XP based “proxy” machine (calling it a proxy by name, not by actual usage). Configuring a Windows XP machine, enabling the “guest” account on it, and configuring file shares, will allow users on the network to dump files on these “proxy” network shares, in turn which will be browsable and accessible to the Windows CE machine. This Windows XP machine can be joined to the domain, to allow seamless authentication with other network users/computers, and also contains it’s own local user database.

The guest account needs to be enabled as the Windows CE machines typically browse and do initial file sharing handshakes as “guest”. You’ll also need a local user account configured on the Windows XP machine, which is the account that the actual Windows CE machine will use to connect/authenticate against the share and it’s access.

Please note, you may also have to go in to the “Local Security” policy, and allow guest access to file shares and browsing on the Windows XP machine.

 

As always, since Windows XP has reached it’s end of life, no more security updates are available. You want to make sure you have other security measures in place to mitigate any security concerns that could arise from having an active XP OS running on the network. If anyone else has a better solution or can comment further on this, please do! I’ve had to deal with this issue multiple times for CNC machines with older CE based controllers, as well as handheld Windows CE devices that require network share access.

Nov 172015
 

I recently had a reader reach out to me for some assistance with an issue they were having with a VMWare implementation. They were experiencing issues with uploading files, and performing I/O on Linux based virtual machines.

Originally it was believed that this was due to networking issues, since the performance issues were only one way (when uploading/writing to storage), and weren’t experienced with all virtual machines. Another particular behaviour notice was slow uploading speeds to the vSphere client file browser, and slow Physical to Virtual migrations.

After troubleshooting and exploring the issue with them, it was noticed that cache was not enabled on the RAID array that was providing the storage for the vSphere implementation.

Please note, that in virtual environments with storage based off RAID arrays, RAID cache is a must (for performance reasons). Further, Battery backed RAID cache is a must (for protection and data integrity). This allows write operations to be cached and performed on multiple disks at once, sometimes even optimizing the write procedures as they are processed. This allows writes to occur simultaneously to multiple disks, and also dramatically increases observed performance since the ESXi hosts, and virtual machines aren’t waiting for write operations to commit before proceeding to the next.

You’ll notice that under Windows virtual machines, this issue won’t be observed on writes since the Windows VMs typically cache file transfers to RAM, which then write to disk. This could give the impression that there are no storage issues when typically troubleshooting these issues (making one believe that it’s related to the Linux VMs, the ESXi hosts themselves, or some odd networking issue).

 

Again, I cannot stress enough that you should have a battery backed cache module, or capacitor backed flash module providing cache functions.

If you do implement cache without backing it with a battery, corruption can occur on the RAID array if there is a power failure, or if the RAID controller freezes. The battery backed cache allows cached write procedures to be committed to disk on next restart of the storage unit/storage controller thus providing protection.