Oct 102021
 
VMware vSphere 7 Logo

In this post, I wanted to go over some Backup and Restore tips and tricks when it comes to VMware vCSA Updates and Upgrades.

We’ve almost all been there, performing an update or upgrade of the VMware vCenter Server Appliance when it fails, and we must restore from a backup. There’s also times where the update or upgrade has been successful, however numerous issues occur afterwards prompting for the requirement of a restore from backup.

In this post, I wanted to briefly go over the methods of backups (and restores) for the vCSA, as well as some Tips and Tricks which might help you out for avoiding failed updates or upgrades in the future!

We all want to avoid a failed update or upgrade! 🙂

vCSA Update Installation
vCSA Update Installation

VMware vCSA Update Tips and Tricks for Backup and Restore

Please enjoy this video version of the blog post:

vCSA Update and Upgrade – Tips and Tricks for Backup and Restore

vCSA Backup methods

There are essentially two backup methods for backing up the vCenter Server Appliance:

  1. vCSA Management Interface Backup
  2. vSphere/ESXi Virtual Machine Snapshot

vCSA Management Interface Backup

If you log in to the vCSA Management Interface, you can configure a scheduled backup that will perform a full backup of your vCSA (and vCenter Server) instance.

This backup can be automatically ran and saved to an HTTP, HTTPS, FTP, FTPS, SFTP, NFS, or SMB destination. It’s a no-brainer if you have a Windows File Server or an NFS datastore.

vCSA Backup Screenshot
vCSA Backup

In the event of a failed update/upgrade or a disaster, this backup can be restored to a new vCSA instance to recover from the failure.

For more information on backups from the vCSA Management Interface, please see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-8C9D5260-291C-44EB-A79C-BFFF506F2216.html.

For information on restoring a vCSA file based backup, please see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-F02AF073-7CFD-45B2-ACC8-DE3B6ED28022.html.

vSphere/ESXi Virtual Machine Snapshot

In addition to the scheduled automatic backups configured above, you should snapshot your vCSA appliance VM prior to initiating an update or upgrade. In the event of a failure, you can easily restore the vCSA VM snapshot to get back to a running state.

vCSA Snapshot Screenshot
vCSA Snapshot

Only after you test and confirm the upgrade or update was successful should you delete the snapshot.

You should also have your Backup application or suite performing regularly snapshot based backups of your vCSA.

Additional Tips and Tricks

I have a few very important tips and tricks to share which may help you either avoid a failed update or upgrade, or increase the chances of a successful restore from backup.

  1. Gracefully Shutdown and Restart the vCSA Appliance before Upgrading
  2. Application Consistent Snapshot – Snapshot after graceful shutdown

Let’s dive in to these below.

Gracefully Shutdown and Restart the vCSA Appliance before Upgrading

I noticed that I significantly reduced the amount of failed upgrades by simply gracefully shutting down and restarting the vCenter Server Appliance prior to an upgrade.

This allows you to clear out the memory, virtual memory, and restart all vCenter services prior to starting the upgrade.

Please Note: Make sure that you give the vCSA appliance enough time to boot, start services, and let some of the maintenance tasks run before initiating an upgrade.

Application Consistent Snapshot – Snapshot after graceful shutdown

Most VMware System Administrators I have talked to, usually snapshot the running vCSA appliance and do not snapshot the memory. This creates a crash consistent snapshot.

If you follow my advice above and gracefully shutdown and restart the vCSA appliance, you can use this time to perform a VM snapshot after a graceful shutdown. This will provide you with an application consistent snapshot instead of a crash consistent snapshot.

If you perform an application consistent snapshot by gracefully shutting down the VM prior to creating the snapshot, the virtual machine and database inside of it will be in a cleaner state.

Conclusion

Some of the Tips and Tricks in this post definitely aren’t necessary, however they can help you increase the chance of a successful upgrade, and a successful restore in the event of a failed upgrade.

For more information on upgrading the vCenter Server Appliance, please visit https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-30485437-B107-42EC-A0A8-A03334CFC825.html.

Oct 092021
 

One of the new real cool features of the Synology C2 Cloud and Synology DSM 7.0 is the ability to back your Synology DiskStation NAS up to the cloud.

Using the Synology’s “C2 Storage” service, you can use Synology Hyper Backup on DSM 7.0 to backup your your File Shares, NAS, applications, and data to the cloud.

Below, I will show the process of creating a Hyper Backup Backup task to backup a few Synology File Shares to the Synology C2 Storage Service.

NAS Backup (and Restore) with Synology C2 Cloud Video

For a full video guide and demonstration, please enjoy this video! Please continue scrolling for a text guide (with images).

Synology DiskStation NAS Backup (and Restore) with Synology C2 Storage Cloud!

Backup NAS to Synology C2 Storage

It’s very easy to back your NAS up to the cloud. Here’s a quick guide showing how it’s done.

We’re going to assume you already have some File Shares configured on your NAS. These can be Windows File Shares, NFS Exports, and even Active Backup for Business storage repositories. In my case, I want to back up 2 Windows File Shares to the Synology C2 Storage service.

  1. Open Hyper Backup
  2. If you have no existing backup jobs, the “Backup Wizard” should automatically open to configure a new job. If you have existing jobs, select the “+” on the top left of the window.
  3. Under “Backup Destination”, choose “Synology C2 Storage”.
    Hyper Backup Destination
  4. This will open a new window asking you to log on to the Synology C2 Storage Service. You may have to create a new account, or log in with your existing Synology credentials. From here you can start a free trial.
    Synology C2 Storage - Get Free Trial Now
  5. Choose a subscription and plan. You should get a 30-day free trial.
    Synology C2 Storage - Select your C2 subscription
  6. Confirm your subscription
    Synology C2 Storage - Confirm Subscription
  7. Accept the TOS (Terms of Service)
    Synology C2 Storage - Terms of Service
  8. You must now “Grant access to C2 Storage”. This will allow your user account to access the C2 Storage service.
    Synology C2 Storage - Grant Access
  9. You will now be presented with information on your Synology C2 Storage Account
    Synology C2 Storage Account Information
  10. You will now be redirected back to Hyper Backup. You’ll need to create a name for the new backup task destination directory.
    Hyper Backup Destination Settings
  11. Here’s where you will configure which File Shares (or volumes) to backup to Synology C2.
    Hyper Backup Choose Data Backup Sources
  12. You also have the ability to backup Synology Applications. This is helpful should you wish to backup your entire Synology NAS to Synology’s C2 cloud. I’m only doing shares, so I hit next without selecting anything.
    Hyper Backup Application Backup
  13. Now we will configure the Backup Task settings. This includes the task name, notifications, file change logs, bandwidth limits, backup schedule, and integrity check schedule. We can also “Enable client-side encryption” which we will use to encrypt the data on the cloud.
    Hyper Backup Backup Settings
  14. After proceeding, we will now configure backup rotation settings. I enabled backup rotation with “Smart Recycle”.
    Hyper Backup Rotation Settings
  15. If you enabled encryption, you will be prompted to download your encryption key. Save this file in a safe location as you will need it in the event your NAS fails and you need to access your cloud storage.
    Hyper Backup Synology C2 Encryption PEM key file

And that’s it! You have now configured your NAS to backup to Synology C2 Cloud!

Hyper Backup NAS Backup to Synology C2 Cloud
Hyper Backup NAS Backup to Synology C2 Cloud

We have now configured the backup job. Let’s go ahead and kick off a backup by clicking on “Back up now”.

And that’s it! After some time, your backup should complete and you data should be safe on the Synology C2 Cloud.

What happens when your run of out space

This is a question many of you will be asking, and I was curious myself. I went ahead and uploaded a bunch of garbage data to max out my plan.

Synology Hyper Backup C2 Storage - Not enough quota available on the target
Synology Hyper Backup C2 Storage – Not enough quota available on the target

To my surprise Synology allowed me to exceed my plan by a decent amount of storage until the backup job went in to a “Suspended” state reporting “Not enough quota available on the target”. I actually exceeded the storage by 128GB!!! This is extremely generous!

Synology C2 Storage - Storage overage
Synology C2 Storage – Storage overage

To find out how well the NAS would recover from this situation, I logged on to the Synology C2 Storage service and increased my purchased storage. I was able to simply click on “Action” and “Resume” the backup without any issues… It actually resumed where it left off.

Conclusion

The Synology C2 Storage Cloud service is a great option for backing up your DiskStation NAS and all your important files to the cloud. In my case I use my NAS as a backup, and then further backup my NAS both to a removable hard drive, and the Synology C2 Cloud, keeping me compliant with the 3-2-1 backup rule.

The pricing is amazing and there’s extra cost to backup and restore, which means no upload or download costs. They even provide de-duplication so you’re not paying to store duplicated data.

As part of the backup process, Hyper Backup only copies over blocks of data and files that have changed, which significantly helps bandwidth usage as only what’s changed is copied.

I highly recommend using Synology C2 Storage with your Synology DiskStation NAS!

Oct 092021
 
Windows 11 Logo

When attempting to do a fresh install of Windows 11 using the ISO, you may receive the message “This PC can’t run Windows 11”. Additionally, “This PC doesn’t meet the minimum system requirements to install this version of Windows.”

Windows 11 has a new set of minimum system requirements and these include certain CPUs as well as a TPM 2.0 (Trusted Platform Module Version 2.0) chip, Secure Boot, and 8GB of RAM.

If you’re trying to do an upgrade instead of a fresh install, please see Windows 11 Upgrade – This PC doesn’t currently meet Windows 11 system requirements.

Below you’ll find an explanation of the problem, and two different methods to workaround it.

The Problem

You’ll see this message while performing a fresh install if your system does not meet the minimum requirements.

Windows 11 Fresh Install - This PC can't run Windows 11
Windows 11 Fresh Install – This PC can’t run Windows 11

Just like my previous post on upgrading to Windows 11, you’ll encounter this when attempting a fresh install because some pre-requisite checks are failing:

  • CPU is not supported
  • Windows 11 Installer cannot find a TPM 2.0 chip
  • Secure Boot is not enabled
  • EFI or UEFI is Required

One thing to note is that you may see these messages even if your system has a TPM 2.0 chip.

Most computers purchased in the last 6 years probably have a TPM 2 chip that just needs to be enabled via the system UEFI/EFI. If you boot to your UEFI, you can attempt to enable the TPM 2.0 chip.

It may already be enabled, however it may be configured to run at version 1.2. If this is the case, change it to version 2.0. You’ll also need to make sure you have “Secure boot” enabled.

If this doesn’t work, please see below for multiple workarounds.

The Fix

At this point in time, there are two different methods to workaround the minimum system requirements:

  1. Method 1 – Use Rufus to create a modified Windows 11 Installer from ISO and disable the TPM 2.0, Secure Boot, and 8GB of RAM requirement.
  2. Method 2 – Use native Windows 11 installer and ISO to modify registry during Windows Setup.

You can either either method, depending on which one you may find easier or more convenient.

Method 1 – Use Rufus to create a modified Windows 11 Installer from ISO and disable the TPM 2.0, Secure Boot, and 8GB of RAM requirement.

You can use a utility called “Rufus” (Reliable USB Formatting Utility, with Source) to convert the Windows 11 ISO in to a bottable USB key to install Windows.

Using the latest version of Rufus, you can modify the Windows 11 Setup installer to bypass the requirements for TPM 2.0, Secure Boot, and 8GB of RAM.

To use this method, you’ll need the following files:

Please enjoy this video demonstrating the process:

Windows 11 Fresh Install – TPM and Secure Boot Bypass for “This PC can’t run Windows 11”

To use this method as a workaround:

  1. Download Rufus and place in a folder
  2. Download Windows 11 ISO and place in a folder
  3. Insert USB key that is larger than the size of the Windows 11 ISO (larger than 5.5GB)
  4. Open Rufus
  5. Select your USB key under “Device”
  6. Under “Boot Selection”, click on “SELECT”
  7. Navigate to and select the Windows 11 ISO file
  8. Under “Image option”, choose “Extended Windows 11 Installation (no TPM/no Secure Boot/8GB- RAM”
  9. Click “Start”.
    PLEASE NOTE: This will erase and repartition your USB drive. All existing data on the USB drive will be deleted.
Rufus – Windows 11 Fresh Install TPM, Secure Boot, and RAM bypass

Now simply wait for the USB key to be created. It can take 30-90 minutes depending on the speed of your USB drive.

Once you have created the USB key, make sure your computer is configured to use UEFI and make sure you disable Secure Boot in the UEFI.

Simply boot from the USB Key your created above, and install Windows 11.

Method 2 – Use native Windows 11 installer and ISO to modify registry during Windows Setup.

Bypass the check for TPM 2.0

If you don’t have TPM 2.0 or it’s not working, you can disable the TPM 2.0 check on the Windows 11 installer. Please note, you still require TPM 1.2 for this bypass to function. This workaround only disables the requiremnt for TPM 2.0. You still need to have Secure Boot enabled, and you must have a TPM 1.2 chip.

To do this, boot from the Windows 10 ISO:

Windows 11 Installer
Windows 11 – Installer

When you see the above screen, press “SHIFT + F10” and a Windows Command Prompt should open.

From the command prompt, type “regedit” and press enter.

Windows 11 Installer with command prompt and Registry Editor "regedit"
Windows 11 Installer – Registry Editor “regedit”

Now we must create a registry key called “MoSetup” and a DWORD Value to disable the TPM and CPU check.

  1. Navigate to “HKEY_LOCAL_MACHINE\SYSTEM\Setup”
  2. Right click on “Setup”, select “New”, and choose “Key”, name it “MoSetup”
  3. Navigate to “MoSetup”
  4. On the right pane, right-click an empty space, select “New”, and select “DWORD (32-bit) Value”
  5. Name it: “AllowUpgradesWithUnsupportedTPMOrCPU” (without quotations)
  6. Set it to “1” (without quotations)

After performing the above, it should look like this.

Windows 11 Installer - MoSetup and AllowUpgradesWithUnsupportedTPMOrCPU
Windows 11 Installer – MoSetup and AllowUpgradesWithUnsupportedTPMOrCPU

Now simply close the Registry Editor, type “exit” to close out of the command prompt and continue with the Windows 11 Installer.

After performing the above, you should now be able to successfully perform a fresh install of Windows 11 with the TPM and CPU check disabled.

Oct 072021
 
Windows 11 Logo

When attempting to upgrade to Windows 11, you may receive the message “This PC doesn’t currently meet Windows 11 system requirements”.

Windows 11 has a new set of minimum system requirements and these include certain CPUs as well as a TPM 2.0 (Trusted Platform Module Version 2.0) chip.

I ran in to this issue on a Lenovo X1 Carbon as well as an HP Z240 Workstation. The Lenovo X1 Carbon does have a TPM 2 chip, however still would not install.

If you’re trying to a fresh installation instead of an upgrade, please see Windows 11 Fresh Install – This PC can’t run Windows 11 for instructions on performing a Fresh install with TPM and Secure Boot bypass.

The Problem

You’ll see this message if your system doesn’t meet the minimum requirements.

Windows 11 installer failing with "Windows 11 - This PC doesn't currently meet Windows 11 system requirements"
Windows 11 – This PC doesn’t currently meet Windows 11 system requirements

On most systems, you’ll see the following 2 prequisite checks fail:

  • “The processor isn’t supported for this version of Windows”
  • “The PC must support TPM 2.0.”

One thing to note is that you may see these messages even if your system has a TPM 2.0 chip.

You’ll also need to make sure your system has UEFI/EFI and has Secure Boot enabled.

The Fix

You have TPM 2.0 but can’t upgrade to Windows 11

Try to check and see if you have a TPM 2.0 chip. Most systems purchased in the last 6 years probably have a TPM 2 chip that just needs to be enabled via the system BIOS or UEFI.

If you boot to your BIOS/UEFI, you can attempt to enable the TPM 2.0 chip.

You may also already have it enabled, however it is configured to run at version 1.2. If this is the case, change it to version 2.0.

You’ll also need to make sure you have “Secure boot” enabled.

Bypass the check for TPM 2.0

If you don’t have TPM 2.0, you can disable the TPM 2.0 check on the Windows 11 installer. Please note, you still require TPM 1.2 for this bypass to function.

To do this, we must make a registry key.

  1. Start -> Run -> “regedit.exe” (without quotations)
  2. Navigate to “HKEY_LOCAL_MACHINE\SYSTEM\Setup\MoSetup”
  3. On the right pane, right-click an empty space, select “New”, and select “DWORD (32-bit) Value”
  4. Name it: “AllowUpgradesWithUnsupportedTPMOrCPU” (without quotations)
  5. Set it to “1” (without quotations)

After creating this, it should appear like so:

REG_DWORD: AllowUpgradesWithUnsupportedTPMOrCPU set to “1”

After setting this you should now be able re-launch the Windows 11 installer, and successfully install Windows 11. You’ll now notice the new message below:

Windows 11 – Bypass TPM and CPU Disclaimer

Simply “Accept” the warning and continue!

Please Note: Microsoft has warned that by using this TPM 2.0 bypass, you may run in to compatibility issues: “Your device might malfunction due to these compatibility or other issues. Devices that do not meet these system requirements will no longer be guaranteed to receive updates, including but not limited to security updates.”

You’ll see this disclaimer and warning on the Windows 11 installer after enabling the TPM 2.0 check bypass.

Additional Resources

Sep 302021
 
ISRG and Let's Encrypt

Today, the DST Root CA X3 certificate expired, leaving many devices on the internet having issues connecting to services and certificates that use this Root CA, including those using Let’s Encrypt certificates.

Some of these problematic devices include Samsung Galaxy phones, iPhones, VDI zero and thin clients, and even Sophos UTM firewalls.

In my environment, I noticed a number of issues when browsing to websites that use the free Let’s Encrypt certificates, as the Web Protection Web Filtering service on my Sophos UTM firewall would report the certificate has expired and not allow me access to the websites using it.

The Problem

Let’s Encrypt originally used the “DST Root CA X3” certificate to issue Let’s Encrypt certificates. However, as time has passed and the service has been used more, they now use “ISRG Root X1” and “ISRG Root X2” as Root CA’s and “Let’s Encrypt R3” as an intermediate certificate.

Older devices may be using the older Root CA which expired today (September 30th, 2021). Please see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ for more information.

The Fix

To fix this issue, you need to add the 2 new Root CAs to your computer or device.

Root CA Certificates (PEM format):

Intermediate Certificate (PEM format):

You can download them by clicking the links above or go to https://letsencrypt.org/certificates/ for more information and to download if you don’t trust the above links.

After downloading and adding these Root CAs and the Intermediate CA to your computer or device, you should have the full certificate chain to validate the Let’s Encrypt certificates. You only need to add the two root certificates. The Let’s Encrypt certificates that are used on websites that you visit and that you might have deployed on your servers should now work without any issues.

If you’re still having issues, you can try deleting the “DST Root CA X3” certificate from your existing Root CAs. Also, you may need to close and reopen any software and/or browsers for it to work with the new certificate.

HTTPS Scanning/Filtering Firewall Fix (Sophos UTM as example)

If you have a firewall that scans HTTPs traffic, you’ll need to add the two root certificates above to the HTTPS Certification authority list.

As an example, to fix this on the Sophos UTM firewall, follow the instructions below:

  1. Download the 3 certificates above.
  2. Log on to your Sophos UTM
  3. Navigate to “Web Protection”, “Filtering Options”, and “HTTPS CAs” tab.
  4. Disable the old “Digital Signature Trust Co. DST Root CA X3” Certificate in the list.
  5. Using the “Upload local CA”, browse to and select 1 of the 3 certificates, then click upload.
  6. Repeat step 5 for each of the 3 certificates listed above.
  7. The issue has been fixed! You should now see all 3 certificates in the “Local verification CAs” list.

The steps should be similar for other firewalls that provide HTTPS Scanning and Filtering.