Veeam WAN Accelerator Service Fails to Start

Nov 232024

In some scenarios, you may encounter an issue where the Veeam WAN Accelerator service fails to start.

This will cause backup and backup copy jobs to fail that use the Veeam WAN Accelerator, which is how this issue is usually first diagnosed.

In this post I’ll explain the problem, what can cause it, and how to resolve the issue.

The Problem

When this issue occurs, and when a Backup or Backup Copy job runs, it will usually first fail with the following error from the Veeam console:

Error: The RPC server is unavailable. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP.OF.WAN.ACC:6464].

Failed to process (VM Name).

See below for a screenshot of the example:

Veeam Backup Copy Job Failing due to Veeam WAN Accelerator Service failing

From the error above, the next step is usually to check the logs to find out what’s happening. Because this Backup Copy job uses the WAN accelerator, we’ll look at the log for the Veeam WAN Accelerator Service.

Svc.VeeamWANSvc.log

[23.11.2024 11:46:24.251] <  3440> srv      | RootFolder = V:\VeeamWAN
[23.11.2024 11:46:24.251] <  3440> srv      | SendFilesPath = V:\VeeamWAN\Send
[23.11.2024 11:46:24.251] <  3440> srv      | RecvFilesPath = V:\VeeamWAN\Recv
[23.11.2024 11:46:24.251] <  3440> srv      | EnablePerformanceMode = true
[23.11.2024 11:46:24.255] <  3440> srv      | ERR |Fatal error
[23.11.2024 11:46:24.255] <  3440> srv      | >>  |boost::filesystem::create_directories: The system cannot find the path specified: "V:\"
[23.11.2024 11:46:24.255] <  3440> srv      | >>  |Unable to apply settings. See log for details.
[23.11.2024 11:46:24.255] <  3440> srv      | >>  |An exception was thrown from thread [3440].
[23.11.2024 11:46:24.255] <  3440> srv      | Stopping service...
[23.11.2024 11:46:24.256] <  3440> srv      | Stopping retention thread...
[23.11.2024 11:46:24.257] <  4576>          | Thread started.  Role: 'Retention thread', thread id: 4576, parent id: 3440.
[23.11.2024 11:46:24.258] <  4576>          | Thread finished. Role: 'Retention thread'.
[23.11.2024 11:46:24.258] <  3440> srv      | Waiting for a client('XXX-Veeam-WAN:6165')
[23.11.2024 11:46:24.258] <  3440> srv      | Stopping server listening thread.
[23.11.2024 11:46:24.258] <  3440> srv      |   Stopping command handler threads.
[23.11.2024 11:46:24.258] <  3440> srv      |   Command handler threads have stopped.
[23.11.2024 11:46:24.258] <  4580>          | Thread started.  Role: 'Server thread', thread id: 4580, parent id: 3440.

In the Veeam WAN Service log file above, you’ll note a fatal error where the service is unable to find the paths configured, which caused the service to halt and stop.

In some configurations, iSCSI is used to access Veeam backup repository storage hosted on iSCSI targets. Furthermore, in some iSCSI configurations special vendor plugins are used to access the iSCSI storage, and configure items like MPIO (multipath input output), which can take additional time to initialize.

In this scenario, the Veeam WAN Accelerator Service was starting before the Windows iSCSI service, MPIO Service, and Nimble Windows Connection Manager plugin had time to initialize, resulting in the WAN accelerator failing because it couldn’t find the directories it was expecting.

The Solution

To resolve this issue, we want the Veeam WAN Accelerator Service to have a delayed start on the Windows Server operating system bootup sequence.

Open Windows Services
Select “Veeam WAN Accelerator Service”
Change “Startup Type” to “Automatic (Delayed Start)”
Click “Apply” to save, and then click “Start” to start the service.

As per the screenshot below:

Veeam WAN Accelerator Service Properties

The Veeam WAN Accelerator Service will now have a delayed start on system bootup, allowing the iSCSI initiator to establish and mount all iSCSI Target block devices, before starting the WAN service.

VMware Explore 2024

Sep 012024

I’m finally back at home after an amazing week at VMware Explore 2024, in Las Vegas!

As always, VMware Explore 2024 was the place to be, and THE most important conference of the year, at least for me, my clients, and those who need to be in the know.

See below for my insights on the conference, information and links, as well as some videos I created to talk about!

The Focus moving forward

This was the first VMware Explore with VMware by Broadcom. With the acquisition of VMware by Broadcom, there’s been a number of changes they have implemented, including their new strategy moving forward, SKU consolidation, as well as solution refinement.

The focus this year was on VCF, also known as VMware Cloud Foundation, with the announcement of VCF 9.0, and enabling technologies like AI (Artificial Intelligence) with VMware Private AI Foundation, and NVIDIA AI enterprise.

Why VMware Explore is important for me

For me, VMware Explore is important because of a number of different things. A brief of this is:

VMware By Broadcom
- Strategy
- Product Releases
- Technical Sessions (Learning, Training)
- Hands on Labs (HoL)
Community
- VMUG – VMware User Group
- VMware vExpert
Networking
- Reconnecting in person with clients in other cities (and other countries)
- Meeting customers face to face for the first time
- New Business

I can’t recommend enough how important VMware Explore is for career development. It’s a face paced conference, and you won’t have time to do everything you plan and want to, but you just need to go with the flow and let it happen naturally.

Technical Sessions

There was a number of AMAZING technical sessions, which are all available via the VMware Explore content catalog (click the link to view).

You can also view various community sessions, by going to VMware TV Live (click for link).

Check out my videos on VMware Explore 2024

I hope you enjoy the videos I created covering VMware Explore during the experience, and after.

VMware Explore 2024: A chat with Matt Heldstab about VMUG and Community

In this video, I sat down with Matt Heldstab, Global VP for VMUG (VMware User Group), to talk about the importance of community, VMUG, and how VMware Explore brings everyone together in person. This provides opportunities to reconnect with old friends, make new friends, and share a passion of technology (and VMware By Broadcom solutions).

VMware Explore 2024: A chat with Matt Heldstab about VMUG and Community

VMware Explore 2024: Day 1, Let’s Explore

On Day 1, I chatted with Daniel Keer (owner and writer at theDXT), to talk about his experience at VMware Explore. This was Daniel’s first VMware conference, and large tech conference for that matter.

Daniel was having a blast, commenting on how he was able to connect with VCF experts even before the conference started, and how friendly and welcoming everyone was.

VMware Explore 2024: Day 1, Let’s Explore

Daniel and I covered:

The Conference
Networking Oppurtunities
Broadcom Aquisition of VMware
NVIDIA AI Enterprise (NIMs and the NGC Catalog)
Private AI
VMware EUC split to Omnissa
World of EUC
- EUC World: Independence Conference
VMware Communities
- VMUG
- VMware vExpert Program
- Community Theatre

I was able to catch up with Daniel later, and he told me he was enjoying back to back technical sessions, and VMware Hands on Labs.

VMware Explore 2024: Day 3, Talking VCF and AI with Johan van Amersfoort

Johan van Amersfoort (owner/writer at vHojan.nl – Johan van Amersfoort) is a very well known presenter, often presenting on technologies such as EUC (with Omnissa Horizon), and now on AI and the endeavors both in his own home lab, as well as the work his organization is doing for customers with AI.

Johan and I talked about:

VMware Explore recap
Discussion on VCF, Private AI Foundation, and NVIDIA AI Enterprise (NIVIDIA NIMs)
Johan’s Presentations (see below for links)
Enterprise AI Workloads
VMware EUC divestiture (separation of Omnissa)
GPU Discussion
Tanzu, Cloud Foundry
Simplification of IT and Virtualization with VMware VCF
Community

At VMware Explore, Johan had two sessions (click the links to view):

VMware Explore 2024: Debrief and takeaways

Getting back home, there was a lot to take in and process, however I wanted to do a quick debrief and takeaway video.

In this video, I talk about:

Broadcom’s vision for VCF
Evolving world of IT
Corporate IT strategy shift
- Private AI
- Private Cloud
Practical AI use cases

Towards the end of the video, I also shared my thoughts on the last session I attended, called “GenAI-Powered Insights in Avi LB: A Journey into Next-Gen Observability“. You can click on the session name to find out more information and watch the video.

The session covered a real-world practical use of AI in IT, using an LLM to provide insight, analysis, administration, monitoring, and management of the VMware Avi Loadbalancer, using LLMs, with the choice of using Google Gemini in the cloud, or your own LLM on-prem using NVIDIA AI Enterprise, NIMs, and VMware Private AI Foundation.

Cool stuff!

VMware Explore, moving forward…

Moving forward, I’m aligned with and believe in the strategy of Private Cloud first, enabling technologies such as AI, while providing existing solutions such as compute.

I can’t wait for VMware Explore 2025!

Windows 11 Microsoft Account Center

Omnissa Horizon, Windows 11 4 Responses »

Aug 162024

Recently, new deployments of Windows 11 (23H2 images with the latest updates) have changed the behavior of the start menu and introduced the Windows 11 Microsoft Account Center.

This also effects 24H2, however 24H2 isn’t supported on Omnissa Horizon as of yet (to my knowledge) and probably most other VDI platforms, but this will be a concern once support is available.

The introduction of the Microsoft Account Center in the Windows Start Menu will become an issue for VDI deployments, as it changes the behavior of the Start Menu, and introduces some complexities for logging off users as well as introducing the need for training or alternative methods for users to log off.

Update – October 27 2024

When completing the latest Windows Updates, as of October 27th, 2024, the behavior has now changed.

The new behavior is now suitable for easy logoffs.

Behavior

When clicking on Start and proceeding to click on the user name, users are no longer prompted with options like “Sign out, Switch User, Account Settings”. Users are now presented with the new “Microsoft Account Center”, which on non-VDI deployments provides actions for the Microsoft Account. Optimizing your image may slightly change the behavior of the Microsoft Account Center.

Here is an example of the original start menu:

Screenshot of the traditional Windows 11 start menu with user icon.

Here is an example of the new start menu with the new Microsoft Account Center:

Screenshot of new Microsoft Account Center on Windows 11 Start Menu

Users, who are expecting to be able to sign out, will not have to click on the “…” on the top right.

Additional Considerations

In addition to the examples provided above, the following behaviors can be expected:

On the base image, clicking the user icon will do nothing (and may possibly kill the start menu)
In deployments with Hybrid domain joined Instant Clones
- If the machine hasn’t achieved Hybrid Domain Join, clicking the user account icon will function.
- If the machine has Hybrid domain joined but a PRT has not been issued, click the user account icon will kill the start menu.
- If the machine has Hybrid domain joined and a PRT has been issued, it will perform properly using the new style.
In deployments with Hybrid Domain Joining and PRT disabled, the new “Microsoft Account Center” from the user icon, should function properly with the new style.

As of today, I haven’t seen the latest Windows Updates change older base images, but I haven’t had the opportunity to sample a large enough number of environments. If this occurs, you may need to brief users on how to log out using the new “Microsoft Account Center”, using the “Log Off” function on the Horizon Client, or possibly even create a desktop shortcut for the users.

Workaround

To workaround this issue, you may need to train users on the new behaviour, advise them to Log off with the VMware Horizon client (proper logoff, not just clicking the “X” which will only disconnect sessions), or create a “Log off” shortcut on their desktop.

I will continue to investigate and update this post, hopefully ultimately with a fix.

I have created a thread on the Omnissa Community Forums, Windows 11 23H2 – Start Menu, Account Icon Behavior Warning.

vGPU Mixed Size Virtual GPU types with VMware vSphere 8U3

NVIDIA, NVIDIA vGPU, Omnissa Horizon, vSphere 12 Responses »

Jun 262024

I’m happy to announce today that you can now deploy vGPU Mixed Size Virtual GPU types with VMware vSphere 8U3, also known as “Heterogeneous Time-Slice Sizes” or “Heterogeneous vGPU types”.

VMware vSphere 8U3 was released yesterday (June 26th, 2024), and brought with it numerous new features and functionality. However, mixed vGPU types deserves it’s own blog post as it’s a major game-changer for those who use NVIDIA vGPU for AI and VDI workloads, including Omnissa Horizon.

NVIDIA vGPU (Virtual GPU) Types

When deploying NVIDIA vGPU, you configure Virtual GPU types that provide Workstation RTX (vWS Q-Series), Virtual PC (vPC B-Series), or Virtual Apps (vApps A-Series) class capabilities to virtual machines.

On top of the classifications above, you also needed to configure the Framebuffer memory size (or VRAM/Video RAM) allotted to the vGPU.

Historically, when you powered the first VM, the physical GPU that provides vGPU, would then only be able to serve that Virtual GPU type (class and Framebuffer size) to other VMs, locking all the VMs on running on that GPU to same vGPU type. If you had multiple GPUs in a server, you could run different vGPU types on the different physical GPU, however each GPU would be locked to the vGPU type of the first VM started with it.

NVIDIA Mixed Size Virtual GPU Type functionality

Earlier this year, NVIDIA provided the ability to deploy heterogeneous mixed vGPU types through the vGPU drivers, first starting with the ability to run different classifications (you could mix vWS and vPC), and the later adding support for mixed-size frame buffers (example, mixing a 4Q and 8Q profile on the same GPU).

While the NVIDIA vGPU solution supported this, VMware vSphere did not immediately add support so it couldn’t take advantage of this until the new release of VMware vSphere 8U3, VMware vCenter 8U3, and VMware ESXi 8U3.

To configure different classifications (vWS mixed with vPC), it requires no configuration other than using a host-driver and guest-driver that support it, however to use different sized framebuffers, it needs to be enabled on the host.

To Enable vGPU Mixed Size Virtual GPU types:

Log on to VMware vCenter
Confirm all vGPU enabled Virtual Machines are powered off
Select the host in your inventory
Select the “Configure” tab on the selected host
Navigate to “Graphics” under “Hardware”
Select the GPU from the list, click “Edit”, and change the “vGPU Mode” to “Mixed Size”

Screenshot showing the "Graphics Properties" for GPU adapters on VMware ESXi 8U3 with the "vGPU Mode" set to "Mixed Size"

Once you configure this, you can now deploy mixed-size vGPU profiles.

When you SSH in to your host, you can query to confirm it’s configured:

[root@ESXi-HOST:~] nvidia-smi -q

    vGPU Device Capability
        Fractional Multi-vGPU             : Supported
        Heterogeneous Time-Slice Profiles : Supported
        Heterogeneous Time-Slice Sizes    : Supported
        vGPU Heterogeneous Mode           : Enabled

It’s supported, and enabled!

Additional Notes

Please note the following:

When restarting your hosts, resetting the GPU, and/or restarting the vGPU Manager daemon, the ESXi host will change back to it’s default “Same Size” mode. You will need to manually change it back to “Mixed Mode”.
When enabling mixed-size vGPU types, the number of some types of vGPU profiles may be reduced vs running the GPU in equal-size mode (to allow other profile types). Please see the additional links for information on Mixed-Size vGPU types inside the “Virtual GPU Types for Supported GPUs” link.
Only “Best Effort and “Equal Share” schedulers are supported with mixed mode vGPU. Fixed Share scheduling is not supported.

Additional Links

Join me at VMware Explore 2024

VMware, VMware Explore, VMware Explore 2024 No Responses »

Jun 222024

I hope that you’ll have a chance to Join me at VMware Explore 2024 this year!

VMware Explore 2024, is being held at the Venetian and Palazzo in Las Vegas, on August 26 to 29th, 2024.

Image showcasing that Registration is open for VMware Explore 2024

VMware Explore is one of my favorite, and most important annual conferences for a number of reasons.

Networking

Through technology we make friends, connections, and friendships with those in our community. Chances are, you’ll see all your favorite people and that community, at VMware Explore.

You’ll have the chance to catch up with communities like VMUG (VMware User Group), the vExpert community, the vCommunity and I’m sure even some folks from World of EUC (like myself).

Additionally, you’ll get to network with like minded people passionate about the technology, experts in the field, and a diverse group of individuals from all over the world.

Learning

I can’t say how important the technical sessions are…

The sessions at VMware Explore help you learn in so many ways:

Learn about products you have interest in, but no experience
Learn more about the products you’re familiar with, become an expert!
Have a chat and dialogue with Product Managers, Presenters, and Staff about the solutions you work with, or are curious about
Catch up with and connect with experts (like vExperts)!

You can also save on certification by getting certified and taking exams at VMware Explore at a 50% discount!

Business

As the President and Owner of Digitally Accurate Inc. (a VMware and Broadcom Partner), attending this conference is crucial as it allows me to connect with customers, vendors, and VMware/Broadcom staff.

Deals get done, laughs are had, and these interactions really help advance and move business forward.

You get to have fun

And don’t forget, this event is FUN! There’s numerous events and parties that are held by vendors and community programs (such as vCommunity, VMUG, and more).

I highly recommend you keep your eyes glued to Discord, Slack, E-Mails, and the web to find the invite links to all the parties. Ask around!

Join me at VMware Explore 2024

With all that said, I hope to see you there!

Follow these hashtags to stay up to date

#VMwareExplore
#VMwareExploreHOL
#VMwareExploreSelfie
#VMwareExploreParty

Follow the official VMware Explore Social Media pages

Older Entries