Aug 122018
 
VMware Horizon View Icon

On VMware Horizon view after updating the view agent on the VM, you may notice that USB redirection stops working with the error “USB Redirection is not available for this desktop”. This is due to an issue with the certificates on the VDI host (The VM running the VDI OS), after the VMware view agent upgrade is completed.

To resolve this you must use MMC, open the local computer certificate store, browse to “VMwareView\Certificates”, delete the agent certificates (for the local agent), and finally reboot for the agent to regenerate the certificates.

See below for instructions:

  1. While connected to the VM running the VDI OS, click Start, type “mmc.exe” (without quotations), and open the Microsoft Management Console.
    mmc.exe

    Open MMC by running mmc.exe

     

  2. Open the “Add/Remote Snap-in” wizard.

    Open the Add/Remove Snap-in Wizard

     

  3. We must now open the local certificate store on the local computer. Select “Certicates” on the Available Snap-ins, click “Add”, select “Computer Account”, then proceed to choose “Local Computer” and complete the wizard.

    Select the Computer account certificate store on the local computer

     

  4. Expand the “Certificates (Local Computer)” on the left underneath “Console Root”. Expand “VMwareView”, then expand and select “Certificates”. Select the certificate on the right that matches the local computer name of the VDI host, right click and select “Delete”. You may have to do this multiple times if multiple certificates exist for the local computer.

    Delete the VMwareView local agent certificate

     

  5. Restart the VDI host. And USB redirection should now be working!

    VMware View USB Redirection issue resolved

     

Cheers to VDI!

May 172018
 
Digitally Accurate Inc. Logo

Looking for Calgary IT Managed Services or Calgary IT Consulting Services? We can help!

My company Digitally Accurate Inc. (https://www.digitallyaccurate.com/) has been helping businesses for almost 12 years with their IT strategies.

Feel free to reach out via E-mail, telephone, or LinkedIn to see how I can help! You can also visit the “Hire Stephen Wagner” tab, and yes, you’ll get to meet me! 🙂

My company is also partnered with numerous companies and can design, configure, and sell solutions including the following:

  • VMware Solution Design and Licensing
  • Veeam (Veeam Backup and Replication, Veeam Availability Suite, Veeam Backup Essentials)
  • HPE Servers, Storage, Networking
  • Aruba Networking
  • Microsoft Licensing (Including Office 365)
  • Sophos (Including Sophos UTM, and Sophos XG appliances)
  • 10ZiG Zero Clients
  • Duo Security (MFA)
  • Symantec (including Symantec Protection Suite)
  • Redhat (including RHEL: Redhat Enterprise Linux Server and Workstation)
  • Eaton UPS (Eaton Uninterrupted Power Supply and other Eaton power equipment)

Again, feel free to reach out for more information!

May 112018
 
Veeam Logo

This morning I noticed that CentOS 7.5 (1804) was available for upgrade via yum. After upgrading my CentOS instance from 7.4 to 7.5 on Microsoft Azure, I noticed that when running a backup using the Veeam Linux Agent, the system would crash and become completely unresponsive. I would have to manually restart the OS.

Upon reboot, I analyzed the console messages log and ran the backup again to see what was happening.

Here’s a look at my /var/log/messages:

May 11 07:24:46 HOSTNAME kernel: Request for unknown module key 'Veeam Software AG: 9d063645550b483bec752cb3c0249d5ede714b3e' err -11
May 11 07:24:46 HOSTNAME kernel: veeamsnap: loading out-of-tree module taints kernel.
May 11 07:24:46 HOSTNAME kernel: WARNING: module 'veeamsnap' built without retpoline-enabled compiler, may affect Spectre v2 mitigation
May 11 07:24:46 HOSTNAME kernel: veeamsnap: module verification failed: signature and/or required key missing - tainting kernel
May 11 07:24:46 HOSTNAME kernel: veeamsnap: applying kernel_stack fix up
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Loading
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Version: 2.0.0.400
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Author: Veeam Software AG
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init licence: GPL
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init description: Veeam Snapshot Kernel Module
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init zerosnapdata: 1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init debuglogging: 0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init snapstore enabled
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. container_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. container_sl_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. mem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. vmem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_pipe_init .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Module major=243
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_direct_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_redirect_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_deferred_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:snapimage_init .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:snapimage_init Snapimage block device was registered. major=252
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. container_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. container_sl_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. mem_cnt=1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. vmem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_open file=0xffff95e4543b1800
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_pipe_new .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ioctl_compatibility_flags Get compatibility flags
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ioctl_tracking_collect Collecting tracking device:
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracking_collect Have not device under CBT.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracking_add Adding. dev_id=8:1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create dev_id 8:1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create SectorStart    =0x800
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create SectorsCapacity=0xfa000
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_cbt_start .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:cbt_map_create CBT map create.
May 11 07:24:47 HOSTNAME kernel: general protection fault: 0000 [#1] SMP
May 11 07:24:47 HOSTNAME kernel: Modules linked in: veeamsnap(OE) nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_security ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr i2c_piix4 sg hv_utils i2c_core ptp pps_core hv_balloon ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix hv_storvsc hv_netvsc libata scsi_transport_fc hid_hyperv hyperv_keyboard scsi_tgt hyperv_fb crct10dif_pclmul crct10dif_common crc32c_intel hv_vmbus floppy serio_raw
May 11 07:24:47 HOSTNAME kernel: CPU: 1 PID: 1712 Comm: Lpb Server thre Tainted: G           OE  ------------   3.10.0-862.2.3.el7.x86_64 #1
May 11 07:24:47 HOSTNAME kernel: Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
May 11 07:24:47 HOSTNAME kernel: task: ffff95e447378000 ti: ffff95e45cbe0000 task.ti: ffff95e45cbe0000
May 11 07:24:47 HOSTNAME kernel: RIP: 0010:[]  [] page_array_memset+0x4d/0xa0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: RSP: 0018:ffff95e45cbe3d60  EFLAGS: 00010246
May 11 07:24:47 HOSTNAME kernel: RAX: 0000000000000000 RBX: ffff95e449615200 RCX: 0000000000000200
May 11 07:24:47 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff187288716000
May 11 07:24:47 HOSTNAME kernel: RBP: ffff95e45cbe3d60 R08: ffffffffbe274fef R09: ffff95e460affa60
May 11 07:24:47 HOSTNAME kernel: R10: ffff95e460affa60 R11: 0000000000000000 R12: 0000000000000001
May 11 07:24:47 HOSTNAME kernel: R13: 00000000000fa000 R14: 0000000000000000 R15: 0000000000800001
May 11 07:24:47 HOSTNAME kernel: FS:  00007f336f7fe700(0000) GS:ffff95e495640000(0000) knlGS:0000000000000000
May 11 07:24:47 HOSTNAME kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 11 07:24:47 HOSTNAME kernel: CR2: 0000000000738df0 CR3: 00000002d3afc000 CR4: 00000000001406e0
May 11 07:24:47 HOSTNAME kernel: Call Trace:
May 11 07:24:47 HOSTNAME kernel: [] cbt_map_allocate+0x6e/0x160 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] cbt_map_create+0x73/0x100 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracker_cbt_start+0x5a/0xc0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracker_Create+0x16a/0x650 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracking_add+0x2e0/0x450 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] ioctl_tracking_add+0x6c/0x170 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] ctrl_unlocked_ioctl+0x4e/0x60 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] do_vfs_ioctl+0x350/0x560
May 11 07:24:47 HOSTNAME kernel: [] ? __sb_end_write+0x31/0x60
May 11 07:24:47 HOSTNAME kernel: [] ? vfs_write+0x182/0x1f0
May 11 07:24:47 HOSTNAME kernel: [] SyS_ioctl+0xa1/0xc0
May 11 07:24:47 HOSTNAME kernel: [] system_call_fastpath+0x1c/0x21
May 11 07:24:47 HOSTNAME kernel: Code: 01 49 89 f9 48 0f af c2 49 8b 79 10 ba 00 10 00 00 40 f6 c7 01 75 47 40 f6 c7 02 75 51 40 f6 c7 04 75 2b 89 d1 c1 e9 03 83 e2 07  48 ab 74 0e 41 89 c8 83 c1 01 39 d1 42 88 34 07 72 f2 49 83
May 11 07:24:47 HOSTNAME kernel: RIP  [] page_array_memset+0x4d/0xa0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: RSP 
May 11 07:24:47 HOSTNAME kernel: ---[ end trace 96b51a664f4baef9 ]---

It appeared the veeamsnap module was causing a protection fault with the kernel, causing the system to crash.

In an effort to troubleshoot, I uninstalled and reinstalled Veeam and tried rebuilding the kernel module, however the issue still persisted. After some searching, I came across these two posts:

https://forums.veeam.com/veeam-agent-for-linux-f41/veeam-agent-for-linux-and-rhel-7-5-crash-t50170.html

https://www.veeam.com/kb2569

According to the KB, the veeamsnap module isn’t compatible with kernel version 3.10.0-862.

Checking my system after upgrading CentOS 7.5:

[root@HOSTNAME ~]# uname -a
Linux HOSTNAME.somedomain.com 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@HOSTNAME ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

 

Essentially, as of right now the Veeam Agent for Linux is not yet supported on CentOS 7.5, RHEL 7.5, or Oracle RHCK 7.5. If you’re running any of these, hold off and do not install Veeam Agent for Linux. If you are scheduling an upgrade, do not perform upgrade unless you want to break your backup. It sounds like this should be fixed in a future update.

May 082018
 

Recently a customer of mine who is using an outdated version of Intuit QuickBooks on their Terminal Server (RDS Remote Desktop Services) started to experience an issue when users attempted to log on. QuickBooks would initialize, then prompt the TLS 1.2 warning message, and then completely crash. This would prevent the users from doing any tasks.

In an effort to troubleshoot this, I tried to use different accounts, checked the QBW.ini file for any flags that could dismiss this message, tried Intuit’s “TLS preparedness tool” (which still scares me because I have no idea what system changes it made on the server), etc… All of these attempts had no effect on the issue.

For a temporary workaround, you’ll need to load up QuickBooks on a workstation (not using terminal services). You’ll need to open the datafile, select “Do Not Show this again”, and then close. You’ll need to do this for each datafile. Please note that if you do not receive the prompts on other datafiles, you’ll need to open the datafile with a different Quickbooks username (QB account, not Windows account) in order to get it to prompt.

This issue should only occur on older versions of Quickbooks when using TS/RDS. To officially resolve this issue, I recommend upgrading to a more recent (and in support) version of Quickbooks.

May 062018
 
DUO

I’m a big fan of MFA, specifically Duo Security‘s product (I did a corporate blog post here). I’ve been using this product for some time and use it for an extra level of protection on my workstations, servers, and customer sites. I liked it so much so that my company (Digitally Accurate Inc.) became a partner and now resells the services.

Here’s a demo of DUO MFA being used with CentOS Linux:

Today I want to write about a couple issues I had when deploying the pam_duo module on CentOS Linux 7. The original duo guide can be found at https://duo.com/docs/duounix, however while it did work for the most part, I noticed there were some issues with the pam configuration files, especially if you are wanting to use Duo MFA with usernames and passwords, and not keys for authentication.

A symptom of the issue: I noticed that when following the instructions on the website for deployment, after entering the username, it would skip the password prompt, and go right for DUO authentication, completely bypassing the password all together. I’m assuming this is because the guide was written for key authentication, but I figured I’d do a quick crash-course post on the topic and create a simple guide. I also noticed that sometimes even if an incorrect password was typed in, it would allow authentication if DUO passed as successful.

Ultimately I decided to learn about PAM, understand what it was doing, and finally configure it properly. Using the guide below I can confirm the password and MFA authentication operate correctly.

To configure Duo MFA on CentOS 7 for use with usernames and passwords

First and foremost, you must log in to your Duo Account and go to applications, click “Protect an Application” and select “Unix Application”. Configure the application and document/log your ikey, secret key, and API hostname.

Now we want to create a yum repo where we can install, and keep the pam_duo module up to date. Create the file /etc/yum.repos.d/duosec.repo and then populate it with the following:

[duosecurity]
name=Duo Security Repository
baseurl=http://pkg.duosecurity.com/CentOS/$releasever/$basearch
enabled=1
gpgcheck=1

We’ll need to install the signging key that the repo uses, and then install the duo_unix package. By using yum, we’ll be able to keep this package regularly up to date when we update the server. Run the following commands:

rpm --import https://duo.com/RPM-GPG-KEY-DUO
yum install duo_unix

Configure the pam_duo module by editing the /etc/duo/pam_duo.conf file. You’ll need to populate the lines with your ikey, secret key, and API hostname that you documented above. We use “failmode=safe” so that in the event of an internet disconnection, we can still login to the server without duo. It’s safe to enable this fail-safe, as the purpose is to protect it against the internet. Please see below:

[duo]
; Duo integration key
ikey = XXXXXXXXXXXXXXXXXXXX
; Duo secret key
skey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
; Duo API host
host = XXXXXXXXX.duosecurity.com
; Send command for Duo Push authentication
pushinfo = yes
; failmode safe if no internet it works (secure locks it up)
failmode = safe

Configure sshd to allow Challenge Response Authentication by editing /etc/ssh/sshd_config, then locate and change “ChallengeResponseAuthentication” to yes. Please note that the line should already be there, and you should simply have to move the comment symbol to comment the old line, and uncomment the below line as shown below:

ChallengeResponseAuthentication yes

And now it gets tricky… We need to edit the pam authentication files to incorporate the Duo MFA service in to it’s authentication process. I highly recommend that throughout this, you open (and leave open) an additional SSH session, so that if you make a change in error and lock yourself out, you can use the extra SSH session to revert any changes to the system to re-allow access. It’s always best to make a backup and copy of these files so you can easily revert if needed.

DISCLAIMER: I am not responsable if you lock yourself out of your system. Please make sure that you have an extra SSH session open so that you can revert changes. It is assumed you are aware of the seriousness of the changes you are making and that you are taking all precautions (including a backup) to protect yourself from any errors.

Essentially two files are used for authentication that we need to modify. One file is for SSH logins, and the other is for console logins. In my case, I wanted to protect both methods. You can do either, or both. If you are doing both, it may be a good idea to test with SSH, before making modifications to your console login, to make sure your settings are correct. Please see below for the modifications to enable pam_duo:

/etc/pam.d/password-auth (this file is used for SSH authentication)

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
#auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_unix.so nullok try_first_pass
auth  sufficient pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok


password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

/etc/pam.d/system-auth (this file is used for console authentication)

auth        required      pam_env.so
auth        sufficient    pam_fprintd.so
#auth        sufficient    pam_unix.so nullok try_first_pass
# Next two lines are for DUO mod
auth        requisite     pam_unix.so nullok try_first_pass
auth        sufficient    pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= ucredit=-1 lcredit=-1 dcredit=-1 ocredit=-1
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=5
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Now, we must restart sshd for the changes to take affect. Please make sure you have your extra SSH session open in the event you need to rollback your /etc/pam.d/ files. Restart the sshd service using the following command:

service sshd restart

Attempt to open a new SSH session to your server. It should now ask for a username, password, and then prompt for Duo authentication. And you’re done!

More information on Duo Multi Factor Authentication (MFA) can be found here.