UniFi Cloud Key Gen 2 Plus – Firmware update fails

I was recently working on a UniFi deployment which included a UniFi cloud key gen 2 plus. The cloud key came from Ubiquiti with firmware version 0.9.8. Attempting to update the firmware resulted in the error “something went wrong” without getting past 1% on the progress.

Updating the controller and protect software was successful, but attempting to update the cloud key firmware from within the network application also failed. Read More

VMWare Vcenter Appliance Install Stuck Stage 2

When deploying VMWare VCSA in a lab environment the installer often gets stuck at stage 2 starting services. In my case, starting authentication network 2%. This seems to be an issue with VCSA wanting to verify the SSO domain through DNS resolution.

The solution for me was to create an entry in the hosts file of the VCSA appliance through SSH to the CLI. Before doing so, it is best to start from scratch. Delete the partially deployed VCSA appliance. Deploy VCSA through Stage 1 but do not enter Stage 2. Read More

Deploy Office 2019 via GPO

With the advent of Office 2019, Microsoft has moved away from GPO deployment via MSI. There is no MSI of Office, Visio, Project, etc available to download anymore. Microsoft is moving toward using SCCM or the Office Deployment tool. I was tasked with coming up with a method for deploying Office via GPO in a fully automated manner.

There may be more than one way to accomplish GPO deployment of Office, and I do not claim to have the best method. It took me quite a bit of research and troubleshooting to get this method to work. I hope it helps someone looking to accomplish the same thing I was. Read More

Cisco – Define default GW based on source

Was recently working on a project for a client where the client needed to route traffic from a specific network over a T1 and send the rest of the traffic out the default gateway of the router.  In order to accomplish the task I configured policy based routing on the 2911 as follows:

interface GigabitEthernet0/0.4

encapsulation dot1q 4

ip address 10.2.1.1 255.255.255.0 Read More

Skype for Business 2015 – Mobility clients not able to find contacts

I was recently asked to look in to a Skype for Business 2015 infrastructure due to reported 2013 mobility client issues.  The infrastructure consisted of a standard edition front end, edge server and KEMP load master reverse proxy.  The issue was that mobility clients could not search for contacts and could not see certain status messages.  All other features were working and users could chat/make calls.

Testing with https://www.testconnectivity.microsoft.com/ shows green across the board.  If you are dealing with this issue, start with this tool and run the following tests:

  • Skype for Business remote connectivity test
  • Skype for Business autodiscover test
  • Exchange server ActiveSync autodiscover test

Testing with Microsoft Lync Connectivity Analyzer showed ready for 2013 mobility client.

After examining the Lync Front End server event log, I found event 32054, LS Storage service:

Storage Service had an EWS Autodiscovery Failure.  The underlying connection was closed.  Could not establish a trust relationship SSL/TLS.

The issue would seem to be the published autodiscover Uri for Exchange not matching the installed certificate on the Exchange 2016 DAG members.  The Uri in the event log was reporting autodiscover.domain.local.  The certificates and all other services in the infrastructure were pointing to autodiscover.domain.org.  On the Exchange server, running powershell Get-ClientAccessService | fl AutoDiscoverServiceInternalUri will display the currently assigned URLs.

Issuing a Set-CsClientAccessService -Identity exchange.domain.local -AutoDiscoverServiceInternalUri https://autodiscover.domain.org/Autodiscover/Autodiscover.xml for both servers in the Exchange DAG solved the mobility client address book issue.

Back from the Dead – Virtualizing DOA Laptops

Simple post on restoring dead laptops using virtualization.  Some people ask me: “Is there any way to bring a laptop back from the dead?” or “I dropped my laptop in the toilet and now it wont start! How do I get to my important stuff?”.   The answer is not exactly simple, but it is effective in most cases.  If you need access to the OS for some reason prior to restoring data on a new system, it may be possible to restore the laptop and boot it as a VM.

First off, you need to remove the physical hard drive from the device.  You will want the primary (boot) disk.  You may also need the secondary hard drives if you installed critical apps in non-standard locations.

Second, you need a hard-drive caddy of some sort. Something like this: http://www.newegg.com/Product/Product.aspx?Item=N82E16817270043 would work just fine.  Any enclosure will due.

Third, you need a working computer to plug the enclosure in to.  This will mount the OS drive as an external drive on the chosen machine.  We need to convert this drive to a format virtualization programs will understand.  The most common of which is Virtual Hard Disk (VHD/VHDX) format.  To do this, we need a program like Disk2vhd http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx

Run Disk2vhd, check the appropriate partitions from the connected external drive.  Choose a location with enough space and name your VHD.  Un-check VHDX if you intend to use VirtualBox as the VM player.

Once the VHD is created, install a VM player.  I will be using VirtualBox in this example.

Run VirtualBox and create new VM.  Choose the OS settings from the drop downs.  This must match the version of the OS that was running on the old machine.  If you don’t know the exact version, one way to check is to inspect the ntoskrnl file under /Windows/System32/, look at details and find the product version number.  Reference the version number here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms724832%28v=vs.85%29.aspx and here: http://www.gaijin.at/en/lstwinver.php

Select your memory amount and boot the VM.  The old OS should boot fine and install all new drivers needed.  You may need to perform some system repair, or OS repair due to driver conflicts but most of the time the boot is clean.

 

Cisco 7960G 3.x Firmware Upgrade Issues

Working on a project to bring a batch of Cisco 7960G phones online with a newer 7.1 CUCM server.  The issue is that the phones were donated, and although new in the box they were using just about every firmware version from 3.x to 8.x, SCCP and SIP.  The newer revisions were simple enough to manually upgrade via TFTP, but the 3.x revisions absolutely would not upgrade. We tried upgrading from 3.x to 5.x, 7.x, etc with no luck.  We would have liked to upgrade them from 3.1 to a newer 3.x revision, but all of those files from Cisco are packaged in an exe intended for CME.  This limited our options.  Finally found an obscure way to get around the UAL conflicts.

Workaround upgrade from 3.x to 7.x SIP then back to SCCP.

You will need the following things:

  • A switch isolated from the production network
  • A TFTP server (TFTPD32)
  • A DHCP server (either the switch or using TFTPD32)
  • Cisco P0S3-07-4-00 SIP zip from Cisco support or here: http://radiotwenterand.nl/~graver/cisco/SIP-7960/
  • Cisco 8.1(1) SCCP zip from Cisco support
  • Cisco 8.1(2) SCCP zip from Cisco support

Set up the switch and plug the PC that you have a TFTP server installed on to the same switch.  Unzip the different revisions to individual folders under the root TFTP directory.  We will be switching back and forth a lot.  Configure the DHCP server with the scope of our choice, and specify option 150.  Point option 150 to the IP address of your TFTP server.  My switch DHCP pool looked something like this:

ip dhcp pool PHONELAB

network 10.1.1.0 255.255.255.0

default-router 10.1.1.1 Read More

Exchange 2013 ECP – :-( Something Went Wrong

Fresh deployment of Exchange 2013.  As you try to connect to the management console https://localhost/ecp/ with a valid administrator account, it redirects to OWA and responds with “:-( Something Went Wrong, A problem occurred when trying to use your mailbox.”

This seems to be related to left over Exchange attributes in Active Directory.  I tried everything from a multitude of TechNet articles including the following:

  1. Rebuilding the ECP front end and back end (Reference)
  2. Adding appropriate permissions to the administrator account
  3. Connecting directly to the back end by using port 444
  4. Messing with certificates (Reference)
  5. Many articles mentioned having CAS and MBX roles installed, but these were installed on the same machine at the same time
  6. Manually creating the administrator mailbox using the exchange shell

Finally I tried just creating another user account in AD and gave it permissions to manage Exchange by adding it to the group ADUC–>Domain.local–>Microsoft Exchange Security Groups–>Organization Management.

That allowed access to the ECP and after some more digging in ADSIedit, I can see old exchange attributes attached to the enterprise Administrator account.  I believe this is the problem as the management console lists Administrator as already having a “legacy” mailbox.  The attributes are referencing exchange servers that no longer exist in the environment.

I plan to start ripping out old exchange attributes to test this theory.

Exchange 2013 Schema Prep – ADC Found

Installing a new Exchange 2013 deployment on an existing domain with questionable history.  The existing domain had untold number of previous Exchange deployments with remnants scattered throughout Active Directory.  After using ADSI edit to manually remove all instances of the old Exchange servers, running a schema prep runs in to an error about an existing Active Directory Connector.  Microsoft suggests (TechNet) disabling the ADC service on the running computer, then uninstalling the service using the exchange server installation CD.  In this situation of course the detected ADC was probably on a computer that was long ago trashed or removed.

Lots of websites describe the method for removing ADCs from AD using ADSIedit.

(Reference)

(Reference)

Open ADSIedit
Expand Configuration –> Services –> Microsoft Exchange –> Active Directory Connections
Delete ADC under Active Directory Connections
Replicate changes to all Domain Controllers

Running the Schema prep another time results in the same error:

After some digging we found the rogue connector buried in Active Directory Sites and Services.

Open the sites and services mmc and look for any machines that aren’t active servers.  Delete anything that doesn’t belong and run the tool again.  It should finish successfully.

Cisco Stackwise Port Flapping

I recently was confronted with a stack of six Cisco 3750X switches that were experiencing intermittent outages.  The symptoms were random, but included switches being removed from the stack randomly, PoE drops, and full blown switch crashes with reloads.  If anyone has worked with Cisco stacks, you know how long it takes the entire stack to reload and elect a new master.

Troubleshooting the situation, I found that the switch stackwise ports were flapping.  All of them were reporting up/down notifications at random intervals.

I brought the stack down, removed all of the stack cables and brought the stack up one switch at a time.  I wanted to test the physical stack cables as I assumed there must one or more cables with issues.

By connecting the stackwise cable in a loop on the master, one can verify the physical operation of each cable.

SM: Detected stack cables at PORT1 PORT2

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state UP

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state UP

Switch# show switch stack-ports

Switch# Port 1       Port 2   ——–    ——       ——     1           Ok           Ok

I monitored each cable for 5 minutes until satisfied that there was no issue.  However; one cable when connected began to flap up/down.

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state UP

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state UP

%STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN

Interestingly enough, once the bad cable was removed from the stack, ALL of the issues with the switch stack were resolved.