Re-blog: Exchange 2016 CU7 Bug Causes HCW to Fail

Posted by Terence Kolstad on Nov 18, 2017 10:04:00 AM

Original blog post by Zoltan Erszenyi

Hi There,

If you plan to deploy a hybrid Exchange server using Exchange 2016 CU7, then don't. It's broken. Use a pre-CU7 server instead.

A bug slipped into Exchange 2016 CU7 which prevents the HCW from completeing. The HCW fails to get past the domain ownership validation:

No matter how hard you try, you can't get past this screen.

Looking at the HCW log at C:\Users\<username>\AppData\Roaming\Microsoft\Exchange Hybrid Configuration, the following errors are logged:

*ERROR* 10300 [Client=UX, Thread=1] System.ArgumentOutOfRangeException: The UTC time represented when the offset is applied must be between year 0 and 10,000.
                                Parameter name: offset
                                   at System.DateTimeOffset.ValidateDate(DateTime dateTime, TimeSpan offset)
                                   at System.DateTimeOffset..ctor(DateTime dateTime)
                                   at Microsoft.Online.CSE.Hybrid.App.AppData.BuildSessionProperties(NotificationType type)

 

...

*ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=14] FINISH Time=2598.6ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object.

...

*ERROR* 10224 [Client=UX, Page=DomainProof, Thread=14] Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object. ---> System.Management.Automation.RemoteException: Object reference not set to an instance of an object.
                                   at System.Management.Automation.Runspaces.AsyncResult.EndInvoke()
                                   at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
                                   at Microsoft.Online.CSE.Hybrid.Provider.PowerShell.PowerShellProvider.PowerShellInstance.Invoke(String cmdlet, IReadOnlyDictionary`2 parameters, Int32 millisecondsTimeout)
                                   --- End of inner exception stack trace ---
                                   at Microsoft.Online.CSE.Hybrid.PowerShell.RemotePowershellSession.RunCommandInternal(Cmdlet cmdlet, SessionParameters parameters, Int32 millisecondsTimeout, PowerShellRetrySettings retrySettings, Boolean skipCmdletLogging)
                                   at Microsoft.Online.CSE.Hybrid.Session.PowerShellOnPremisesSession.SetFederatedOrganizationIdentifier(SmtpDomain accountNamespace, String delegationTrustLink, SmtpDomain defaultDomain)
                                   at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.DomainInfo.AddFederatedDomain(IOnPremisesSession session, AppData appData)
                                   at System.Collections.Generic.List`1.ForEach(Action`1 action)
                                   at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.VerifyActivity(IOnPremisesSession session, EnvironmentBase environment)

 
Issues with the HCW have also been reported by admins on Rhoderick Milen's blog at https://blogs.technet.microsoft.com/rmilne/2017/09/19/exchange-2016-cu7-released/
 
Microsoft is aware of it, and a fix will be included with CU8. Some suggest using a pre-CU7 server to run the HCW on - I haven't tried, not (yet) an option in the affected environment.

Update:

  • Tried running the HCW on a CU6 server that I installed temporarily. Since the CU7 server was still selected as the hybrid server, the wizard FAILED.
  • Tried the workaround suggested by AloneInTheDarK. While some reported success, it didn't work for me - see https://social.technet.microsoft.com/Forums/en-US/3ac7cfe6-6c97-4f68-84b9-d498aafd4ea1/exchange-2016-cu6?forum=Exch2016Adm. Microsoft advised that it is not a recommended workaround, even if it would have worked.
  • Downgraded the CU7 server to CU6. SUCCESSFUL. It involved:
    • Installed a temporary CU6 server to host all the mailboxes specific to Exchange 2016 while the CU7 server was downgraded (Arbitration, AuditLog etc. - see http://ezoltan.blogspot.com/2016/06/delete-that-stubborn-exchange-2016.html and see https://technet.microsoft.com/en-us/library/mt441791(v=exchg.150).aspx). This is needed so that the CU7 server can be uninstalled.
    • Uninstalled Exchange 2016 CU7 and deleted the binaries from the installation folder.
    • Installed Exchange 2016 CU6 on the same machine.
    • Configured the downgraded server (virtual directories, certificate etc.), and moved all system mailboxes back to it..
    • Uninstalled the temp Exchange 2016 server.
    • Ran the HCW wizard - SUCCESSFUL
It looks like Microsoft has a hotfix. At this time I don't know whether it will ever be published as CU8 is due in the near future, but if you are in dire need then open a case and request the hotfix. I didn't get to test because it came in one day late, after I went through the downgrade ordeal.
 
Take care,
Zoltan

Topics: Insider, Education

About this blog

Welcome to the Cyber Advisors Blog.  Please take a moment to read through our content.  If you would like more information on any of these topics, simply reach out to us via contact information below.  If you find our content valuable, please subscribe.  

 

 
 
Would you like to hear from us? Click Below!
Learn More

Subscribe Here!

Recent Posts

Posts by Tag

See all