Original blog post by Zoltan Erszenyi
Hi There,
If you plan to deploy a hybrid Exchange server using Exchange 2016 CU7, then don't. It's broken. Use a pre-CU7 server instead.
A bug slipped into Exchange 2016 CU7 which prevents the HCW from completeing. The HCW fails to get past the domain ownership validation:
No matter how hard you try, you can't get past this screen.
Looking at the HCW log at C:\Users\<username>\AppData\Roaming\Microsoft\Exchange Hybrid Configuration, the following errors are logged:
*ERROR* 10300 [Client=UX, Thread=1] System.ArgumentOutOfRangeException: The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
at System.DateTimeOffset.ValidateDate(DateTime dateTime, TimeSpan offset)
at System.DateTimeOffset..ctor(DateTime dateTime)
at Microsoft.Online.CSE.Hybrid.App.AppData.BuildSessionProperties(NotificationType type)
...
*ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=14] FINISH Time=2598.6ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object.
...
*ERROR* 10224 [Client=UX, Page=DomainProof, Thread=14] Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object. ---> System.Management.Automation.RemoteException: Object reference not set to an instance of an object.
at System.Management.Automation.Runspaces.AsyncResult.EndInvoke()
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.Online.CSE.Hybrid.Provider.PowerShell.PowerShellProvider.PowerShellInstance.Invoke(String cmdlet, IReadOnlyDictionary`2 parameters, Int32 millisecondsTimeout)
--- End of inner exception stack trace ---
at Microsoft.Online.CSE.Hybrid.PowerShell.RemotePowershellSession.RunCommandInternal(Cmdlet cmdlet, SessionParameters parameters, Int32 millisecondsTimeout, PowerShellRetrySettings retrySettings, Boolean skipCmdletLogging)
at Microsoft.Online.CSE.Hybrid.Session.PowerShellOnPremisesSession.SetFederatedOrganizationIdentifier(SmtpDomain accountNamespace, String delegationTrustLink, SmtpDomain defaultDomain)
at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.DomainInfo.AddFederatedDomain(IOnPremisesSession session, AppData appData)
at System.Collections.Generic.List`1.ForEach(Action`1 action)
at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.VerifyActivity(IOnPremisesSession session, EnvironmentBase environment)
Update:
- Tried running the HCW on a CU6 server that I installed temporarily. Since the CU7 server was still selected as the hybrid server, the wizard FAILED.
- Tried the workaround suggested by AloneInTheDarK. While some reported success, it didn't work for me - see https://social.technet.microsoft.com/Forums/en-US/3ac7cfe6-6c97-4f68-84b9-d498aafd4ea1/exchange-2016-cu6?forum=Exch2016Adm. Microsoft advised that it is not a recommended workaround, even if it would have worked.
- Downgraded the CU7 server to CU6. SUCCESSFUL. It involved:
- Installed a temporary CU6 server to host all the mailboxes specific to Exchange 2016 while the CU7 server was downgraded (Arbitration, AuditLog etc. - see http://ezoltan.blogspot.com/2016/06/delete-that-stubborn-exchange-2016.html and see https://technet.microsoft.com/en-us/library/mt441791(v=exchg.150).aspx). This is needed so that the CU7 server can be uninstalled.
- Uninstalled Exchange 2016 CU7 and deleted the binaries from the installation folder.
- Installed Exchange 2016 CU6 on the same machine.
- Configured the downgraded server (virtual directories, certificate etc.), and moved all system mailboxes back to it..
- Uninstalled the temp Exchange 2016 server.
- Ran the HCW wizard - SUCCESSFUL