Last week we started deployment of a customer’s 2013 servers into their Exchange 2010 environment. We started experiencing issues with the first of ten servers we were deploying immediately after install. Initially the thought was it was related to AD as there were some AD and network related issues occurring in parallel. Once the all clear was given on the AD and networking work we retested Exchange but the issue remained, so we decided to deploy 2013 server #2. The issue was then also experienced on #2. Sad panda.
The Issue
The behavior was simply that cmdlets executed in the Exchange Management Shell(EMS), seemed to never complete. Only to come back later and retry a cmdlet later on and it succeeded. Then retry shortly after that and the hung state appeared.
We were also unable to log onto the Exchange Admin Center(EAC) too, it sat there and spun like you were eventually going to get on but that never occurred.
In troubleshooting the issue I decided to let a Get-ExchangeServer -verbose cmdlet run without ctrl+c/breaking it. The cmdlet eventually completed after 10 minutes. Output below.
[PS] C:\>Get-ExchangeServer -Verbose
VERBOSE: [20:58:18.661 GMT] Get-ExchangeServer : Runspace context: Executing user: subdomain.rootdomain.com/Users/AdminUser, Executing user organization: , Current organization: , RBAC-enabled: Enabled.
VERBOSE: [20:58:18.676 GMT] Get-ExchangeServer : Initializing Active Directory server settings for the remote Windows
PowerShell session.
VERBOSE: [21:08:52.790 GMT] Get-ExchangeServer : Active Directory session settings for 'Get-ExchangeServer' are: View
Entire Forest: 'False', Default Scope: ‘subdomain.rootdomain.com, Configuration Domain Controller: ‘DC02.subdomain.rootdomain.com,
Preferred Global Catalog: ‘DC01.subdomain.rootdomain.com, Preferred Domain Controllers: '{ DC01.subdomain.root.com }'
VERBOSE: [21:08:52.790 GMT] Get-ExchangeServer : Beginning processing Get-ExchangeServer
VERBOSE: [21:08:52.853 GMT] Get-ExchangeServer : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write
Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s):
{}, Exclusive Configuration Scope(s): {} }
VERBOSE: [21:08:52.853 GMT] Get-ExchangeServer : Resolved current organization: .
VERBOSE: [21:08:52.853 GMT] Get-ExchangeServer : Searching objects of type "Server" with filter "$null", scope
"SubTree" under the root "$null".
VERBOSE: [21:08:52.853 GMT] Get-ExchangeServer : Request filter in Get Task:
(&(objectCategory=msExchExchangeServer)(|(&(msExchVersion<=1125899906842624)(!(msExchVersion=1125899906842624)))(!(msEx
chVersion=*)))).
VERBOSE: [21:08:52.931 GMT] Get-ExchangeServer : Previous operation run on domain controller
‘DC02.subdomain.rootdomain.com.
VERBOSE: [21:08:52.931 GMT] Get-ExchangeServer : Preparing to output objects. The maximum size of the result set is
"Unlimited".
After this cmdlet finished, I executed a number of others and they behaved how you’d expect them to, snappy. Also, if another admin was on the server, their cmdlets started executing properly and quickly too.
This.was.really.odd.
I opened a new EMS window and retried a ‘get-mailbox john’ command and we had our hang again.
We decided to open a case with Microsoft Premier Support since the logs were clean and nothing else was apparent during further troubleshooting. Support came back quickly and said this is a known bug with Exchange 2013 CU5 and has already been logged with the Product team. They provided a work around for the EMS in that you could open PowerShell and directly import the Exchange Management Snapin even though that’s technically unsupported. Unfortunately there isn’t a work around for the EAC.
Resolution
Microsoft Support’s resolution is to rollback to CU4/SP1 or if you can wait, wait for CU6 where this bug is scheduled to be resolved. CU6 is also is slated to bring usable public folders to Exchange 2013. We’re in the process of rolling back to CU4/SP1 since waiting for CU6 would hold up a rather large customer deployment.
We have deployed many other Exchange 2013 CU5 servers recently for other customers and have yet to run into this problem until this customer. Whether certain environment variables are required for this bug to appear are unknown and MS Support didn’t disclose any information stating as such.
In doing research to see if others are talking about this, I didn’t find much of anything. Hopefully this will help someone else in the event they run into this issue.
Kaynak
https://www.lopo.com/2014/08/06/exchange-2013-cu5-bug-iammec/