[SOLVED] CRITICAL INCIDENT - Some users may be unable to search in SharePoint Online

 

Updates (all in GMT):

27th November @ 07:30 *SOLVED*

26th November @ 08:40 

25th November @17:30  

25th November @14:10

25th November @10:50

25th November @10:30

Details:

27th November @ 07:30

All (but one) of the clients who reported this issue to us have confirmed it is now resolved, consistent with the final update on the official Microsoft ticket, which is now closed with a "Service Restored" status. 

If you are still experiencing this issue, do try refreshing your browser, and if the issue persists, arrange for a support ticket to be raised to Microsoft, referring to MO941162

26th November @ 08:40

During the deployment of the fix, Microsoft experienced another issue with the speed with which their fix was clearing a queue of affected users. 

For the handful of clients who reported this to us, the search service looks to be returning to normal. If you are still experiencing issues with the search service, I would expect that to clear at some point today. 

The latest Microsoft update (@08:04 GMT) states that, "We’ve isolated the cause of mail queue delays and have restarted the affected infrastructure to drain stalled queues. We’ll remain in an extended monitoring phase until this draining is completed and we can consider the incident fully recovered. " 

25th November @17:30

A couple of hours ago, it looked as though Microsoft had deployed the fix to about 60% of the affected infrastructure, but the latest update from them shows, "We're rerouting traffic to alternate infrastructure to expedite mitigation. In parallel, we're resolving complications leading to the delays in targeted server restarts. This quick update is designed to give the latest information on this issue." 

The issue does look to have settled in the ClearPeople tenant, but it is ongoing for users in the Atlas community. Do try clearing your Atlas cache and refresh your browser, there are reports of this working inconsistently. 
Fingers crossed this is resolved by the morning 🤞

25th November @14:10

It appears that Microsoft have deployed a fix, with the latest update reading, "We’ve started to deploy the fix which is currently progressing through the affected environment. While this progresses, we’re beginning manual restarts on a subset of machines that are in an unhealthy state."

This is great news....but all of us should expect disruption to continue throughout the day.  

25th November @10:50

The earlier Microsoft incident number we shared here has been closed and moved to this reference: Service health - Microsoft 365 admin center

25th November @10:30

We are aware that a Critical Microsoft Incident is affecting the search experience with SharePoint this morning. 

For Atlas clients this will manifest with In Focus Results webparts failing to load, displaying a pink error ribbon, referring to [http]:500 errors 

You, or your IT admins can read more about this issue here: Service health - Microsoft 365 admin center

We are expecting Microsoft to publish an update on the issue at 11:00 GMT.

If you have any concerns do please reach out to Microsoft directly: Raising a Microsoft Support Ticket with Evidence – Atlas by ClearPeople