NOTE: This article will only apply if you have Deployed Atlas AI with your Atlas Deployment. Atlas AI is an optional Atlas feature that customer may or not choose to deploy during Atlas Installation or Updates. If you do NOT have Atlas AI Deployed, this article is NOT applicable to you.
If your initial deployment of Atlas is Prior to April 3rd 2024, this article may be of use to you. Prior to this date, the Azure AI Search Service created during the Deployment of Atlas had a Partition size of 2GB for the Basic SKU and was also limited to 3 Replicas and 1 Partition. The Atlas team deemed this sufficient for the Search Service workloads required for Atlas at the time.
With the advent of Atlas AI and all the additional capabilities provided by the service, this partition size and the inability to add additional partitions can prove to be a limitation, specially with the added usage of AI in your environment. The 2GB size limit applied to the AI Search Service Partition may run out of free space and cause Atlas to stop working. All activities that require the use of of Azure AI Search Service like Provisioning Workspaces will fail.
Error message you may encounter in this situation:
Storage quota has been exceeded for this service. You must either delete documents first, or use a higher SKU for additional quota.
Accessing the Indexes under Search management in the Atlas Search Service will show all your environment Indexes, the Total Storage Size should not add up to over 2GB (if Search Service was created before April 3rd 2024).
Normally the atlas-memory-index will be the Index with highest consumption. If we click on the Index Name we can see the total storage for this. Below image we can see that this total doesn't exceed the 2GB limit.
In the bottom screen shot, we can see that the 2GB has been exceeded.
To resolve this issue you will be required to Delete the Atlas Search Resource found in the Atlas Resource Group, and then Run the Atlas Deployment script that will re-create a new Atlas Search Resource. This new Search Resource will be created with a 15GB size (instead of 2GB) and can have 3 partitions configured given a total maximum size of 45GB if necessary.
Before doing this you will have to verify if any of your current Workspaces in ConneX are in a non-provisioned state. This can be done selecting under Provisioning Status any status that is not Provisioned.
If any of the Non Provisioned Workspaces are important, you will have to correct the provisioning errors for these workspaces before deleting the Atlas Search Service and creating the New Resource. Because the Provisioning Process requires you have free space in the existing Atlas Search Service, you will delete the atlas-memory-index that will more than likely be what is consuming most of your Atlas Search Service Space. Click the delete button.
Once deleted, you will be required to Review all the Workspaces in an Non-Provisioned status and try to correct any issue and put these in a Provisioned state.
NOTE: If the workspaces in a Non-Provisioned state are not being used or not important, these can be deleted.
Before deleting the Atlas Search Service Resource, we recommend that you try to create a new Search Service Resource in the same region as the current Atlas Search Service. This can be done manually from the Azure Portal. Microsoft has recently indicated that some regions and Search Service SKU´s are at capacify and can´t be used for new Search Services. Choose a service tier - Azure AI Search | Microsoft Learn
If you are able to create a Search Service in the same region as your current Atlas Search Service, it is OK to proceed with the next steps. Firstly delete the Test Search Service Resource you just created.
Stop all API and All Function Apps in the Atlas Resource Group.
Click on each resource one by one and once the resource pane is open, click on Stop.
From the Atlas Resource Group locate the Atlas Search Service, click on it and from the Overview section, click the Delete button.
Locate the Atlas Deployment Files folder for the current version of Atlas (Version you currently have deployed). This folder should contain the last successfully deployed Atlas Update scripts that you ran in your environment. You will now access the following path in the deploymen files folder: Deployment Files\Standalone deployments\v5_3. Note that in the below image this is the 5.3 version of the atlas deployment files and the Variable_Group.csv file is located where the Deploy-Release_v5_3.ps1 in the folder.
Open the Deploy-Release_v5_3.ps1 file in your favorite editor. You will need to locate the Task: Renew Azure Resource Keys section in this .ps1 file. In the below screen shot we see this is at Lines 169 to 174 (but this may vary depending on the Version of the Atlas Script).
You will need to comment this block of code by adding # symbol at the beginning of each of the lines in the above section of code. Once this code has been commented, save the changes to the Edited file. You may now close the editor.
You will now run the Deployment Script for this version of Atlas. You can open the corresponding Atlas Runbook to identity the script that needs to be executed. Below image shows that the file being called in the deployment script is the Deploy-Release_v5_3.ps1 edited previously.
NOTE: The Pre-Deployment and Post-Deployment Steps in the Runbook will NOT be executed. This is because you have run this script prior when Installing or Updating to this version of Atlas, and there is no need to re-run these other steps for this.
This deployment script will take about 1 hour to run, and during its execution, it will create a new Atlas Search Service Resource. Once the script has completed its execution you can go to the Atlas Resource Group, locate the Atlas Search Service. Click on the resource and click on Scale under Settings. You will firstly notice that the new Atlas Search Service should have 15GB instead of 2GB of the old one. You will also see that the partitions can be increased from 1 to 3 (if desired). This section in the 2GB sized resource was grayed out and not modifiable.
Atlas version 5.4 or less
This section can be carried out on all versions of Atlas version 5.4 or lower.
Once the deployment is completed, you can run a Atlas AI Full Sync. This can be done by anybody that is an Atlas AI Administrator. Click on the My Atlas Icon in the top bar - then Click on Settings.
Click on the AI Assistant.
Scroll to the bottom to select the Full Sync button and click Apply.
The sync will take some time to complete, depending on the Knowledge Collection Document Libraries and the number of documents within. Other Indexes like workspaces and workspaces-users will also take a while to sync up to the New Search Service.
Atlas Version 6.0 or higher
This section can be carried out on all versions of Atlas version 5.4 or lower.
Once the deployment is completed, you can run a Atlas AI Full Sync. This can be done by anybody that is an Atlas AI Administrator. Click on the My Atlas Icon in the top bar - then Click on Settings. Click on Intelligent Knowledge Studio.
For each Knowledge Collection you may have created with Knowledge Sources, click on the 3 vertical dots and click on Full Sync in the context menu.
The sync will take some time to complete, depending on the Knowledge Collection Document Libraries within the Knowledge Collections and the number of documents within. Other Indexes like workspaces and workspaces-users will also take a while to sync up to the New Search Service.
Comments
0 comments
Please sign in to leave a comment.