Storage Account Increase Costs

This article describes a situation detected back in November 2024 on some Atlas Environments, where a substantial and abnormal increase in the Atlas Resource Costs were detected.  Customers affected by this would have seen that the Atlas Storage Account costs for November 2024 increased considerably. 

This articles offers a Temporary Fix to Mitigate this Cost Increase while Microsoft further investigates the issue and provides a Permanent Fix to this issue and the corresponding refund from Microsoft.

Check Atlas Resource Costs

Customers can verify if their Atlas Environment is affected by checking the Atlas Resource Group Costs for the end of 2024. 

NOTE:  We recommend that you review your monthly costs from September 2024 onwards, and verify that your Storage Costs haven't increased dramatically over this time.

• Open the Azure Portal and the Subscription where your Atlas Resource Group is located.  This resource group is normally identified by the following naming convention rg-atlas-xxxx-yyy (where xxxx is the client acronym and yyy is the environment key), although the name may not comply with this naming convention.
• From the Atlas Resource Group navigate to Cost Management and Cost Analysis.  You will then need to configure a view to show the Storage Account Costs for the end of 2024:
  
  Date:  Set a custom range from Beginning of September 2024 until Early 2025. (Our recommendation is to review until the current month)
  Resource Type: Add a filter to just show Storage Accounts.
  Group: by Resource.
  Granularity: Monthly
  Grouped columns.
  
  

If you see a considerable price increase from November onwards your Atlas Deployment is being affected by this issue.  If this price hike isn't visible from November onwards, we would recommend that you review until the current date, to verify that you are not being affected, as this may not affected you on November 2024 but at a later date.

The storage account affected by this issue is one named atlaswf1xxxxyyyy (wf1 in the middle of the name).

Our investigation has concluded that any Azure region can be affected, but we have seen this primarily in the UK South region.

The following article in GitHub mentioned this back in November 2024: Unexplained rise in storage activity causing higher costs · Issue #2981 · Azure/azure-functions-durable-extension
In this article Microsoft admitted that a bug was introduced on specific versions of:

• Microsoft.Azure.Functions.Worker.Extensions.DurableTask
• Microsoft.Azure.WebJobs.Extension.DurableTask

They also defined that this was affecting apps that are configured to use Runtime Scale Monitoring (which Atlas doesn't use).  In the ticket comments, customers affected by this issue were recommended to open a Microsoft Support Ticket and request Microsoft to refund the excess usage costs.  A patch was deployed by Microsoft to resolve this issue, but Atlas was not affected by this update and above average costs were still being registered in some Atlas Deployments in specific Azure Regions.

ClearPeople has Opened various Support Tickets on behalf of ourselves, as we have a few Atlas deployments affected by this issue, as well as to help customers that are in this same situation as us.  We also recommend that customers with this same issue, open support requests with Microsoft to address this.  

Although this issue has still not been resolved by Microsoft, we have been given a Workaround Temporary Fix that will at least reduce the excessive costs incurred until Microsoft deploys a Permanent Fix that resolves this issue.

Workaround - Temporarily Disabling Target-Based Scaling for Atlas

We have been given this recommendation by Microsoft Support as a temporary solution to reduce the excessive storage costs.  We have been testing over the past few days, and have not detected any adverse affects to Atlas's functionality.   Additional information can be found in the following link: Target-based scaling in Azure Functions | Microsoft Learn

IMPORTANT:  We recommend that you carry this out within a Scheduled Maintenance Window, since the Atlas Function Apps will required to be restarted after adding the Environment Variable.

To apply this to your Atlas Environment:

• Open the Atlas Resource Group from your Azure Portal, and from the Overview Pane list all the Atlas Function Apps.  There should a list of 9 Function Apps.
  
  
  
  
• Open each of the 9 Function apps by clicking on each one, and under Settings click Environment Variables.  Under App Settings click on Add to add a new Environment Variable to the existing list.  Enter the following values and click Apply.
  
  

  Name	Value
  TARGET_BASED_SCALING_ENABLED	0

  
  
  
  You will need to click Apply Again and Confirm.

IMPORTANT:  You will be required to restart the Function App from the Overview Pane, so the New Environment Variable change takes affect.

Do this for each of the 9 Atlas Function Apps in your Atlas Resource Group.

IMPORTANT:  These new Environment Variables will be lost if you Upgrade your Atlas Deployment, so these settings will have to be re-applied after each Upgrade as a Post Deployment Step.  ClearPeople will apply this new Environment Variable setting in our Upcoming Hot Fix Build in the Middle of April 2025.  This will affect only Atlas 6.0 Deployment Files.  Customers that are still on older versions of Atlas, that are affected by this issue, will have to make these changes manually for subsequent updates until they apply the Atlas 6.0 Update with the Hotfix.

Verification Checks 

To verify if the change has taken affect there are 2 checks we can make:

• After a few days of making this change you should be able to see that the daily cost for the Atlas Resource Group under Cost Management and Cost Analysis, you should see the daily cost has been reduced considerably.
  
  Date:  Set a custom range for a few days before the change and a few days after the change.
  Resource Type: Add a filter to just show Storage Accounts.
  Group: by Resource.
  Granularity: Daily
  Grouped columns.
  
  NOTE:  In the below graph you can see that the change was carried out on the 24th March which shows the blue bar at half the height to previous dates.  This is because of the abnormally high costs before the change on that day.  Subsequent days show the real 24 hour costs.

• A more immediate way of checking is to access the cost affected storage account atlaswf1xxxxyyyy (wf1 in the middle of the name) from withing the Atlas Resource Group.  Select it.
  
  Click on Metrics under Monitoring.  Configure the following Metric:
  
  Metric Namespace: Account
  Metric: Transactions
  Aggregation: Sum

After applying the Temporary Fix, you should see the Transaction Metrics go down from over 60K transactions to around 3K transactions.