The Struggle of Managing Mail-Enabled Groups With Azure Functions

There’s a clear lack of love for the Exchange group and the Graph API, particularly when it comes to managing mail-enabled groups. Currently, the Graph API can only handle simple read commands for mail-enabled groups, which is a pain when dealing with real-world scenarios.

Let me explain.

When you create a user on-premise that syncs into Entra ID (or even a cloud-only user), something happens behind the scenes that synchronizes the user from Entra ID into Office. I’m not entirely sure what that process entails, but it directly impacts managing users with Exchange. The user doesn’t exist in Exchange until this sync completes. If you attempt to run a Get-User command on the newly created user before this sync finishes, you’ll encounter the following error:

error:Ex6F9304|Microsoft.Exchange.Configuration.Tasks.ManagementObjectNotFoundException|The operation couldn't be performed because object '<upn>' couldn't be found on '<serverName>.PROD.OUTLOOK.COM timestamp: <timestamp>

Most of the time, the sync finishes within seconds after creating a user. However, in some cases, it can take hours. Microsoft support informed me that it could take up to 24 hours for the sync to complete — a frustrating revelation. I was aiming for real-time automation, where users would be added to the correct mail-enabled groups as soon as they’re created. But since the Graph API doesn’t support user management in this context, I had two options: either use the portal or the Exchange Online PowerShell module.

Since this process needed to be automated, I chose to use the Exchange Online PowerShell module, setting up an Azure PowerShell Function. Initially, I considered adding a loop to check periodically if the user existed, but running into the Azure Function consumption plan’s HTTP timeout (230 seconds) quickly proved this approach impractical.

Next, I explored Azure Durable Functions. My thinking was that I could use async polling to check when the user was successfully created, then proceed to add them to the mail-enabled group. This seemed like a solid solution to the sync timing issue. However, another obstacle emerged: the durable function would throw an exception if the user didn’t exist, leading to additional retries. With multiple retries, I noticed the function running out of disk space.

It turns out that each time the durable function executed, it loaded the Exchange PowerShell module into a temporary directory. This caused the function to quickly run out of disk space, but I couldn’t directly view or manage the temporary storage because it’s handled by Microsoft. After some digging, I found a set limit on temporary storage for Azure Functions, which is detailed https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale

I needed to track my temporary storage usage, so I headed to the “Diagnose and Solve Problems” section in my App Service blade, and under “Best Practices,” I found the “Temp File Usage On Workers” option. Sure enough, I could see a spike in usage, which caused the out-of-disk-space exception. At this point, I could switch to an app service plan, but that would negate the cost savings of using a consumption plan. My tests showed that the temporary usage leveled off around 1.5 GB.

The above picture shows the temp usage leveling off with a basic app service plan. I proved it could be done, but I want the cheapest path possible.

So, back to the drawing board. The most cost-effective solution I came up with was returning to a regular Azure Function and utilizing queues. I’d drop a message onto a queue and periodically check if the user existed. Once confirmed, I’d add them to the group, place a message on another queue for further processing, and notify the user.

Azure Hybrid Worker + PowerShell 7.2

I encountered a scenario where I needed to use PowerShell 7’s foreach parallel feature to significantly reduce processing time. However, my hybrid worker VMs did not have PowerShell 7 installed. I installed the latest version, 7.2, and started the runbook worker with a PS7 job, but it just hung in the queued state. Time to investigate…

Since we’re all using extension-based hybrid workers (as the agent is being phased out), you need a system variable called POWERSHELL_7_2_PATH set to the location of your pwsh.exe. I verified that this variable was set correctly.

Looking at the job logs on the hybrid worker VM, it showed the following error:

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2024-07-18T21:05:45.3704708Z]  An unhandled exception was encountered while handling the job action.  The sandbox will terminate immediately. [jobId=54d069c4-a986-4e59-a22f-effd94a83c5b][source=Queue][exceptionMessage=System.InvalidOperationException: Runbook type '17' not supported.

Runbook type ’17’ not supported? There was no information on the Microsoft site about requirements beyond setting the environment variable. Their troubleshooting page also lacked relevant information. When I ran a PS7 command on the VM, it worked perfectly, but the job still hung in the hybrid worker. This suggested an issue with the extension. My current extension version was 1.1.11.

I wished for a changelog, but alas, that would be too easy. To list the available versions of the extension, run the following command:

az vm extension image list-versions --publisher 'Microsoft.Azure.Automation.HybridWorker' --location <region> --name 'HybridWorkerForWindows'

This showed versions 1.1.12 and 1.1.13 available. But why didn’t the 1.1.11 extension auto-upgrade? Odd. The following command upgrades the extension:

Set-AzVMExtension -ResourceGroupName <rg> -Location <region> -VMName <vmName> -Name "HybridWorkerExtension" -Publisher "Microsoft.Azure.Automation.HybridWorker" -ExtensionType HybridWorkerForWindows -TypeHandlerVersion 1.1 -Settings $settings -EnableAutomaticUpgrade $true

Note that the documentation mentions a parameter called -Settings, but doesn’t specify what it should be. I omitted it, and the command still upgraded my extension to the latest version.

After the upgrade, my PS7 job worked flawlessly. Cheers!