Tag: azure government

We began noticing an odd issue in our Microsoft Teams environment: newly created Teams were assigning the correct owner on the group, but this owner was not showing up in the Teams client. From a backend perspective, everything looked correct in EntraID, but when users opened the Teams app, the owner list was empty.

This wasn’t just a cosmetic bug, it affected team functionality, governance workflows, and user trust.

We opened a case with Microsoft and it was a known bug with no ETA on a fix. Bummer.
Microsoft recommended the following as a workaround:

1. Remove the owner from the group.
2. Add them back as an owner.

This action effectively “refreshed” the team and made the ownership visible in the Teams client. However, this method was:

1. Manual
2. Time consuming
3. Prone to human error
4. Not scalable for a large enterprise

We needed a robust, scalable, automated fix.

To solve this at scale, I leveraged Microsoft Graph Change Notifications (webhooks) combined with Azure Functions and Graph API. Here’s how it works:

• I subscribed to change notifications on Teams group resources.
• When a new team is created, a notification is sent to our webhook.
• My function then:
  1. Validates and decrypts the Graph notification.
  2. Retrieves the current owners.
  3. Adds a temporary owner to force a group ownership update.
  4. Removes and then re-adds the original owner(s).
  5. Removes the temporary owner.

This process resyncs the ownership so that it reflects properly in the Teams client.

To get started, we need to create an application and log in with that to create the permissions. We cannot do it with delegated auth.

connect-mggraph -Environment USGov  -ApplicationId "appidhere"  -CertificateThumbprint 'logginginwiththumbprint' -TenantId tenantidhere
$cert = New-SelfSignedCertificate `
  -Subject            "CN=GraphWebhookCert" `
  -CertStoreLocation  "Cert:\CurrentUser\My" `
  -KeyExportPolicy    Exportable `
  -KeyUsage           KeyEncipherment, DataEncipherment `
  -KeyAlgorithm       RSA `
  -KeyLength          2048 `
  -NotAfter           (Get-Date).AddYears(1)

# 2) Export the public key to a .cer file
$publicPath = "c:\temp\GraphWebhookCert.cer"
Export-Certificate `
  -Cert   $cert `
  -FilePath $publicPath

# 3) Export the private key to a PFX (for local testing / Key Vault)
$pfxPath  = "c:\temp\GraphWebhookCert.pfx"
$pfxPwd   = ConvertTo-SecureString -String "MySecurePasswordHere" -Force -AsPlainText
Export-PfxCertificate `
  -Cert     $cert `
  -FilePath $pfxPath `
  -Password $pfxPwd

# 4) Base64-encode the public .cer for inclusion in New-MgSubscription
$rawBytes = [IO.File]::ReadAllBytes($publicPath)
$base64Cert = [Convert]::ToBase64String($rawBytes)

# 5) Grab the thumbprint you’ll use as encryptionCertificateId
$thumbprint = $cert.Thumbprint

Write-Host "Public cert (base64):`n$base64Cert"
Write-Host "`nEncryptionCertificateId (thumbprint): $thumbprint"

$certId = '9f0345e7-bb60-4240-a4cb-0936dbc57bae' #(new-guid).Guid
$clientState = 'dcde8a93-acd1-4154-9a5b-2e83e63aa49f' #(new-guid).Guid

# Prepare your parameters
$notificationUrl         = "https://notificationFunctionUrl.azurewebsites.us/api/teams?code=codehere"
$lifecycleNotificationUrl = "https://notificationFunctionUrl.azurewebsites.us/api/lifecyclenotifications?code=codehere"
$resource                = "/teams"
$expiration              = (Get-Date).ToUniversalTime().AddDays(3).ToString("yyyy-MM-ddThh:mm:ssZ")
$certBytes               = [System.IO.File]::ReadAllBytes("C:\temp\GraphWebhookCert.cer")
$base64Cert              = [Convert]::ToBase64String($certBytes)
$certId                  = $certId
$clientState             = $clientState

# Create the subscription
New-MgSubscription `
  -ChangeType created `
  -NotificationUrl           $notificationUrl `
  -LifecycleNotificationUrl  $lifecycleNotificationUrl `
  -Resource                  $resource `
  -IncludeResourceData       `
  -EncryptionCertificate     $base64Cert `
  -EncryptionCertificateId   $certId `
  -ExpirationDateTime        $expiration `
  -ClientState               $clientState

Login-AzAccount -Environment AzureUSGovernment
$MsiName = "functionNameForMSI" # Name of system-assigned or user-assigned managed service identity. (System-assigned use same name as resource).

$oPermissions = @(
  "GroupMember.ReadWrite.All"
  "Group.ReadWrite.All"
)

$GraphAppId = "00000003-0000-0000-c000-000000000000" # Don't change this.

$oMsi = Get-AzADServicePrincipal -Filter "displayName eq '$MsiName'"
$oGraphSpn = Get-AzADServicePrincipal -Filter "appId eq '$GraphAppId'"

$oAppRole = $oGraphSpn.AppRole | Where-Object {($_.Value -in $oPermissions) -and ($_.AllowedMemberType -contains "Application")}

Connect-MgGraph -Environment USGov

foreach($AppRole in $oAppRole)
{
  $oAppRoleAssignment = @{
    "PrincipalId" = $oMSI.Id
    "ResourceId" = $oGraphSpn.Id
    "AppRoleId" = $AppRole.Id
  }
  
  New-MgServicePrincipalAppRoleAssignment `
    -ServicePrincipalId $oAppRoleAssignment.PrincipalId `
    -BodyParameter $oAppRoleAssignment `
    -Verbose
}

I initially tried creating a function with PowerShell code to handle this, but something with the runtime function would not let me decrypt with the certificate. Once I switched to C#, it all worked in Azure Gov.

A couple of things to note. EncryptionCertificateId is a string I created to check to make sure it is a valid certificate coming in. This certificate is used to decrypt the Teams resource data payload to get details. Always verify the authenticity of change notifications before processing them. This prevents your app from triggering incorrect business logic by using fake notifications from third parties. Also, ClientState is a secret value that must match the value originally submitted with the subscription creation request. If there’s a mismatch, don’t consider the change notification as valid. It’s possible that the change notification isn’t originated from Microsoft Graph and may have been sent by a rogue actor.

// This function handles both decryption and lifecycle events, including re-creating subscriptions.
// It is specifically configured for the Azure Government cloud.
// 1. Add NuGet Packages: Microsoft.Graph, Azure.Identity
// 2. Enable System-Assigned Managed Identity on the Function App.
// 3. Grant Managed Identity API Permissions
// 4. Set App Settings: 'WEBSITE_LOAD_CERTIFICATES', 'GraphNotificationCertThumbprint', 
//    'TempOwnerUserId', 'ExpectedCertificateId', 'NotificationUrl', 'LifecycleNotificationUrl',
//    'SubscriptionClientState', 'SubscriptionChangeTypes'.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using Azure.Identity;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using Microsoft.Graph;
using Microsoft.Graph.Models;
using Microsoft.Graph.Models.ODataErrors;
using Newtonsoft.Json;

namespace Company.Function;

#region Data Models for Resource Notifications
public class GraphNotificationPayload
{
    [JsonProperty("value")]
    public List<Notification> Value { get; set; }

    [JsonProperty("validationTokens")]
    public List<string> ValidationTokens { get; set; }
}

public class Notification
{
    [JsonProperty("subscriptionId")]
    public string SubscriptionId { get; set; }
    
    [JsonProperty("changeType")]
    public string ChangeType { get; set; }
    
    [JsonProperty("clientState")]
    public string ClientState { get; set; }
    
    [JsonProperty("subscriptionExpirationDateTime")]
    public DateTimeOffset SubscriptionExpirationDateTime { get; set; }

    [JsonProperty("resource")]
    public string Resource { get; set; }

    [JsonProperty("resourceData")]
    public ResourceDataObject ResourceData { get; set; }

    [JsonProperty("encryptedContent")]
    public EncryptedContent EncryptedContent { get; set; }

    [JsonProperty("tenantId")]
    public string TenantId { get; set; }
}

public class ResourceDataObject
{
    [JsonProperty("id")]
    public string Id { get; set; }

    [JsonProperty("@odata.type")]
    public string ODataType { get; set; }

    [JsonProperty("@odata.id")]
    public string ODataId { get; set; }
}

public class EncryptedContent
{
    [JsonProperty("data")]
    public string Data { get; set; }

    [JsonProperty("dataSignature")]
    public string DataSignature { get; set; }

    [JsonProperty("dataKey")]
    public string DataKey { get; set; }

    [JsonProperty("encryptionCertificateId")]
    public string EncryptionCertificateId { get; set; }

    [JsonProperty("encryptionCertificateThumbprint")]
    public string EncryptionCertificateThumbprint { get; set; }
}
#endregion

#region Data Models for Lifecycle Notifications
public class LifecycleNotificationPayload
{
    [JsonProperty("value")]
    public List<LifecycleNotificationItem> Value { get; set; }
}

public class LifecycleNotificationItem
{
    [JsonProperty("lifecycleEvent")]
    public string LifecycleEvent { get; set; }
    
    [JsonProperty("subscriptionId")]
    public string SubscriptionId { get; set; }

    [JsonProperty("resource")]
    public string Resource { get; set; }
    
    [JsonProperty("subscriptionExpirationDateTime")]
    public DateTimeOffset SubscriptionExpirationDateTime { get; set; }
}
#endregion

/// <summary>
/// This function handles the primary change notifications containing encrypted resource data.
/// </summary>
public class teams
{
    private readonly ILogger<teams> _logger;

    public teams(ILogger<teams> logger)
    {
        _logger = logger;
    }

    [Function("teams")]
    public async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequest req)
    {
        _logger.LogInformation("C# HTTP trigger function 'teams' processed a request.");

        string validationToken = req.Query["validationToken"];
        if (!string.IsNullOrEmpty(validationToken))
        {
            _logger.LogInformation($"'teams' validation token received: {validationToken}");
            return new ContentResult { Content = validationToken, ContentType = "text/plain", StatusCode = 200 };
        }

        _logger.LogInformation("'teams' received a new resource data notification.");
        try
        {
            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            var notifications = JsonConvert.DeserializeObject<GraphNotificationPayload>(requestBody);

            foreach (var notification in notifications.Value)
            {
                _logger.LogInformation($"Processing notification for resource: {notification.Resource}. ClientState: {notification.ClientState}");
                var expectedClientState = Environment.GetEnvironmentVariable("SubscriptionClientState");
                if (!string.Equals(notification.ClientState, expectedClientState, StringComparison.OrdinalIgnoreCase))
                {
                    _logger.LogWarning($"ClientState mismatch. Expected: '{expectedClientState}', Received: '{notification.ClientState}'. Skipping notification.");
                    continue;
                }
                #region Decryption Logic
                var expectedCertId = Environment.GetEnvironmentVariable("ExpectedCertificateId");
                if (!string.IsNullOrEmpty(expectedCertId) && !notification.EncryptedContent.EncryptionCertificateId.Equals(expectedCertId, StringComparison.OrdinalIgnoreCase))
                {
                    _logger.LogError($"Certificate ID mismatch. Expected: '{expectedCertId}', Actual: '{notification.EncryptedContent.EncryptionCertificateId}'. Skipping.");
                    continue;
                }
                _logger.LogInformation("Certificate ID validation successful.");

                var certThumbprint = notification.EncryptedContent?.EncryptionCertificateThumbprint ?? Environment.GetEnvironmentVariable("GraphNotificationCertThumbprint");
                if (string.IsNullOrEmpty(certThumbprint))
                {
                    _logger.LogError("Certificate thumbprint not found in payload or app settings.");
                    return new StatusCodeResult(500);
                }
                
                X509Certificate2 certificate;
                using (var store = new X509Store(StoreName.My, StoreLocation.CurrentUser))
                {
                    store.Open(OpenFlags.ReadOnly);
                    var certs = store.Certificates.Find(X509FindType.FindByThumbprint, certThumbprint, false);
                    certificate = certs.Count > 0 ? certs[0] : null;
                }

                if (certificate == null)
                {
                    _logger.LogError($"Certificate with thumbprint '{certThumbprint}' not found.");
                    return new StatusCodeResult(500);
                }

                using RSA rsa = certificate.GetRSAPrivateKey();
                byte[] decryptedSymmetricKey = rsa.Decrypt(Convert.FromBase64String(notification.EncryptedContent.DataKey), RSAEncryptionPadding.OaepSHA1);

                byte[] encryptedPayload = Convert.FromBase64String(notification.EncryptedContent.Data);
                byte[] expectedSignature = Convert.FromBase64String(notification.EncryptedContent.DataSignature);
                
                using (var hmac = new HMACSHA256(decryptedSymmetricKey))
                {
                    if (!hmac.ComputeHash(encryptedPayload).SequenceEqual(expectedSignature))
                    {
                        _logger.LogError("Signature validation failed.");
                        continue; 
                    }
                }
                _logger.LogInformation("Signature validation successful.");

                using Aes aesProvider = Aes.Create();
                aesProvider.Key = decryptedSymmetricKey;
                aesProvider.Padding = PaddingMode.PKCS7;
                aesProvider.Mode = CipherMode.CBC;

                byte[] iv = new byte[16];
                Array.Copy(decryptedSymmetricKey, 0, iv, 0, 16);
                aesProvider.IV = iv;

                string decryptedResourceData;
                using (var decryptor = aesProvider.CreateDecryptor())
                using (var msDecrypt = new MemoryStream(encryptedPayload))
                using (var csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read))
                using (var srDecrypt = new StreamReader(csDecrypt))
                {
                    decryptedResourceData = await srDecrypt.ReadToEndAsync();
                }
                
                _logger.LogInformation($"Successfully decrypted payload: {decryptedResourceData}");
                #endregion

                #region Graph API Call to Rotate Team Owners
                var match = Regex.Match(notification.Resource, @"\('([^']+)'\)");
                if (!match.Success)
                {
                    _logger.LogWarning($"Could not parse Team ID from resource: {notification.Resource}");
                    continue;
                }
                var teamId = match.Groups[1].Value;

                var options = new DefaultAzureCredentialOptions { AuthorityHost = AzureAuthorityHosts.AzureGovernment };
                var credential = new DefaultAzureCredential(options);
                var scopes = new[] { "https://graph.microsoft.us/.default" };
                
                var graphClient = new GraphServiceClient(credential, scopes, "https://graph.microsoft.us/v1.0");
                _logger.LogInformation("GraphServiceClient created and configured for Azure Government.");
                
                // 1. Get original owners with retry logic for replication delay 
                _logger.LogInformation($"Fetching owners for Team ID: {teamId}");
                DirectoryObjectCollectionResponse originalOwners = null;
                bool success = false;
                int maxRetries = 3;
                int delaySeconds = 15;

                for (int i = 0; i < maxRetries; i++)
                {
                    try
                    {
                        originalOwners = await graphClient.Groups[teamId].Owners.GetAsync();
                        _logger.LogInformation($"Successfully fetched owners for Team ID: {teamId}");
                        success = true;
                        break; // Exit loop on success
                    }
                    catch (ODataError odataError) when (odataError.Error?.Code == "Request_ResourceNotFound")
                    {
                        _logger.LogWarning($"Attempt {i + 1} of {maxRetries}: Team '{teamId}' not found yet due to potential replication delay. Retrying in {delaySeconds}s...");
                        if (i < maxRetries - 1)
                        {
                            await Task.Delay(delaySeconds * 1000);
                            delaySeconds *= 2; // Exponential backoff
                        }
                    }
                }
                
                if (!success)
                {
                    _logger.LogError($"Could not find Team '{teamId}' after {maxRetries} attempts. Aborting owner rotation for this notification.");
                    continue; // Skip to the next notification
                }
                
                // 2. Add new temporary owner 
                var tempOwnerUserId = Environment.GetEnvironmentVariable("TempOwnerUserId");
                if (string.IsNullOrEmpty(tempOwnerUserId))
                {
                    _logger.LogError("TempOwnerUserId app setting is not configured.");
                    continue;
                }

                _logger.LogInformation($"Attempting to add new owner: {tempOwnerUserId}");
                var newOwner = new ReferenceCreate { OdataId = $"https://graph.microsoft.us/v1.0/users/{tempOwnerUserId}" };
                try
                {
                    await graphClient.Groups[teamId].Owners.Ref.PostAsync(newOwner);
                    _logger.LogInformation("Successfully added new owner.");
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to add new owner. Full Exception: {ex.ToString()}");
                    continue; // Stop if we can't add the temp owner
                }

                // 3. Remove original owners 
                if (originalOwners?.Value?.Count > 0)
                {
                    _logger.LogInformation("Removing original owners...");
                    foreach (var owner in originalOwners.Value)
                    {
                        if (owner.Id.Equals(tempOwnerUserId, StringComparison.OrdinalIgnoreCase))
                        {
                            _logger.LogInformation($"Skipping removal of '{owner.Id}' as they are the new temporary owner.");
                            continue;
                        }
                        _logger.LogInformation($" Removing original owner: {owner.Id}");
                        await graphClient.Groups[teamId].Owners[owner.Id].Ref.DeleteAsync();
                    }
                    _logger.LogInformation("Finished removing original owners.");
                }
                
                // 4. Add original owners back 
                if (originalOwners?.Value?.Count > 0)
                {
                    _logger.LogInformation("Adding original owners back...");
                    foreach (var owner in originalOwners.Value)
                    {
                         if (owner.Id.Equals(tempOwnerUserId, StringComparison.OrdinalIgnoreCase)) continue;
                        
                        _logger.LogInformation($" Re-adding original owner: {owner.Id}");
                        var originalOwnerToAddBack = new ReferenceCreate { OdataId = $"https://graph.microsoft.us/v1.0/directoryObjects/{owner.Id}" };
                        try
                        {
                            await graphClient.Groups[teamId].Owners.Ref.PostAsync(originalOwnerToAddBack);
                        }
                        catch (Exception ex)
                        {
                            _logger.LogError(ex, $"Failed to re-add original owner '{owner.Id}'.");
                        }
                    }
                    _logger.LogInformation("Finished adding original owners back.");
                }

                // 5. Remove the temporary owner
                _logger.LogInformation($"Attempting to remove temporary owner: {tempOwnerUserId}");
                try
                {
                    await graphClient.Groups[teamId].Owners[tempOwnerUserId].Ref.DeleteAsync();
                    _logger.LogInformation("Successfully removed temporary owner.");
                }
                catch (Exception ex)
                {
                     _logger.LogError(ex, $"Failed to remove temporary owner '{tempOwnerUserId}'.");
                }

                #endregion
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex.ToString());
            return new StatusCodeResult(500);
        }

        return new StatusCodeResult(202);
    }
}


/// This function handles lifecycle notifications for the Graph subscription (e.g., reauthorization).
/// Its URL should be used for the 'lifecycleNotificationUrl' property when creating a subscription.
public class LifecycleNotifications
{
    private readonly ILogger<LifecycleNotifications> _logger;

    public LifecycleNotifications(ILogger<LifecycleNotifications> logger)
    {
        _logger = logger;
    }

    [Function("LifecycleNotifications")]
    public async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequest req)
    {
        _logger.LogInformation("C# HTTP trigger function 'LifecycleNotifications' processed a request.");

        string validationToken = req.Query["validationToken"];
        if (!string.IsNullOrEmpty(validationToken))
        {
            _logger.LogInformation($"'LifecycleNotifications' validation token received: {validationToken}");
            return new ContentResult { Content = validationToken, ContentType = "text/plain", StatusCode = 200 };
        }

        _logger.LogInformation("'LifecycleNotifications' received a new event.");
        try
        {
            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            var lifecycleNotifications = JsonConvert.DeserializeObject<LifecycleNotificationPayload>(requestBody);

            foreach(var notification in lifecycleNotifications.Value)
            {
                _logger.LogWarning($"Received lifecycle event: '{notification.LifecycleEvent}' for subscription '{notification.SubscriptionId}' on resource '{notification.Resource}'.");
                
                if (notification.LifecycleEvent.Equals("subscriptionRemoved", StringComparison.OrdinalIgnoreCase))
                {
                    await RecreateSubscription(notification);
                }
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex.ToString());
        }

        return new StatusCodeResult(202);
    }

    private async Task RecreateSubscription(LifecycleNotificationItem item)
    {
        _logger.LogInformation($"Attempting to re-create subscription for resource: {item.Resource}");
        try
        {
            var options = new DefaultAzureCredentialOptions { AuthorityHost = AzureAuthorityHosts.AzureGovernment };
            var credential = new DefaultAzureCredential(options);
            var scopes = new[] { "https://graph.microsoft.us/.default" };
            
            var graphClient = new GraphServiceClient(credential, scopes, "https://graph.microsoft.us/v1.0");

            var changeTypes = Environment.GetEnvironmentVariable("SubscriptionChangeTypes") ?? "created,updated,deleted";
            var notificationUrl = Environment.GetEnvironmentVariable("NotificationUrl");
            var lifecycleUrl = Environment.GetEnvironmentVariable("LifecycleNotificationUrl");
            var clientState = Environment.GetEnvironmentVariable("SubscriptionClientState") ?? Guid.NewGuid().ToString();
            var certId = Environment.GetEnvironmentVariable("ExpectedCertificateId");

            if (string.IsNullOrEmpty(notificationUrl) || string.IsNullOrEmpty(lifecycleUrl) || string.IsNullOrEmpty(certId))
            {
                _logger.LogError("Cannot re-create subscription. Required app settings (NotificationUrl, LifecycleNotificationUrl, ExpectedCertificateId) are missing.");
                return;
            }

            var newSubscription = new Subscription
            {
                Resource = item.Resource,
                ChangeType = changeTypes,
                NotificationUrl = notificationUrl,
                LifecycleNotificationUrl = lifecycleUrl,
                ExpirationDateTime = DateTimeOffset.UtcNow.AddHours(71), // ~3 days for Teams
                ClientState = clientState,
                EncryptionCertificateId = certId
            };

            var createdSubscription = await graphClient.Subscriptions.PostAsync(newSubscription);
            _logger.LogInformation($"Successfully re-created subscription. New ID: {createdSubscription.Id}, Expires: {createdSubscription.ExpirationDateTime}");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex.ToString());
        }
    }
}

I did put in a max retries on getting the owner. There is a slight delay from Teams provisioning and querying Entra ID. Other than that, it worked great. Microsoft did push the fix out, so this is no longer needed. Some times, you just need to think out of the box to keep the business moving forward!