I was testing a couple of conditional access policies against specific Office 365 cloud applications and came across a weird situation when doing a what-if analysis. Let’s take a simple scenario of blocking access to Exchange Online if the user is is not in our trusted IP list. Doing a what-if analysis, it works as expected when I test my account against that application.
Now, let’s setup a CAP targeting Teams and do a what-if analysis.
Both Exchange and Teams CAP will be applied, even though I am only targeting Teams. After digging more into this, there are service dependencies for conditional access based on the application being used. Looking at Teams, the user needs to satisfy access to SharePoint and Exchange before signing into Teams.
I was going absolutely nuts trying to figure out what I did wrong configuring this policy. In disbelief, I tried logging in with the user against the specific cloud app and sure enough, the TOS came up. I went back to the what if tool and it kept saying that the policy would not be applied. I thought maybe it was something to do with the TOS and switched it over to MFA in my CA policy. Same issue 😦 The only thing I thought of was that it had something to do with the group. I set the user in the group specifically on the CA policy and bingo, the what if tool worked perfectly.
I starting googling at github for this specific issue, but I could not find any. A quick CSS ticket with some emails back and forth has shown this is a bug and will be fixed, but no hard ETA other than this year. So, if you want to use the what if, make sure to assign the specific user and not depend on the group for your testing. I hope google indexes this page to save you the frustration and time wasted that happened to me 🙂