CSP and Delegated Admin security concerns: MFA is great, but what about the several outstanding Microsoft limitations?

[ziesemer]

Bringing-forward from https://www.yammer.com/office365partners/?show_login=true#/Threads/show?threadId=145300852449280...  (Why the multiple communities here?  Which is the go-forward platform here, Yammer or microsoftpartnercommunity.com ?)

How to support CSP customers while protecting the security of their tenant? Critical for security-conscious customers, and/or those with high-security requirements. Permissions granted by delegated admin are too far-reaching, do not allow for fine-grained access, and even the ability to audit use is unclear or non-existent.

This is something I had first started looking at just over a year ago. It doesn't look like anything has improved since, but customers are understandably only becoming increasingly concerned with this.

Here is some feedback we just received earlier today from a current customer looking to move to CSP licensing:

> Even with a NDA with <direct CSP provider> I am not at all comfortable making a third party of a third party delegated admins for our environment and granting them the ability to access all our cloud data which is what the terms state we’d be agreeing to. ... I really like the benefits of CSP otherwise so if there’s some other way to make this work we’d probably still do it. I also have reservations about a vendor having these all-encompassing permissions. I understand it can be needed for some admin tasks and troubleshooting but I don’t like how much of a free for all Microsoft makes it.

(I do believe there is some confusion in the above, in that our direct CSP provider / reseller would not have delegated admin - but only access to contact, subscription, and billing details. However, even this would be good to have further detailed in a format that we can share with concerned customers.)

If I were in the customer's place, I would absolutely be asking the exact same questions, and imposing the same requirements - and have so in prior organizations.

I have several other customers I've been working with, who I have not, in good faith, been able to recommend putting into CSP and the required delegated admin for - given the concerns outlined here.

We currently have a few hundred customers enrolled in CSP - and have an opportunity to bring even more into the CSP program, but the concerns outlined here do not help with the effort.

See also: https://docs.microsoft.com/en-us/azure/cloud-solution-provider/customer-management/administration-delegation

A number of specific points and concerns:

1) The list of roles and permissions that can be applied through Partner Center are available at https://docs.microsoft.com/en-us/partner-center/permissions-overview . Of these, only "Admin agent", "Helpdesk Agent", and maybe "Sales agent" apply to the customer tenants. "Admin agent" is basically the equivalent of "Global Administrator" within a customer tenant, and "Helpdesk Agent" is effectively a "Helpdesk Administrator" (Password Administrator). The admin roles from a tenant perspective are documented at https://docs.microsoft.com/en-us/office365/admin/add-users/about-admin-roles . As an O365 admin in my own organization, I can delegate permissions to others through at least 262,144 different combinations of the 18 customized administrator roles currently available in my tenant. (This increases to 38 roles available and nearly 275 billion combinations when using the roles available in Azure AD.) As a partner with delegated admin to a customer, there are only 2: "Everything", or "Helpdesk Agent" (or none). See also: https://www.yammer.com/office365partners/#/Threads/show?threadId=119473804279808 and https://www.yammer.com/office365partners/#/Threads/show?threadId=804328021 .

2) Everything is "all or nothing". We also have well over 100 engineers in our organization (not counting back office staff, etc.). To assign either "Admin agent" or "Helpdesk Agent" to one of our staff, means that they have that same permission across a few hundred customers. There is no way to filter a staff member's access to only one customer, or ideally a group of customers. (While we do have some additional flexibility in granting access through guest accounts in Azure AD, these permissions do not carry over to / are not usable in O365.)

3) We could consider falling-back to the use of extended and manually-reviewed audit logging as a compensating control. However, there are many limitations here - including that all logging appears to be held within each individual customer tenant only - and for us to review delegated admin activity from our partner account as a whole, it appears that we'd need to query each and every one of our customer tenants. Further concerns are raised here at https://www.yammer.com/office365partners/#/Threads/show?threadId=62199238885376 (unanswered).

While I'm on the topic, the "pretty wicked security hole" documented at https://office365.uservoice.com/forums/264636-general/suggestions/33233917-powershell-mfa-for-csp-delegated-admin-privileges also still needs to be addressed, well over a year later now. Can someone from Microsoft please review and respond to this UserVoice entry? Everyone else - please vote...

What are other CSP partners doing in terms of security controls to meet customer needs and expectations as it relates to CSP licensing? (We do have our own internal security program that I am a part of - which includes even stronger MFA controls across our entire organization than we'd otherwise been able to have approved, partially because of the customer security concerns here.)

As per https://blogs.technet.microsoft.com/uspartner_ts2team/2016/04/30/removing-the-csp-as-delegated-admin-in-office-365-customer-tenants/ , it looks like the simplest answer may to just be to have needing customers remove the delegated admin permissions. However: 1) Is this still accurate and working, 3 years later? And 2) we'll need to determine what limitations, if any, may then exist that would prohibit us from providing CSP licensing in this model, even if we then are not able to directly assign new licenses to users, or otherwise provide direct support to the customer as per CSP terms (at least not without having other customer-provided access given to us).

Just looking at this again now, it looks like we *might* be able to put the "Admin agent" and "Helpdesk Agent" roles behind the new "Azure AD entitlement management". This is less than 2 months old, still in preview, and what Privileged Identity Management really should have been all along, in my opinion. It does require Azure AD Premium P2 (or its parent EMS E5 offfering).

[VNJoe]

The overarching problem with this solution is that Microsoft forced users to enable MFA for ALL accounts on their primary tenant in order to retain our partnership with Microsoft, yet the workaround you propose completely bypasses those obligations in that we don't need to enable MFA for these newly created global admins on our tenants.  If we do enable it, we need one for every employee we have in order to use MFA, and keep the client protected.  Otherwise, we are simply creating unprotected global admin accounts on every tenant that are a significant vector of brute force attacks.

GDAP is fine if that's what the bigger customers of Microsoft want, but it doesn't solve anything for partners, doesn't make anything simpler or more secure, and adds significant levels of complexity to manage customers.

It's a poor concept.

The solution here IMHO is to make the advanced auditing and alerting features available in Azure part of every base package so the customers (and partners) can get the alerts they need to know when an account has been breached or logged in, and we can create automation around that.  Upcharging for securing the environment you've set up is pretty bold.  At least if alerts are available, we all can help keep O365 secure, no?

I was looking forward to participating in the GDAP program, but was disallowed because I'm an indirect reseller, like most people.  The only participants in the pilot now are likely larger orgs who don't see the SMB picture that was the foundation of your customer base.

Just some food for thought.

[simbur666]

Microsoft are working on more options for this - the partner portal has been coded with Admin / Helpdesk roles for years.  I suspect adding more granular permission options is a HUGE undertaking, so let's be patient.  In the meantime, you do have options - think outside the box:

1. Did you know more than one person can register MFA against a single account?  Yes they can!  So for say 10-20 staff they can all scan the same QR code, or register separately under the same account.  If you use IT Glue, it can even store the code and keeps rolling over as normal, so your support staff can use IT Glue rather than their phone to enter the code. 

2. For Global Admin tasks, support staff could use a jumphost from your office to perform the admin functions.  If it's a generic account, use conditional access to block the account from siging in- unless it is coming from the external IP associated with the jump host.  This is secure (but not really good to use generic accounts).

3.  Invite your support staff as guests to the client tenant (I'm sure you can whip up a csv with their email and names in it fairly quickly using export from your portal).  Create a group in customer tenant that has the required roles assigned.  Bulk add your support staff guest accounts to the group.  Create a conditional access policy to block the group from signing in, uinless using MFA or from the external IP of a jump host without MFA.

More info:

https://www.howdoiuseacomputer.com/index.php/2021/09/19/msp-csp-fine-grained-access-to-customer-tenants/

[JanoschUlmer]

@simbur666 : Indeed, changing the delegated admin model is a huge undertaking, and the solution - GDAP - is almost there - announced to be available in January: December 2021 announcements - Partner Center | Microsoft Docs

Thanks for providing some suggestions, I think those are valid approaches to think about. Only one small thing - I'm always a bit nervous when somebody configures an exception for essential things like MFA based on the network they access from. With a zero trust security strategy one should never assume the internal network is secure 🙂

[simbur666]

Thanks Janosch, I'm not sure what all the fuss is about really... from my perspective I am working on giving our service desk and managed services staff 'least-privilege' access to be able to perform day-to-day across our CSP tenants, and that is currently possible albeit with a few caveats as per my previous post. I also appreciate 'Rome was not built in a day', so it's taking some time for MS to develop a well-rounded solution. More importantly, any tasks our staff perform in a customer tenant should be recorded in the customers audit logs with their own name, not a generic account as you suggested above.  Zero trust does not allow for a generic admin account in a customer tenant.  BUT, in my opinion, to take it that next level and say 'you must always MFA, even from a trusted location' is getting a little bit OTT.  With that premise, how do I let a printer scan to email, or an unattended Teams app in a meeting room function, without manually enabling / disabling MFA per-user in the old MFA UI?  That comes with a cumbersome administrative overhead. And how about the whole point of Intune compliant devices? If you are nervous to trust a location, then you surely wouldn't trust a device by default, regardless of it's configuration. So then you are kind of dismissing a big part of the functionality MS is providing us, which is not to annoy users all the time with MFA prompts. Surely if an office building has card-based access for employees (or some other physical security for access to premises), we can exclude the MFA requirement from that location without much worry?  And if I am on my trusted company laptop in Guatemala, I should be able to exclude the MFA requirment based on my device being 'compliant' in Intune... or is MS looking to move away from these 'trusted' scenarios?  That would be a shame, as too many MFA prompts would no doubt stifle productivity with users not bothering to work because it's 'too hard' or 'I haven't got my phone'.  I really like where MS is heading with the 'no passwords' story, but to require a code from my phone when I'm at the office, or not at the office but on a managed machine / phone seems a bit counter intuitive to me.

Some great discussion in this thread! 🙂  

[JanoschUlmer]

@simbur666 : #

Reg. "With that premise, how do I let a printer scan to email, or an unattended Teams app in a meeting room function, without manually enabling / disabling MFA per-user in the old MFA UI?  "

Long term this old MFA admin UI will be gone, it is only relevant for basic authentication, and basic authentication will also go away long term. And luckily you don't need it configure exceptions - for AzureAD Security defaults they would not apply anyway, when you use Conditional Access you can confugire those in the CA policies.

Agree, currently there are those challenges, and not every device vendor will be able to solve them short-term. If such an exception is necessary, make it as specific as possible. Push device vendors to support modern authentications, for Teams rooms specifically, while it is modern auth there is special user account that indeed is not leveraging MFA. E.g. when using AzureAD Security defaults, MFA protection would only kick in when you attempt to use this account for interactive authentication. 

Reg. "And how about the whole point of Intune compliant devices? If you are nervous to trust a location, then you surely wouldn't trust a device by default, regardless of it's configuration."

I don't agree, there is a fundamental difference - with checking device configuration status you can check the device current health and not assume it is healthy just because it talks from a certain network location - e.g. integrating Device compliance policies with things like (Defender) Endpoint protection status. 

Reg. "which is not to annoy users all the time with MFA prompts."

With normal OAuth scenarios, number of MFA prompts is very low, since application usually only need to do MFA once and can then replay & renew tokens, unless configured with a more secure (paranoid ? 😉) setting to shorten the session lifetime. E.g. I need to do MFA for everything, as long as I can log on to Windows using Hello for Business I will hardly notice MFA at all, for most apps. Same on my mobile phone.

Reg. "Surely if an office building has card-based access for employees (or some other physical security for access to premises)," - Might be an option, if both physical side is secured and users have no way to install software ... or receive emails 😁 . Sure, you can provide high security using other measures as well, not saying that there is just one right option. However, for whatever you do, please "assume breach". Customers that do disable security measures just because they assume there can't be a compromised device within their internal network make attackers very happy.

I can fully understand that MFA can be a burden sometimes for users, but there are solution for this. Register more than one app, use app & hardware tokens etc.

[JanoschUlmer]

@VNJoe : Thank you for your honest feedback. Of course, there is no need to use GDAP, feel free to continue using admin accounts for each customer tenant, AzureAD Security Defaults are enabled for most customer tenants so they can get those alerts you are seeking for free and so non-MFA protected accounts are only available if you bypass the basic security recommendations.

I did not propose the "idea" of using per-customer user accounts as a better workaround, it seems you did misunderstood the context of why I did mention this as one option. And I also think that you misunderstand the concept of GDAP and overestimate the complexity, I'm talking daily to Partners working with SMB customers using the same concept with regard to Azure Lighthouse and are interested in GDAP for the same reason. Your decision of course.

[ITGDaryl]

You state that AzureAD Security Defaults are enabled for most tenants, that would be correct, however, it seems to be an all or nothing deal.

For example, if the client requires SMTP auth for a legacy product, the security defaults block this, but can you simply enable SMTP auth while retaining all the other security practices? oh no, you have to turn it all off and pay extra for that customizability just to achieve a similar level of security.

It seems entirely backwards from a security standpoint, I understand you prefer to use modern authentication and I'm with you, but many SMB clients that MSP's have to manage use legacy products or cheaper alternatives to major well renowned brands simply because it makes financial sense for them and we as MSP's are expected to keep things running smoothly where possible.

Could the ability to ONLY allow SMTP auth be an option with a warning? alternatively could we have the ability to setup conditional access rules etc to mimic the Security Defaults behaviour without having to pay for extra rules and Azure licensing in a standard package?

We are all for improving upon best security practices and putting them into place, but when those practices restrict day to day requirements for clients, most times they have to be "worked around" or worse, switched off entirely.

[JanoschUlmer]

@ITGDaryl : Valid feedback, having this option would certainly make those scenarios easier. The issue is, that investigations show that it is exactly those remaining legacy devices and legacy auth for those that are a known, and broadly used entry point for attackers. Pretty sure that AzureAD team said for this reason that they make no compromise because of this (Actually, there was an option as preview that allowed just that, remember "baseline policies"?  But his path was not pursued for a reason). And reg. SMTP AUTH - It might that Exchange Online changes will hit those customers in this regard anyway sooner or later. There is no date for when we announced to disable SMTH Auth via Basic Auth, but it will happen at some point in time. It is now already disabled by default: Basic Authentication and Exchange Online – September 2021 Update - Microsoft Tech Community and for SMTP Auth options see Basic Authentication and Exchange Online – July Update - Microsoft Tech Community

I think, having AzureAD Security Defaults is a huge step forward, it delivers AAD P2 features for free, where before there were no no-cost options at all. 

[ITGDaryl]

Agreed, ideally it should not be used, however, we can only advise clients or refuse to offer support, we cannot make those decisions for them when they choose to use legacy software.

We have a compromise, turning it off, but in turn it disables all the other default policies such as enforcing MFA for admins and various other beneficial measures all for the sake of being able to use SMTP authentication to send an email from an application, short of setting up some relay as you say (assuming it's financially viable for the client) it becomes a little problematic as we are making the sites less secure for the sake of the client not wanting to spend the money to have something more modern and compatible with the latest security standards.

I just wish that there was an ability to keep the other beneficial security features intact while this was disabled but it seems that will not be possible, we will push to get them updated as always given that support will soon be ending completely.

[simbur666]

I think the main problem with security defaults is that Microsoft allows people to turn them off!  Usually for this very reason, thereby opening themselves up for attack.

In the case you describe, it makes sense to have one WIndows server onsite, even if it is installed on a NUC or PC tucked away in the broom cupboard.  Then you can use it as an SMTP relay (and for any other things like wireless controller or security camera software).  Connect an EOL connector for the relay (which is allowed).  Then your client  is still secure and can send SMTP from multiple devices.

Then again you should really convince them to get AAD P1 ($8/user), then you have conditional access and can enable SMTP for one user as you describe without requiring the relay.

Pretty reasonable pricing for the functionality you are requesting (IMO).

[simbur666]

I think the main problem with security defaults is that Microsoft allows
people to turn them off! Usually for this very reason, thereby opening
themselves up for attack.

In the case you describe, it makes sense to have one WIndows server onsite,
even if it is installed on a NUC or PC tucked away in the broom cupboard.
Then you can use it as an SMTP relay (and for any other things like
wireless controller or security camera software). Connect an EOL connector
for the relay (which is allowed). Then your client is still secure
and can send SMTP from multiple devices.

Then again you should really convince them to get AAD P1 ($8/user), then
you have conditional access and can enable SMTP for one user as you
describe without requiring the relay.

Pretty reasonable pricing for the functionality you are requesting (IMO).

[VNJoe]

And a final note:  Partners are going to continue what you mention, use a global admin account on our customer's tenants instead of attempting to configure 15 roles on hundreds of clients...  bypassing this entire process. I haven't used my partner account in years because of how useless it's become.

[Leho]

Again - I really can't understand why there is still no soluton or even notes about possible solutions? 2 years have passed from initial post. Sorry but MS finds time and resources for pointless product rebranding projects but not for critical issue.

And for suggested workaround - I would be more than happy to ask MS to explain to customers why they have to pay for partners' users if they will not grant admin rights for partner.

[Leho]

Sorry, I really can't understand so numb attitude from Microsoft side regarding so serious topic. 

For me as Dynamics partner current topic begins to be obstacle to take customers to cloud. But that is what Microsoft wants and where he expects partners should focus. 

From customer perspective:

I always have to explain any bigger customer why they need to grant us permissions what in theory allow is to do almost everything. Half of them accept that and half not. For example no international companies grant us such permissions cause they have higher security standards and don't let us see their whole AD. So we have to find alternative solutions. 

From Dynamics partner perspective:

We have 60 employees who are serving Dynamics customers on everyday basis. They need to access customer environments and it's possible through Partner Central where are listed all our Dynamics SaaS customers - some of them smaller, some bigger, some members of the exchange etc. And there is no limitation where our people can access. I hope MS understands to what kind of data actually all these people can access and what can be the worst scenario if somebody has bad intensions. Or is just curious. It's highly confidential data starting from financial statements till sales companies pricing information or manufacturing companies self-cost formulas.

OK, the alternative is that Dynamics partner has no delegated rights and will be defined in ERP as ordinary user but then the partner's user is not free of charge anymore. Majority of customer can't understand why they have to pay for partner's users to be able get consultation service.

So I really hope that this topic finds at least some kind of solution asap and keeps improving continuously.

[simbur666]

I've come across a few companies deploying Business Central or Dynamics with our customers that have demanded 'Global Admin' permissions that I had intentially removed from the customer tenant after they had accepted the CSP invitation.  Most had no understanding of zero-trust, or the rights they required to do their bit, or that giving them access to create users, groups or anything else within the environment should be considered unacceptable if another company manages and is therefore (potentially legally) responsible should an incident occur.  Although I know it's 'frustrating' not to have god-level access to get something done, it is how the future will be so we best get used to it. Your customer should be inviting your staff as B2B 'guests', adding them to a group that has the 'Dynamics Admin' role (which is free and allows the initial set up and on-going administration).  Yes, you are correct that means they then need a licensed generic 'super user' account within the system, but is that really a big deal? You can pitch from the outset that you will need one licensed account in the system, or even include it in your proposal so that it is absorbed into the implementation costs.     

[JanoschUlmer]

@Leho : As mentioned in before discussion, this is already being worked on.

If you want to give feedback: Join the Partner Center API and SDK Early Adopter Program - Partner Center app developer | Microsoft Docs (not only applicable to API topics, this is where you will learn about upcoming improvements first and where you can give feedback to the PMs)

[JanoschUlmer]

@AlexCBrewer & All others: This might be of interest for you: https://techcommunity.microsoft.com/t5/small-and-medium-business-blog/announcing-microsoft-365-lighthouse-for-managed-service/ba-p/1698181

This might be the first step in removing some of the limitations and visible effort by the PGs to address specific MSP scenarios in a better way. There is also more to come specifically for CSP Partners, though it will take a while until something will be shared. 

[AlexCBrewer]

@JanoschUlmer Thanks for the mention.

I have seen Lighthouse and we have registered for access to the preview, although at this stage it looks like not much more than a reporting dashboard on the Intune device estate of our clients. I can't see anything, as yet, for AAD user reporting and nothing with regards to actual management on client tenants.

Whilst I do see the benefits Lighthouse will bring us, it does not solve the problems highlighted in this and other threads about how our delegated admin permissions do not give full global admin rights over client tenants which is critical for proper full MSP offerings

[JanoschUlmer]

As mentioned - "first step" 🙂  For Partners there are also changes on the roadmap.

[AlexCBrewer]

Hi @JanoschUlmer , is that roadmap public? To partners at least?

[JanoschUlmer]

For Direct CSP and Indirect Providers there is an early adopter program, but I can't currently tell if new Partner will be accepted. See this thread if you have access to the Cloud Partner Yammer network: https://www.yammer.com/cloudpartnercommunity/#/Threads/show?threadId=678930850922496