CSP and Delegated Admin security concerns: MFA is great, but what about the several outstanding Microsoft limitations?
Bringing-forward from https://www.yammer.com/office365partners/?show_login=true#/Threads/show?threadId=145300852449280... (Why the multiple communities here? Which is the go-forward platform here, Yammer or microsoftpartnercommunity.com ?)
How to support CSP customers while protecting the security of their tenant? Critical for security-conscious customers, and/or those with high-security requirements. Permissions granted by delegated admin are too far-reaching, do not allow for fine-grained access, and even the ability to audit use is unclear or non-existent.
This is something I had first started looking at just over a year ago. It doesn't look like anything has improved since, but customers are understandably only becoming increasingly concerned with this.
Here is some feedback we just received earlier today from a current customer looking to move to CSP licensing:
> Even with a NDA with <direct CSP provider> I am not at all comfortable making a third party of a third party delegated admins for our environment and granting them the ability to access all our cloud data which is what the terms state we’d be agreeing to. ... I really like the benefits of CSP otherwise so if there’s some other way to make this work we’d probably still do it. I also have reservations about a vendor having these all-encompassing permissions. I understand it can be needed for some admin tasks and troubleshooting but I don’t like how much of a free for all Microsoft makes it.
(I do believe there is some confusion in the above, in that our direct CSP provider / reseller would not have delegated admin - but only access to contact, subscription, and billing details. However, even this would be good to have further detailed in a format that we can share with concerned customers.)
If I were in the customer's place, I would absolutely be asking the exact same questions, and imposing the same requirements - and have so in prior organizations.
I have several other customers I've been working with, who I have not, in good faith, been able to recommend putting into CSP and the required delegated admin for - given the concerns outlined here.
We currently have a few hundred customers enrolled in CSP - and have an opportunity to bring even more into the CSP program, but the concerns outlined here do not help with the effort.
A number of specific points and concerns:
1) The list of roles and permissions that can be applied through Partner Center are available at https://docs.microsoft.com/en-us/partner-center/permissions-overview . Of these, only "Admin agent", "Helpdesk Agent", and maybe "Sales agent" apply to the customer tenants. "Admin agent" is basically the equivalent of "Global Administrator" within a customer tenant, and "Helpdesk Agent" is effectively a "Helpdesk Administrator" (Password Administrator). The admin roles from a tenant perspective are documented at https://docs.microsoft.com/en-us/office365/admin/add-users/about-admin-roles . As an O365 admin in my own organization, I can delegate permissions to others through at least 262,144 different combinations of the 18 customized administrator roles currently available in my tenant. (This increases to 38 roles available and nearly 275 billion combinations when using the roles available in Azure AD.) As a partner with delegated admin to a customer, there are only 2: "Everything", or "Helpdesk Agent" (or none). See also: https://www.yammer.com/office365partners/#/Threads/show?threadId=119473804279808 and https://www.yammer.com/office365partners/#/Threads/show?threadId=804328021 .
2) Everything is "all or nothing". We also have well over 100 engineers in our organization (not counting back office staff, etc.). To assign either "Admin agent" or "Helpdesk Agent" to one of our staff, means that they have that same permission across a few hundred customers. There is no way to filter a staff member's access to only one customer, or ideally a group of customers. (While we do have some additional flexibility in granting access through guest accounts in Azure AD, these permissions do not carry over to / are not usable in O365.)
3) We could consider falling-back to the use of extended and manually-reviewed audit logging as a compensating control. However, there are many limitations here - including that all logging appears to be held within each individual customer tenant only - and for us to review delegated admin activity from our partner account as a whole, it appears that we'd need to query each and every one of our customer tenants. Further concerns are raised here at https://www.yammer.com/office365partners/#/Threads/show?threadId=62199238885376 (unanswered).
While I'm on the topic, the "pretty wicked security hole" documented at https://office365.uservoice.com/forums/264636-general/suggestions/33233917-powershell-mfa-for-csp-delegated-admin-privileges also still needs to be addressed, well over a year later now. Can someone from Microsoft please review and respond to this UserVoice entry? Everyone else - please vote...
What are other CSP partners doing in terms of security controls to meet customer needs and expectations as it relates to CSP licensing? (We do have our own internal security program that I am a part of - which includes even stronger MFA controls across our entire organization than we'd otherwise been able to have approved, partially because of the customer security concerns here.)
As per https://blogs.technet.microsoft.com/uspartner_ts2team/2016/04/30/removing-the-csp-as-delegated-admin-in-office-365-customer-tenants/ , it looks like the simplest answer may to just be to have needing customers remove the delegated admin permissions. However: 1) Is this still accurate and working, 3 years later? And 2) we'll need to determine what limitations, if any, may then exist that would prohibit us from providing CSP licensing in this model, even if we then are not able to directly assign new licenses to users, or otherwise provide direct support to the customer as per CSP terms (at least not without having other customer-provided access given to us).
Just looking at this again now, it looks like we *might* be able to put the "Admin agent" and "Helpdesk Agent" roles behind the new "Azure AD entitlement management". This is less than 2 months old, still in preview, and what Privileged Identity Management really should have been all along, in my opinion. It does require Azure AD Premium P2 (or its parent EMS E5 offfering).
Do I understand correctly from the perspective of a CSP's customer, that there's currently no way to use CSP subscriptions without providing full admin rights to the whole subscription? I.e. without giving the CSP unrestricted access to all Azure resources in that subscription?
@zoltanlehoczky : That is not correct. Partner can provision licenses and Azure subscriptions without having permissions. However, with the CSP being the trusted service provider for the customer (at least, this is the idea) the question is if this would make sense.
- In order to sell something into a customer tenant, the Partner needs to set up a reseller relationship with the customer tenant. This reseller relationship can be set up with or without delegated admin permissions. However, in order to be able to open support tickets on behalf of the customer (since customer can not do this themselves), the Partner needs to have delegated admin - if not present, the Partner can just send the relationship invite again, this time with delegated admin. And customer can remove it later without impact on the billing of licenses. This is where the discussed new "GDAP" comes into play - currently delegated admin means full global admin, in the future this can be more restricted.
- When Partner provisions an Azure subscription, the Partner will by default always be added as "Owner" on the Azure subscription, even when no delegated admin is established. Without delegated admin, the Partner could not set/customize permission on the subscription, but control services. This "Foreign Principal" created automatically on the Azure subscription permissions can be removed and/or re-added using different set of permissions, e.g. support request contributor. Again, the Partner needs some form of access to be able to open support tickets.
However, for Azure the Partner also needs some form of access 24/7 to receive a margin (Partner Earned Credit) - it does not need to be owner though. Again, the idea of acting as CSP is that the CSP plays a crucial role in managing the customer environment, so some form of access for the specific Azure services the Partner manages should usually be acceptable. GDAP, when this goes GA;: will currently not change the Azure behavior, but this might be addressed as well at a later time..
There are more details, tips & tricks to discuss for specific customer scenarios - simply open an advisory request with my team: https://aka.ms/technicalservices to discuss with a Partner consultant.
Thanks for the detailed reply.
Let's suppose that the customer only wants a billing relationship, nothing else (no management of services, no support tickets, etc.). What's the way to do this?
The subscription created by the Partner will have them as an Owner of it, which as far as I understand gives them unrestricted access (see docs). This access can be removed after provisioning but you're saying "Partner also needs some form of access 24/7". What other form can this be, if not an Owner nor delegated admin? Or rather, what's the role with the minimal level of access that's sufficient? Simply adding a user of the Partner with the Support Request Contributor role would suffice?
@zoltanlehoczky : Yes, adding a user from the Partner tenant as B2B guest or adding a Foreign Principal - both with "Support Request Contributor" Azure RBAC - would work to be able to open support tickets as Partner, and become eligible for Partner Earned Credit.
However, having just the billing relationship somehow defeats the purpose of acting as CSP, if the customer has this ask there is some reason for doubt CSP is the right option for them - if they want to act independently from a Partner maybe they should buy directly from Microsoft instead - this way they would also have the option to raise support tickets themselves in case a problem occurs.
I agree with the need for granular levels of delegated roles/permissions, one thing I have not been able to figure out though, even with the current setup is if it is at all possible for a partner to be granted access to a customers sharepoint sites for management of document libraries etc.
While the original post is about limiting control (and I'm all for that as it's best practice) as a delegated global admin, I can access the sharepoint admin page but I cannot for example, get into their sharepoint site nor can I find any way for the customer to be able to add myself as a member/owner with their permission in order to be able to manage these things for them as a delegated administrator.
Anyone know of a way to do this? it would make our lives a lot easier if there is a way to enable this functionality such that when asked, we do not have to login as the clients tenant global admin account just to make some changes or check something for them and would rather be able to have not only access but granular control for the engineers as discussed in the OP.
@ITGDaryl With GDAP there is also a broader effort from the product teams to make delegated admin work in the respective admin portals (M365 Security & Compliance Center, Teams as a few examples). If have not heard about specific improvements for Sharepoint though - the main issue is that delegated admin do not exist in the customer tenant, and thus the Sharepoint sites can not identify the users.
You can still onboard the pilot, which would also allow you to give feedback: https://docs.microsoft.com/en-us/partner-center/announcements/2021-december#9
Should have updated this earlier, but here it is - GDAP preview now available:
While we wait for lighthouse to become a solution for this, we looked into other ways to achieve the goal. You can do most things using B2B guest access and client targeted URLs (appending the client tenant to the admin URL):
Azure - https://portal.azure.com/customer.com
Intune - https://endpoint.microsoft.com/customer.com
Security Center - https://security.microsoft.com/customer.com
Exchange - https://email@example.com
SharePoint - https://customer-admin.sharepoint.com/
Unfortunately this doesn't seem to apply for the 365 Admin Center, which is where you would want
Helpdesk staff to be performing User / Exchange tasks, rather than jumping between Azure AD and Exchange (but at least it works, and achieves the goal your client is after).
Here is the process:
1. customer accepts MSP invitation
2. customer removes the admin / helpdesk agent privileges from the partners area of the customers portal (this keeps the association and you can still procure licensing for them, but removes the default partner permissions)
3. create two or more groups in customers Azure AD; one for your Helpdesk and one for Admins (with assign roles to group enabled)
4. assign the roles to the Helpdesk group (change to fit your needs or use custom roles):
Exchange Recipient Administrator
5. assign roles to the Admin group as required from this list (you can use more admin groups to assign roles to different support groups if required):
Conditional Access Administrator
Cloud App Security Administrator
Azure AD Joined Device Local Administrator
Privileged Role Administrator
Azure Information Protection Administrator
6. assign Azure subscription roles to the Admin groups as required:
7. you can also use these groups to assign permissions to certain Azure objects, using the IAM blade under the resource
8. Use Bulk Invite in customers Azure AD users blade to invite all your support staff to the customer tenant as guests
9. Add the invited guest accounts to the groups you created
After a support staff member accepts the invitation, they can open the URLs mentioned above using their standard user account to perform tasks in a customer tenant.
Not perfect, but it does work until MS refines lighthouse.
Looks like the first (public) use of this functionality to bulk compromise the clients of a Partner (or in this case even worse, the CSP disti to multiple Partners)? https://www.theregister.com/2021/07/07/synnex_rnc_microsoft_attack/
Synnex have been emailing clients/partners asking them to remove their delegated admin access immediately.
In light of a very similar thing happening via Kaseya last weekend it is now time for Microsoft to close this crazy risk ASAP.
Have there been any improvements to this situation since the last post? From a customer perspective, I am not comfortable granting delegated global admin to partners. In fact, I agree with previous posters who point out that this betrays even basic security practice and common sense. Any alternatives?
@jschw78 Not yet. Alternatives are as discussed here - create a user account in the customer tenant to do administration, and only establish/allow delegated admin when Partner needs to escalate a problem to support.
The overarching problem with this solution is that Microsoft forced users to enable MFA for ALL accounts on their primary tenant in order to retain our partnership with Microsoft, yet the workaround you propose completely bypasses those obligations in that we don't need to enable MFA for these newly created global admins on our tenants. If we do enable it, we need one for every employee we have in order to use MFA, and keep the client protected. Otherwise, we are simply creating unprotected global admin accounts on every tenant that are a significant vector of brute force attacks.
GDAP is fine if that's what the bigger customers of Microsoft want, but it doesn't solve anything for partners, doesn't make anything simpler or more secure, and adds significant levels of complexity to manage customers.
It's a poor concept.
The solution here IMHO is to make the advanced auditing and alerting features available in Azure part of every base package so the customers (and partners) can get the alerts they need to know when an account has been breached or logged in, and we can create automation around that. Upcharging for securing the environment you've set up is pretty bold. At least if alerts are available, we all can help keep O365 secure, no?
I was looking forward to participating in the GDAP program, but was disallowed because I'm an indirect reseller, like most people. The only participants in the pilot now are likely larger orgs who don't see the SMB picture that was the foundation of your customer base.
Just some food for thought.
Microsoft are working on more options for this - the partner portal has been coded with Admin / Helpdesk roles for years. I suspect adding more granular permission options is a HUGE undertaking, so let's be patient. In the meantime, you do have options - think outside the box:
1. Did you know more than one person can register MFA against a single account? Yes they can! So for say 10-20 staff they can all scan the same QR code, or register separately under the same account. If you use IT Glue, it can even store the code and keeps rolling over as normal, so your support staff can use IT Glue rather than their phone to enter the code.
2. For Global Admin tasks, support staff could use a jumphost from your office to perform the admin functions. If it's a generic account, use conditional access to block the account from siging in- unless it is coming from the external IP associated with the jump host. This is secure (but not really good to use generic accounts).
3. Invite your support staff as guests to the client tenant (I'm sure you can whip up a csv with their email and names in it fairly quickly using export from your portal). Create a group in customer tenant that has the required roles assigned. Bulk add your support staff guest accounts to the group. Create a conditional access policy to block the group from signing in, uinless using MFA or from the external IP of a jump host without MFA.
@simbur666 : Indeed, changing the delegated admin model is a huge undertaking, and the solution - GDAP - is almost there - announced to be available in January: December 2021 announcements - Partner Center | Microsoft Docs
Thanks for providing some suggestions, I think those are valid approaches to think about. Only one small thing - I'm always a bit nervous when somebody configures an exception for essential things like MFA based on the network they access from. With a zero trust security strategy one should never assume the internal network is secure 🙂
@VNJoe : Thank you for your honest feedback. Of course, there is no need to use GDAP, feel free to continue using admin accounts for each customer tenant, AzureAD Security Defaults are enabled for most customer tenants so they can get those alerts you are seeking for free and so non-MFA protected accounts are only available if you bypass the basic security recommendations.
I did not propose the "idea" of using per-customer user accounts as a better workaround, it seems you did misunderstood the context of why I did mention this as one option. And I also think that you misunderstand the concept of GDAP and overestimate the complexity, I'm talking daily to Partners working with SMB customers using the same concept with regard to Azure Lighthouse and are interested in GDAP for the same reason. Your decision of course.
You state that AzureAD Security Defaults are enabled for most tenants, that would be correct, however, it seems to be an all or nothing deal.
For example, if the client requires SMTP auth for a legacy product, the security defaults block this, but can you simply enable SMTP auth while retaining all the other security practices? oh no, you have to turn it all off and pay extra for that customizability just to achieve a similar level of security.
It seems entirely backwards from a security standpoint, I understand you prefer to use modern authentication and I'm with you, but many SMB clients that MSP's have to manage use legacy products or cheaper alternatives to major well renowned brands simply because it makes financial sense for them and we as MSP's are expected to keep things running smoothly where possible.
Could the ability to ONLY allow SMTP auth be an option with a warning? alternatively could we have the ability to setup conditional access rules etc to mimic the Security Defaults behaviour without having to pay for extra rules and Azure licensing in a standard package?
We are all for improving upon best security practices and putting them into place, but when those practices restrict day to day requirements for clients, most times they have to be "worked around" or worse, switched off entirely.
@ITGDaryl : Valid feedback, having this option would certainly make those scenarios easier. The issue is, that investigations show that it is exactly those remaining legacy devices and legacy auth for those that are a known, and broadly used entry point for attackers. Pretty sure that AzureAD team said for this reason that they make no compromise because of this (Actually, there was an option as preview that allowed just that, remember "baseline policies"? But his path was not pursued for a reason). And reg. SMTP AUTH - It might that Exchange Online changes will hit those customers in this regard anyway sooner or later. There is no date for when we announced to disable SMTH Auth via Basic Auth, but it will happen at some point in time. It is now already disabled by default: Basic Authentication and Exchange Online – September 2021 Update - Microsoft Tech Community and for SMTP Auth options see Basic Authentication and Exchange Online – July Update - Microsoft Tech Community
I think, having AzureAD Security Defaults is a huge step forward, it delivers AAD P2 features for free, where before there were no no-cost options at all.