Hero Banner

Key Resources and Guides

Find key resources and guides that you can accelerate implementations

Reply
ziesemer
Level 4 Contributor

CSP and Delegated Admin security concerns: MFA is great, but what about the several outstanding Microsoft limitations?

Bringing-forward from https://www.yammer.com/office365partners/?show_login=true#/Threads/show?threadId=145300852449280...  (Why the multiple communities here?  Which is the go-forward platform here, Yammer or microsoftpartnercommunity.com ?)

 

How to support CSP customers while protecting the security of their tenant? Critical for security-conscious customers, and/or those with high-security requirements. Permissions granted by delegated admin are too far-reaching, do not allow for fine-grained access, and even the ability to audit use is unclear or non-existent.

 

This is something I had first started looking at just over a year ago. It doesn't look like anything has improved since, but customers are understandably only becoming increasingly concerned with this.

 

Here is some feedback we just received earlier today from a current customer looking to move to CSP licensing:

 

> Even with a NDA with <direct CSP provider> I am not at all comfortable making a third party of a third party delegated admins for our environment and granting them the ability to access all our cloud data which is what the terms state we’d be agreeing to. ... I really like the benefits of CSP otherwise so if there’s some other way to make this work we’d probably still do it. I also have reservations about a vendor having these all-encompassing permissions. I understand it can be needed for some admin tasks and troubleshooting but I don’t like how much of a free for all Microsoft makes it.

 

(I do believe there is some confusion in the above, in that our direct CSP provider / reseller would not have delegated admin - but only access to contact, subscription, and billing details. However, even this would be good to have further detailed in a format that we can share with concerned customers.)

 

If I were in the customer's place, I would absolutely be asking the exact same questions, and imposing the same requirements - and have so in prior organizations.

 

I have several other customers I've been working with, who I have not, in good faith, been able to recommend putting into CSP and the required delegated admin for - given the concerns outlined here.

 

We currently have a few hundred customers enrolled in CSP - and have an opportunity to bring even more into the CSP program, but the concerns outlined here do not help with the effort.

 

See also: https://docs.microsoft.com/en-us/azure/cloud-solution-provider/customer-management/administration-delegation

 

A number of specific points and concerns:

 

1) The list of roles and permissions that can be applied through Partner Center are available at https://docs.microsoft.com/en-us/partner-center/permissions-overview . Of these, only "Admin agent", "Helpdesk Agent", and maybe "Sales agent" apply to the customer tenants. "Admin agent" is basically the equivalent of "Global Administrator" within a customer tenant, and "Helpdesk Agent" is effectively a "Helpdesk Administrator" (Password Administrator). The admin roles from a tenant perspective are documented at https://docs.microsoft.com/en-us/office365/admin/add-users/about-admin-roles . As an O365 admin in my own organization, I can delegate permissions to others through at least 262,144 different combinations of the 18 customized administrator roles currently available in my tenant. (This increases to 38 roles available and nearly 275 billion combinations when using the roles available in Azure AD.) As a partner with delegated admin to a customer, there are only 2: "Everything", or "Helpdesk Agent" (or none). See also: https://www.yammer.com/office365partners/#/Threads/show?threadId=119473804279808 and https://www.yammer.com/office365partners/#/Threads/show?threadId=804328021 .

 

2) Everything is "all or nothing". We also have well over 100 engineers in our organization (not counting back office staff, etc.). To assign either "Admin agent" or "Helpdesk Agent" to one of our staff, means that they have that same permission across a few hundred customers. There is no way to filter a staff member's access to only one customer, or ideally a group of customers. (While we do have some additional flexibility in granting access through guest accounts in Azure AD, these permissions do not carry over to / are not usable in O365.)

 

3) We could consider falling-back to the use of extended and manually-reviewed audit logging as a compensating control. However, there are many limitations here - including that all logging appears to be held within each individual customer tenant only - and for us to review delegated admin activity from our partner account as a whole, it appears that we'd need to query each and every one of our customer tenants. Further concerns are raised here at https://www.yammer.com/office365partners/#/Threads/show?threadId=62199238885376 (unanswered).

 

While I'm on the topic, the "pretty wicked security hole" documented at https://office365.uservoice.com/forums/264636-general/suggestions/33233917-powershell-mfa-for-csp-delegated-admin-privileges also still needs to be addressed, well over a year later now. Can someone from Microsoft please review and respond to this UserVoice entry? Everyone else - please vote...

 

What are other CSP partners doing in terms of security controls to meet customer needs and expectations as it relates to CSP licensing? (We do have our own internal security program that I am a part of - which includes even stronger MFA controls across our entire organization than we'd otherwise been able to have approved, partially because of the customer security concerns here.)

 

As per https://blogs.technet.microsoft.com/uspartner_ts2team/2016/04/30/removing-the-csp-as-delegated-admin-in-office-365-customer-tenants/ , it looks like the simplest answer may to just be to have needing customers remove the delegated admin permissions. However: 1) Is this still accurate and working, 3 years later? And 2) we'll need to determine what limitations, if any, may then exist that would prohibit us from providing CSP licensing in this model, even if we then are not able to directly assign new licenses to users, or otherwise provide direct support to the customer as per CSP terms (at least not without having other customer-provided access given to us).

 

Just looking at this again now, it looks like we *might* be able to put the "Admin agent" and "Helpdesk Agent" roles behind the new "Azure AD entitlement management". This is less than 2 months old, still in preview, and what Privileged Identity Management really should have been all along, in my opinion. It does require Azure AD Premium P2 (or its parent EMS E5 offfering).

40 REPLIES 40
simbur666
Level 1 Contributor

While we wait for lighthouse to become a solution for this, we looked into other ways to achieve the goal.  You can do most things using B2B guest access and client targeted URLs (appending the client tenant to the admin URL):

 

Azure - https://portal.azure.com/customer.com  
Intune - https://endpoint.microsoft.com/customer.com  
Security Center - https://security.microsoft.com/customer.com  
Exchange - https://outlook.office365.com/ecp/@customer.onmicrosoft.com 
SharePoint - https://customer-admin.sharepoint.com/

 

Unfortunately this doesn't seem to apply for the 365 Admin Center, which is where you would want

Helpdesk staff to be performing User / Exchange tasks, rather than jumping between Azure AD and Exchange (but at least it works, and achieves the goal your client is after).

 

Here is the process:

1. customer accepts MSP invitation

2. customer removes the admin / helpdesk agent privileges from the partners area of the customers portal (this keeps the association and you can still procure licensing for them, but removes the default partner permissions)

3. create two or more groups in customers Azure AD; one for your Helpdesk and one for Admins (with assign roles to group enabled)

4. assign the roles to the Helpdesk group (change to fit your needs or use custom roles):
User Administrator
Exchange Recipient Administrator

5. assign roles to the Admin group as required from this list (you can use more admin groups to assign roles to different support groups if required):

Intune Administrator
Authentication Administrator
Exchange Administrator
User Administrator
Guest Inviter
Application Administrator
Compliance Administrator
Global Reader
Conditional Access Administrator
Cloud App Security Administrator
License Administrator
Azure AD Joined Device Local Administrator
Groups Administrator
SharePoint Administrator
Privileged Role Administrator
Azure Information Protection Administrator
Security Administrator

 

6. assign Azure subscription roles to the Admin groups as required:
Contributor

 

7. you can also use these groups to assign permissions to certain Azure objects, using the IAM blade under the resource

 

8. Use Bulk Invite in customers Azure AD users blade to invite all your support staff to the customer tenant as guests

 

9. Add the invited guest accounts to the groups you created

 

After a support staff member accepts the invitation, they can open the URLs mentioned above using their standard user account to perform tasks in a customer tenant.

Not perfect, but it does work until MS refines lighthouse.

 

Cheers,
Simon

PHarrisonCWSI
Visitor 1

Looks like the first (public) use of this functionality to bulk compromise the clients of a Partner (or in this case even worse, the CSP disti to multiple Partners)?  https://www.theregister.com/2021/07/07/synnex_rnc_microsoft_attack/

Synnex have been emailing clients/partners asking them to remove their delegated admin access immediately.

 

In light of a very similar thing happening via Kaseya last weekend it is now time for Microsoft to close this crazy risk ASAP.

Andra
Community Manager

@v-crcarl addin you on this thread for awareness.

jschw78
Visitor 1

Have there been any improvements to this situation since the last post?   From a customer perspective, I am not comfortable granting delegated global admin to partners.  In fact, I agree with previous posters who point out that this betrays even basic security practice and common sense.   Any alternatives?

JanoschUlmer
Microsoft

@jschw78 Not yet. Alternatives are as discussed here - create a user account in the customer tenant to do administration, and only establish/allow delegated admin when Partner needs to escalate a problem to support.

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
Leho
Level 3 Contributor

Again - I really can't understand why there is still no soluton or even notes about possible solutions? 2 years have passed from initial post. Sorry but MS finds time and resources for pointless product rebranding projects but not for critical issue.

 

And for suggested workaround - I would be more than happy to ask MS to explain to customers why they have to pay for partners' users if they will not grant admin rights for partner.

Leho
Level 3 Contributor

Sorry, I really can't understand so numb attitude from Microsoft side regarding so serious topic. 

For me as Dynamics partner current topic begins to be obstacle to take customers to cloud. But that is what Microsoft wants and where he expects partners should focus. 

 

From customer perspective:

I always have to explain any bigger customer why they need to grant us permissions what in theory allow is to do almost everything. Half of them accept that and half not. For example no international companies grant us such permissions cause they have higher security standards and don't let us see their whole AD. So we have to find alternative solutions. 

 

From Dynamics partner perspective:

We have 60 employees who are serving Dynamics customers on everyday basis. They need to access customer environments and it's possible through Partner Central where are listed all our Dynamics SaaS customers - some of them smaller, some bigger, some members of the exchange etc. And there is no limitation where our people can access. I hope MS understands to what kind of data actually all these people can access and what can be the worst scenario if somebody has bad intensions. Or is just curious. It's highly confidential data starting from financial statements till sales companies pricing information or manufacturing companies self-cost formulas.

OK, the alternative is that Dynamics partner has no delegated rights and will be defined in ERP as ordinary user but then the partner's user is not free of charge anymore. Majority of customer can't understand why they have to pay for partner's users to be able get consultation service.

 

So I really hope that this topic finds at least some kind of solution asap and keeps improving continuously.

JanoschUlmer
Microsoft

@Leho : As mentioned in before discussion, this is already being worked on.

If you want to give feedback: Join the Partner Center API and SDK Early Adopter Program - Partner Center app developer | Microsoft Docs (not only applicable to API topics, this is where you will learn about upcoming improvements first and where you can give feedback to the PMs)

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
JanoschUlmer
Microsoft

@AlexCBrewer & All others: This might be of interest for you: https://techcommunity.microsoft.com/t5/small-and-medium-business-blog/announcing-microsoft-365-lighthouse-for-managed-service/ba-p/1698181

This might be the first step in removing some of the limitations and visible effort by the PGs to address specific MSP scenarios in a better way. There is also more to come specifically for CSP Partners, though it will take a while until something will be shared. 

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
AlexCBrewer
Level 4 Contributor

@JanoschUlmer Thanks for the mention.

I have seen Lighthouse and we have registered for access to the preview, although at this stage it looks like not much more than a reporting dashboard on the Intune device estate of our clients. I can't see anything, as yet, for AAD user reporting and nothing with regards to actual management on client tenants.

Whilst I do see the benefits Lighthouse will bring us, it does not solve the problems highlighted in this and other threads about how our delegated admin permissions do not give full global admin rights over client tenants which is critical for proper full MSP offerings

JanoschUlmer
Microsoft

As mentioned - "first step" 🙂  For Partners there are also changes on the roadmap.

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
AlexCBrewer
Level 4 Contributor

Hi @JanoschUlmer , is that roadmap public? To partners at least?

JanoschUlmer
Microsoft

For Direct CSP and Indirect Providers there is an early adopter program, but I can't currently tell if new Partner will be accepted. See this thread if you have access to the Cloud Partner Yammer network: https://www.yammer.com/cloudpartnercommunity/#/Threads/show?threadId=678930850922496 

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
VNJoe
Level 4 Contributor

How about an actual announcement through the normal Partner Announcement channel? How about a more transparent discussion on this directly to the Partners instead of on some bulletin board? The strategies being implemented are creating difficulties in managing SMB customers and eliminating our ability to properly protect and secure them. Partner accounts are all required to have MFA, but tenant accounts do not. That's a gaping hole when partners are creating multiple global admin accounts on thousands of tenants just to be able to provide them proper configuration of Office 365, which is why they are customers. There's no addressing it, the engineers all state this use by design, but there's been no partner announcement at all, and these impactful changes aren't just in some early adopter program.

 

Auditing activity is obfuscated as well. Instead of seeing a very specific partner user who may have breached security for a client, they see that a generic global admin account did it instead. No accountability.

 

Can't even filter SPAM for customers at this point...

NickatTheta
Level 6 Contributor

I have to admit, I have skimmed some of the responses, but heading back to one of the issues regarding Partner Center roles.

 

I fed back on various other forums and yammer groups that the roles are too broad, and really do not conform to any kind of basic, least priviledge best practice.

 

We have several departments, some are responsible for certain Microsoft offers/products.

Let's take Dynamics 365 as an example.

 

We have many customers with Dynamics 365. We may also support their Office 365, and Azure.

What I don't want is the Dynamics team having access to either Office 365, or Azure.

These are the responsibility of different support teams.

 

Also, what I don't want, is to give every Dynamics 365 team member access to all customers.

Vice versa, I don't want all Office 365 admins having access to Dynamics 365, or Azure.

 

I have to give all of these users Admin Agent roles, and have no control within Partner Center to decide what or who they have access to.

 

In addition to this, I don't want any of them to provision or increase/decrease license counts, or onboard new customers.

 

The roles need to be more granular, and my suggestion was to let Partner create role groups and the permissions for that group. There must also be a way to give either user access only to specific customers, or a simlar mechanism to PIM, where they are only give access when needed, and for a limited period.

 

Appreciate the focus from Microsoft on Partner security, MFA etc, but this is the first entry point for all our users. Microsoft should recognise these broad roles as a significant weak point, and address it soon.

AlexCBrewer
Level 4 Contributor

I agree with @NickatTheta that the two roles in partner portal are far too broad an don't offer any level of true security.

 

Our specific problem, as a smaller MSP, is that when granting Admin agent to our staff which we were told is equivalent to Global Admin in the customers tenant, they don't actually get full Global Admin rights.

 

Some pretty basic functions (such as converting to and from Shared Mailboxes) are simply unavailable, and the worst thing about this is I cant find these restrictions documented anywhere!

 

Our work around, as we do have the full trust from our customers and complete control and ownership over their tenant, is create a global admin account in the customers tenant and share the credentials internally. (Stored securely in our encrypted Knowledge Base, and the account has MFA employed as well as our own accounts to access the KB before anyone raises a security issue on our end 😉 )

 

But doing this we lose any auditing or accountability internally of who did what. @JanoschUlmer can you offer any suggestions on what we should be doing? As a growing business created guest users in each customer tenant is something we had thought about but even as a business of our size that will quickly become an onerous task and as we look to increase customer and employee count will become even more unmanageable.

 

Many Thanks,

Alex C Brewer

JanoschUlmer
Microsoft

@AlexCBrewer : I agree to and share your concerns, there are no really good solution to this.

 

Generally, the question if some action is supported to be done by an delegated admin is up to the decision of each product team, it is not something one (CSP) team alone can drive. So the feedback on missing functionality generally needs to be addressed in the specific teams - e.g. Uservoice: https://office365.uservoice.com/ . There are quite a few feedbacks on this already for different areas. Know areas are O365 Security & Compliance Center, some Sharepoint admin tasks, troubleshooting AAD Connect Sync issues. CSP engineering does of course also ask product teams to enable those scenarios, but still the decision is up the product team.

 

Then some of the features which are not available in the respective GUI, might still be possible using Powershell or the APIs. Sometimes it is just an issue on how the portal feature was designed, e.g. if current users group memberships are assessed.

 

Then having a shared global admin in the end customer tenant is a problem from compliance and auditing perspective, I fully agree. I know that Partner though about building their own PIM-system where Azure Automation was used to create admin identities on the fly and removing them. It is doable, but much work. Since admin accounts can exist in a customer tenant without having licenses applies some partners also create a number of non-shared admin accounts in order to assign them to specific users - some use shared accounts with MFA enabled and then use a centrally deployed token app (e.g. Authy running on a virtual workstation) so at least the most urgent security concerns are addressed.

 

I know that the identity team are working a more sophisticated delegated admin concept which might address some of the shortcomings, but it is to early to mention any specifics.

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
AlexCBrewer
Level 4 Contributor

@JanoschUlmerThank you for the reply and your comments.

 

Whilst the situation is highly frustrating for those of us trying to do the right thing I do understand the complexities of trying to get all the various cogs at Microsoft to turn in the same direction.

 

The most annoying part of it though is the fact that the lacking abilities aren't identified anywhere, you discover them when trying to do something. When raising a support ticket with the Partner support team or searching on channels such as this you discover there is a known list of tasks that we can't do with delegated admin.

 

Is there a reason this information isn't more widely publicised? Even if just to 'already signed-up' partners so that we know what to expect from the outset?

 

Many Thanks,

Alex C Brewer

JanoschUlmer
Microsoft

@AlexCBrewer ON your question before why this is not documented - honestly, I don't know. 

Doing a bit of guessing - because of the same reasons I mentioned above - for these identity topics there is no CSP-centered engineering team, the decision and knowledge about issues is also split between teams. 

I regularly trying to push people in this regard for the known issues, but I also learned that I need to have this conversation with each product team and there is little opportunity in my current role to have any meaningful impact. Will not stop in trying though 🙂 

The best thing all Partner can do is that each of you pushes hard within uservoice, escalating via your account team, creating support requests, providing feedback for the documentation on delegation in partner Center ("not Helpful" option in top right area") - in order to get some more visibility from the respective people in the PGs. 

 

I know, I should provide a more marketing-like answer as an employee, but in this regard I better tell the inconvenient truth 🙂

 

 

Kind regards,
Janosch
Get consultations form Technical Presales & Deployment services team via https://aka.ms/technicalservices
AlexCBrewer
Level 4 Contributor

Hi @JanoschUlmer 

 

Thank you for your candid honesty.

I have submitted feedback on the page as you suggested, and I hope everyone reading this does the same!

 

Many Thanks,

Alex C Brewer