CSP and Delegated Admin security concerns: MFA is great, but what about the several outstanding Microsoft limitations?
Bringing-forward from https://www.yammer.com/office365partners/?show_login=true#/Threads/show?threadId=145300852449280... (Why the multiple communities here? Which is the go-forward platform here, Yammer or microsoftpartnercommunity.com ?)
How to support CSP customers while protecting the security of their tenant? Critical for security-conscious customers, and/or those with high-security requirements. Permissions granted by delegated admin are too far-reaching, do not allow for fine-grained access, and even the ability to audit use is unclear or non-existent.
This is something I had first started looking at just over a year ago. It doesn't look like anything has improved since, but customers are understandably only becoming increasingly concerned with this.
Here is some feedback we just received earlier today from a current customer looking to move to CSP licensing:
> Even with a NDA with <direct CSP provider> I am not at all comfortable making a third party of a third party delegated admins for our environment and granting them the ability to access all our cloud data which is what the terms state we’d be agreeing to. ... I really like the benefits of CSP otherwise so if there’s some other way to make this work we’d probably still do it. I also have reservations about a vendor having these all-encompassing permissions. I understand it can be needed for some admin tasks and troubleshooting but I don’t like how much of a free for all Microsoft makes it.
(I do believe there is some confusion in the above, in that our direct CSP provider / reseller would not have delegated admin - but only access to contact, subscription, and billing details. However, even this would be good to have further detailed in a format that we can share with concerned customers.)
If I were in the customer's place, I would absolutely be asking the exact same questions, and imposing the same requirements - and have so in prior organizations.
I have several other customers I've been working with, who I have not, in good faith, been able to recommend putting into CSP and the required delegated admin for - given the concerns outlined here.
We currently have a few hundred customers enrolled in CSP - and have an opportunity to bring even more into the CSP program, but the concerns outlined here do not help with the effort.
A number of specific points and concerns:
1) The list of roles and permissions that can be applied through Partner Center are available at https://docs.microsoft.com/en-us/partner-center/permissions-overview . Of these, only "Admin agent", "Helpdesk Agent", and maybe "Sales agent" apply to the customer tenants. "Admin agent" is basically the equivalent of "Global Administrator" within a customer tenant, and "Helpdesk Agent" is effectively a "Helpdesk Administrator" (Password Administrator). The admin roles from a tenant perspective are documented at https://docs.microsoft.com/en-us/office365/admin/add-users/about-admin-roles . As an O365 admin in my own organization, I can delegate permissions to others through at least 262,144 different combinations of the 18 customized administrator roles currently available in my tenant. (This increases to 38 roles available and nearly 275 billion combinations when using the roles available in Azure AD.) As a partner with delegated admin to a customer, there are only 2: "Everything", or "Helpdesk Agent" (or none). See also: https://www.yammer.com/office365partners/#/Threads/show?threadId=119473804279808 and https://www.yammer.com/office365partners/#/Threads/show?threadId=804328021 .
2) Everything is "all or nothing". We also have well over 100 engineers in our organization (not counting back office staff, etc.). To assign either "Admin agent" or "Helpdesk Agent" to one of our staff, means that they have that same permission across a few hundred customers. There is no way to filter a staff member's access to only one customer, or ideally a group of customers. (While we do have some additional flexibility in granting access through guest accounts in Azure AD, these permissions do not carry over to / are not usable in O365.)
3) We could consider falling-back to the use of extended and manually-reviewed audit logging as a compensating control. However, there are many limitations here - including that all logging appears to be held within each individual customer tenant only - and for us to review delegated admin activity from our partner account as a whole, it appears that we'd need to query each and every one of our customer tenants. Further concerns are raised here at https://www.yammer.com/office365partners/#/Threads/show?threadId=62199238885376 (unanswered).
While I'm on the topic, the "pretty wicked security hole" documented at https://office365.uservoice.com/forums/264636-general/suggestions/33233917-powershell-mfa-for-csp-delegated-admin-privileges also still needs to be addressed, well over a year later now. Can someone from Microsoft please review and respond to this UserVoice entry? Everyone else - please vote...
What are other CSP partners doing in terms of security controls to meet customer needs and expectations as it relates to CSP licensing? (We do have our own internal security program that I am a part of - which includes even stronger MFA controls across our entire organization than we'd otherwise been able to have approved, partially because of the customer security concerns here.)
As per https://blogs.technet.microsoft.com/uspartner_ts2team/2016/04/30/removing-the-csp-as-delegated-admin-in-office-365-customer-tenants/ , it looks like the simplest answer may to just be to have needing customers remove the delegated admin permissions. However: 1) Is this still accurate and working, 3 years later? And 2) we'll need to determine what limitations, if any, may then exist that would prohibit us from providing CSP licensing in this model, even if we then are not able to directly assign new licenses to users, or otherwise provide direct support to the customer as per CSP terms (at least not without having other customer-provided access given to us).
Just looking at this again now, it looks like we *might* be able to put the "Admin agent" and "Helpdesk Agent" roles behind the new "Azure AD entitlement management". This is less than 2 months old, still in preview, and what Privileged Identity Management really should have been all along, in my opinion. It does require Azure AD Premium P2 (or its parent EMS E5 offfering).
How about an actual announcement through the normal Partner Announcement channel? How about a more transparent discussion on this directly to the Partners instead of on some bulletin board? The strategies being implemented are creating difficulties in managing SMB customers and eliminating our ability to properly protect and secure them. Partner accounts are all required to have MFA, but tenant accounts do not. That's a gaping hole when partners are creating multiple global admin accounts on thousands of tenants just to be able to provide them proper configuration of Office 365, which is why they are customers. There's no addressing it, the engineers all state this use by design, but there's been no partner announcement at all, and these impactful changes aren't just in some early adopter program.
Auditing activity is obfuscated as well. Instead of seeing a very specific partner user who may have breached security for a client, they see that a generic global admin account did it instead. No accountability.
Can't even filter SPAM for customers at this point...
I have to admit, I have skimmed some of the responses, but heading back to one of the issues regarding Partner Center roles.
I fed back on various other forums and yammer groups that the roles are too broad, and really do not conform to any kind of basic, least priviledge best practice.
We have several departments, some are responsible for certain Microsoft offers/products.
Let's take Dynamics 365 as an example.
We have many customers with Dynamics 365. We may also support their Office 365, and Azure.
What I don't want is the Dynamics team having access to either Office 365, or Azure.
These are the responsibility of different support teams.
Also, what I don't want, is to give every Dynamics 365 team member access to all customers.
Vice versa, I don't want all Office 365 admins having access to Dynamics 365, or Azure.
I have to give all of these users Admin Agent roles, and have no control within Partner Center to decide what or who they have access to.
In addition to this, I don't want any of them to provision or increase/decrease license counts, or onboard new customers.
The roles need to be more granular, and my suggestion was to let Partner create role groups and the permissions for that group. There must also be a way to give either user access only to specific customers, or a simlar mechanism to PIM, where they are only give access when needed, and for a limited period.
Appreciate the focus from Microsoft on Partner security, MFA etc, but this is the first entry point for all our users. Microsoft should recognise these broad roles as a significant weak point, and address it soon.
I agree with @NickatTheta that the two roles in partner portal are far too broad an don't offer any level of true security.
Our specific problem, as a smaller MSP, is that when granting Admin agent to our staff which we were told is equivalent to Global Admin in the customers tenant, they don't actually get full Global Admin rights.
Some pretty basic functions (such as converting to and from Shared Mailboxes) are simply unavailable, and the worst thing about this is I cant find these restrictions documented anywhere!
Our work around, as we do have the full trust from our customers and complete control and ownership over their tenant, is create a global admin account in the customers tenant and share the credentials internally. (Stored securely in our encrypted Knowledge Base, and the account has MFA employed as well as our own accounts to access the KB before anyone raises a security issue on our end 😉 )
But doing this we lose any auditing or accountability internally of who did what. @JanoschUlmer can you offer any suggestions on what we should be doing? As a growing business created guest users in each customer tenant is something we had thought about but even as a business of our size that will quickly become an onerous task and as we look to increase customer and employee count will become even more unmanageable.
Alex C Brewer
@AlexCBrewer : I agree to and share your concerns, there are no really good solution to this.
Generally, the question if some action is supported to be done by an delegated admin is up to the decision of each product team, it is not something one (CSP) team alone can drive. So the feedback on missing functionality generally needs to be addressed in the specific teams - e.g. Uservoice: https://office365.uservoice.com/ . There are quite a few feedbacks on this already for different areas. Know areas are O365 Security & Compliance Center, some Sharepoint admin tasks, troubleshooting AAD Connect Sync issues. CSP engineering does of course also ask product teams to enable those scenarios, but still the decision is up the product team.
Then some of the features which are not available in the respective GUI, might still be possible using Powershell or the APIs. Sometimes it is just an issue on how the portal feature was designed, e.g. if current users group memberships are assessed.
Then having a shared global admin in the end customer tenant is a problem from compliance and auditing perspective, I fully agree. I know that Partner though about building their own PIM-system where Azure Automation was used to create admin identities on the fly and removing them. It is doable, but much work. Since admin accounts can exist in a customer tenant without having licenses applies some partners also create a number of non-shared admin accounts in order to assign them to specific users - some use shared accounts with MFA enabled and then use a centrally deployed token app (e.g. Authy running on a virtual workstation) so at least the most urgent security concerns are addressed.
I know that the identity team are working a more sophisticated delegated admin concept which might address some of the shortcomings, but it is to early to mention any specifics.
@JanoschUlmerThank you for the reply and your comments.
Whilst the situation is highly frustrating for those of us trying to do the right thing I do understand the complexities of trying to get all the various cogs at Microsoft to turn in the same direction.
The most annoying part of it though is the fact that the lacking abilities aren't identified anywhere, you discover them when trying to do something. When raising a support ticket with the Partner support team or searching on channels such as this you discover there is a known list of tasks that we can't do with delegated admin.
Is there a reason this information isn't more widely publicised? Even if just to 'already signed-up' partners so that we know what to expect from the outset?
Alex C Brewer
@AlexCBrewer ON your question before why this is not documented - honestly, I don't know.
Doing a bit of guessing - because of the same reasons I mentioned above - for these identity topics there is no CSP-centered engineering team, the decision and knowledge about issues is also split between teams.
I regularly trying to push people in this regard for the known issues, but I also learned that I need to have this conversation with each product team and there is little opportunity in my current role to have any meaningful impact. Will not stop in trying though 🙂
The best thing all Partner can do is that each of you pushes hard within uservoice, escalating via your account team, creating support requests, providing feedback for the documentation on delegation in partner Center ("not Helpful" option in top right area") - in order to get some more visibility from the respective people in the PGs.
I know, I should provide a more marketing-like answer as an employee, but in this regard I better tell the inconvenient truth 🙂
Thank you for your candid honesty.
I have submitted feedback on the page as you suggested, and I hope everyone reading this does the same!
Alex C Brewer
did you get any answers regarding the delegated admin issue. I share your view around customers not wanting to grant full access to us and we have this issue right now. The problem is that if a customer removes delegate admin (or we uncheck the box "Include delegated administration privileges for Azure Active Directory and Office 365." in Partner Center on the 'Request a reseller relationship' page when we set them up) then we cannot provision them any licences or see what they have. This seems to contradict what Woody Walton says in the blog (link in your post). I'm not sure if this is just a fault with Partner Center at the moment but it seems having a transacting only (not delegate admin) relationship is not possible.
Provisioning licenses or checking on licenses you have already provisioned for a customer is possible without having delegated admin permissions (also changing license count and suspending licenses is possible). So, if you experience issues with this this seems to be a technical issue where you can contact support (Just checked again in my test setup and I can manage licenses without having delegated admin - as expected).
Opening support tickets on behalf of the customer is not possible without delegated admin - so this might require that delegated admin is re-established once the customer asks for support and it is determined a ticket has to be raised at Microsoft support.
Sorry, but's that not just an acceptable solution or workaround. I'm surprised that the whole CSP ecosystem is not designed with security in mind. What's even worse, is that several of the activities an Admin Agent performs in the customers tenant, doesn't get logged and customer cannot see who/when an admin agent jumps to the tenant.
There is also 2 twists about this problem:
1. Customers sometimes do remove DAP themself, either by mistake or by intention. Then the customer cannot create support tickets in Azure at all. In M365 it's still technical possible (there's a "New Microsoft service request" button in the bottom of the wizard). If a customer do that, the customer is technically using the ASfP/PremierSupportForPartners agreement for the partner, which is a violation of the CSP itself.
2. Some customers implement Conditional Access rules, to block all access from non-corporate devices (AzureAD/HybridAD joined devices) - which makes sens . In that case, even if DAP is still in place, an Admin Agent cannot access the customer tenant and thereby cannot create support tickets.
Customers who requires a zero-trust approach, cannot use the CSP model as it is now as there is no built-in functionality that allows access to the tenant without allowing any device. I'm quite surprised that the architecture design behind the CSP got approval from a security & compliance perspective, as it contradicts Microsoft's own security guidelines and recommendations.
Thanks for the informative posts! Was trying hard to piece up the information and this really helped a lot.
On delegated administration with customers having conditional access, is there any material specific to CSPs on what customer require to set to be excluded from conditional access. The AD conditional access guides online are rather general and provides steps on how to, if there is specific guidelines on what to set this will help CSPs when onboarding customers greatly.
User activites are logged, the individual user account that is acting as admin agent is visible in the customer logs. What is not logged is when an admin agent just accesses a customer tenant without doing an activity (since there is no login happening, this can not be seen by the customer, though he could set a conditional access policies that can prevent this or force the Partner user to do MFA in his tenant again).
The customer can not create any support tickets himself, regardless if DAP is set or not. When DAP is removed the Partner can not do this anymore though.
It is correct when a customer opens a ticket in M365 admin portal, but he is not using the ASfP support contract. he will receive standard support same as every other customer. Even when he would use the ASfP contract there is no impact to the Partner since thre number of tickets is not limited.
Only for Premier this could happen - but the TAM will immediately notice this.
When the customer wants to restrict access via conditional access he can do so, he can always set certain exclusions for the Partner admin agents and/or specific other security requirements they need to fulfill.
If the customer asks a Cloud Solution Provider to be his managed service provider, but then has zero trust for the service provider of choice, indeed CSP is the wrong model for them. On the other hand, if the CSP partner is not able to show certain attestation that they have certain security implementations in place, they are not the right CSP for a customer that has those requirements.
CSP was designed for a scenario where the Partner is responsible for the management and has a vlaue-add service, not for a scenario where the customer just buys via a CSP, locks them out and only contacts them to raise a support ticket. If this is the intent of the customer, he shoud rather get a direct agreement with Microsoft.
The use case you are not considering here is when a security service provider cannot buy direct, and has to go through another direct partner (distributor).
Distributors generally do not have the same level of security as specialist security MSP's. Therefore the security partner whilst trusting of their own security will have to allow a less secure distributor full access.
At the moment this is the key stumbling block to us signing up to support and sell Microsoft products as a managed security partner. We are not going to sign up to be direct, and even if we did trust the security of the existing direct partners, we would be in breach of our contracts to give such unrestricted access to an unknown third party to our clients.
Selling licenses does not require the Indirect provider or Direct CSP to have delegated admin permissions. So in this case there is the option to only apply delegated admin for a short time when needed, e.g. for support escalation - or use Conditional Access to restrict access only to certain scenarios (e.g. CSp Partner can only access customer tenant with their own credentials when using a certain network).
I agree that as a managed security partner the best option would be to become Direct Partner, so that no other party is involved in the relationship - though directly signing up for Direct is no longer possible unless you have reached the minimum revenue requirements (as indirect).
As mentioned above, this is an area which is currently being worked on, though no ETA can be announced.
@JanoschUlmer I don't agree
Many of our customers has very sensitive information and they not allow us to have any permissions on their resources.
Currentlly We are working with those customers on remote sessions or in their office, using the customer username
But, I understand that on the new “Azure Plan” program the partner must have admin rights on the customer environment to get rebate.
And if the customer remove the permission then the partner is losing the rebate.
By forcing this policy Microsoft is build a system that weaken security significantly
Microsoft own best practice recommendation is no more than 3 global admin, and no more than 3 owners for subscriptions
But when giving partner full access then it’s adding all partner technical users (usully few dozen) with permission equivalent to global admin and subscriptions owners.
The CSP model indeed requires Partner to have strong security and control implemented on his end, since when a managed service provider is responsible for the enbd customers services it should of course not lower the security, but ideally increase it. This is also why the new Partner Agreement requires Partner to have such controls in place that protects privacy and ensure securioty of customer data.
That is what CSP is all about, it is not just a licensing program, but a model for (Cloud) Solution Providers that take over the responsibility for service management, and naturally the "Solution Provider" needs some for of access on the environment. It might not be the right model for all customers of course.
Also, independent from what has been done reg. Azure Plan, the CSP partners needs to have some form of access to open support tickets on behalf of the customer - delegated admin permissions for O365 (which is equivalent of Global Admin), and in Azure there can be more fine grained roles, like Support Contributor.
I agree with you that just using the default permissions that are set (AOBO/Foreign Principal with Owner rights on subscription) are not sufficient for the demands of some customers, this is why there are additional options - like using an account that is created in the customer tenant, but linked to the Partner ID (PAL - Partner Admin Link). For your scenario this sounds like the right approach, in order to be able to open support tickets on behalf of the customer I would also think about using a guest account from your Partner tenant to the subscription.
Specifically take a look at the list of roles that can be used to get PEC
Beneath that there are further options that give the customer more control, like e.g. using MFA for those accounts where only the customer has access to the 2nd factor, use auditing, use Azure policies etc...
Yes, but you are ignoring the most important thing, the money
If partner don’t have admin on behalf (PAL is it’s not enough) then he is not getting any rebate or margin from Microsoft
In practice, he is losing money on every such customer
So partner biggest incentive is to set admin on behalf and not lose ~20% rebait and margin money
On the customer side, if he removes partner permissions then the partner will cancel any discount the customer has on Azure
So by this policy Microsoft encourage both sides (partners and customers) to weaken security
That is not true, AOBO is one of three methods to get PEC. You can also use Lighthouse for solutions you deploy, and you can link an account from customer tenant, a guest account or a service princpal via PAL. And those account do also not need to have owner permissions, other roles work to.
This is all described in the before mentioned links: https://docs.microsoft.com/en-us/partner-center/azure-plan-manage
When customer removes AOBO, there is warning message displayed for the customer that this might impact the Partner relationship, and we also encourage Partner to think about setting up an alert for any of those changes, this can also be set if the specific account is changed/deleted via you have access: https://docs.microsoft.com/en-us/partner-center/azure-plan-manage#create-an-azure-monitor-alert
So this does not weaken security if implemented the right way with least privilege in mind (e.g.the idea to hand over the 2nd factor MFA so only he can approve Partner sign in). It is also important Partner does have the necessary controls in house for basic security on his side. And to me this is the important resulting change - the need to have such controls, to be able to attest to the customer that you as service provider are able to provide secure delegated management, is much more important than before, at least to be able to hand over discounted prices because of course the customer should have questions if giving the Partner the keys to the kingdom is "secure enough" - this was important before, but now there is more impact.
Again, CSP is not just a licensing channel where it is only about margins for reselling service, CSP is, and was also before Azure Plan, a servicing model. I'm not ignoring anyhow that it is about the money, I'm just emphasizing the fact that it is not only about resale-driven revenue, it is also about revenue generated via end-to-end & ongoing servicing. I know that CSP was used by some only to be able to hand over some discounts, but this was never the idea, and the new Azure plans made this much more visible.
If the customer wants to lock a Partner out from certain areas completely, then you are correct that without PEC there will not be any discount for them - incentives are still payed though (Actually is 15% discount for Azure and ~6% incentives, not 20% overall).
However, I'm not agreeing with you that in practice Partner will lose money - it is a matter of discussing the impact with the customer. It would certainly not be a good idea to set up an agreement that forces you to give a fixed discount even when the customer locks you out, so the matter of PEC & the need to have "some" permissions need to be included in the agreements you have with the customer - and also impact on servicing fees when permissions are removed (since you might have incorporated those to some part into the margin).
I do fully understand that those changes are fundamental and will leed to some tough discussions with customers and I appreciate the open & honest feedback on this. I also fully agree that it would be great to have imporovements in the way delegated admin permission can be scoped on Partner side when using AOBO, this is something I discussed a lot internally during the last months (but as a Partner consultant unfortunately I have no power to decide on this).
@JanoschUlmer Thank you for detail answers
Are you sure that PAL with user from the customer tenant is good enough for rebait and margin?
If so it can solve the problem
And Microsoft must accept that partners can give an excellent service without any permissions
Using only remote desktop sharing sessions (the same way as Microsoft support works)
Some customers will never give permission to partner.
Never mind the partner security level.
So for those customers CSP is currently not an options
Yes, PAL will be sufficient, as documented here: https://docs.microsoft.com/en-us/partner-center/azure-plan-manage#link-your-partner-id-mpn-idto-your-credentials-for-managing-customers-azure-resources
Take note of the description for "Directory or Guest user" which says:
Partner needs to associate the MPN ID to the user or service principal in the customer tenant. For more information - Link Partner ID.
And also the information reg. the proper roles (e.g. PAL for a user that only has read access won't work to get PEC).
I agree that by using Remote support some scenarions do work for reactive support, when it comes to proactively managing the environment (e.g. "Hey, we have found that you have some VMs mostly idle, can we discuss how to automate shutdown or change VM sizes?") this fails.
While it is true that Microsoft support will try to resolve issues with the customer/partner using remote support, Microsoft of course has certain access to the backend also (not necessarily the frontline support engineer, and never without an internal approval process - e.g. think about the obvious scenario that customer locks himself out when forgetting the admin password and he contacts support) - and this is where Microsoft provides a lot of proof & attestation in the trust center that while Microsoft has this access, respective controls are in place - and also legal documentation in the contracts that document proper handling of customer data and security controls. So by choosing a cloud provider, customer does grant the cloud provider always an implicit access to his environment - at least thise parts of it where he can't apply additional security concepts like e.g customer lockbox, HYOK encryption, confidential computing etc. - and therefore customer has put some trust in the cloud provider already. So when the customer is using cloud services, it is clearly a sign that they are not saying "we can't let a 3rd party access our systems ever", but "we can only give a 3rd party some limited access to our system when certain conditions are fulfilled".
I personally think that the relation between a managed service provider (=CSP) and a customer is similar in this regard, finally it is a matter of trust & control. Of course, earning the trust and explaining the controls can be a tough job 🙂