Hands off the delegates tab
So, in the last entry I mentioned that a lot of people use the delegation feature in Exchange when their workflow and business processes could benefit from the use of sharing permissions instead. Why would it be useful for you to implement this workaround? Let’s clarify a little bit.
Many companies today are finding the RPC over HTTPs (a.k.a. Outlook Anywhere in 2007) scenario to be quite an interesting deployment method. I’m here to tell you, battle scarred and worn… I’ve deployed RPC over HTTPs as the primary topology for an Exchange environment. We’re talking 80% of the population. It presents some unique challenges, but we won’t go into all of those here and now. Ask me later sometime about non-paged pool memory and I’ll share horror stories with you.
As of Exchange 2003 SP2, Microsoft made one of their famous changes in how things operate within Exchange. Apparently, a complaint from customers resulted in the correction of a “security issue.” The issue at hand was that if you were to pull up the “Connection Status” window in Outlook 2003 or 2007, you would see the names of global catalog servers listed for your directory connections. (Ref: to open Connection Status, ctrl-right-click on the Outlook icon in your systray and choose “Connection Status.” The Directory connections are what I’m referring to). In Exchange 2003 SP2, the global catalog references are now handled by the backend servers. Your requests are proxied by those backend mailbox servers on your behalf, therefore you will see your backend mailbox server listed as your Directory server.
If you’re fortunate enough to have an Outlook client on the same basic network as your servers and you can use straight-up MAPI, this doesn’t pose much of a problem to you. However, you should still see below about the hidden delegates limitations.
In an RPC over HTTPS deployment scenario, let’s take an example architecture and consider how your client connections are processed:
client (port 443 connection) –> ISA Server –> Front-end server –> Backend server (proxy request) –> global catalog DC
This means for each directory request, your transaction passes through three servers before reaching the 4th server to be processed. The request is then routed back through that same chain. As you can see, there’s a lot that can happen there.
Now then. Let’s say you have a shared resource calendar that 30 people use as part of a standard process. It could be a conference room, whatever. Let’s say that you decide that all 30 of these people need access to the calendar as a delegate. We won’t talk about whatever silly decisions you made to get you to this conclusion, we’ll just talk about what happens.
You open your Outlook client, log into the resource mailbox, and open Tools/Options/Delegates tab and begin to add your 30 delegates.
Notice that as you start to add delegates, things start to slow down around the 10th or 15th delegate or so. Open your connection status window and note that your Directory queries are running unusually high. Keep adding those delegates and keep watching your client slow to a crawl - but watch those directory queries go into the thousands.
If you’re lucky enough to get to punch the OK button to save your changes, you’re likely going to be sitting there for another 30-45 minutes. If you’re in a resource forest setup, you could be waiting up to an hour or 90 minutes or so. No, I’m not kidding. It will take that long to process your delegates changes.
Now let’s remember, I did say that if you’re lucky enough to have a direct MAPI connection to your Exchange environment, chances are this isn’t going to happen to you. You will, however, run into the unwritten limitations.
So by this point, you’re probably wondering what has happened to your client. You may even perform an “end task” on Outlook… which, honestly, wouldn’t be a very good idea at this point. You really need to let the client complete, regardless of how hosed it appears. If your Outlook client ever does finish, I challenge you to open Tools/Options/Delegates tab and endure another long wait with your connection status window open. I would wager that your Directory connections spin off into the five digit thousands or so.
Alright, so now you see part of the problem. The customer has a business process that led them to want to use this feature and the feature is essentially broken for a large number of delegates. If you’re like any other sysadmin out there, you’re probably on the phone to Microsoft by now.
Here’s what you’ll be told. Microsoft has actually only tested the delegates feature with no more than 4 delegates. That’s right - 4 delegates. There you have it - an undocumented rule on how many delegates you should have - no more than 4.
It gets better. Delegates count as part of your 32kb limit per mailbox on the rules. If you add too many delegates, you may find yourself out of room to add rules to the mailbox - or vice versa.
Microsoft is going to use this as an out - because quite frankly, up until now, no one has tried to use the delegates feature the way you’re using it, etc. etc. You’re screwed. You now get to play the customer management game and convince them that right-clicking on the calendar folder and adding permissions on the Sharing tab is what they really meant to do.
But you get to convince them of that after you remove all of the delegates… and that means opening that tab and letting it sit overnight, just so you can remove the delegates.
Ghost SIDs
In a resource forest setup, it’s important to note that the identities of your users come from the trusted forests, not the forest that your Exchange organization is built in.
Oh… wait… what do you mean what’s a resource forest?
A resource forest is a Microsoft-supported method for creating a brand new Active Directory and using it to deploy services separately from your user accounts. There may be several technical or political reasons to do this - but should you result in this design decision, you need to make sure that your enterprise has several things going right for it in the process area.
In a resource forest with Exchange deployed into it, you still have user accounts - but they are disabled. They are empty “zombie” user accounts. The “soul” of the account is referenced back to the trusted userid in the trusted forest. How does one do that, you might ask? Create an account and disable it - then right-click on the account and choose “Exchange tasks.” Choose “Associate with external account” and you will do just that - associate the soul of a user with a zombie in the resource forest.
This presents some unique challenges.
When assigning permissions or delegate settings in your Exchange forest, the SID that is associated with the ACL is the SID of the user in the trusted forest, not the SID of the disabled zombie account. This means that if you assign delegates or permissions on a user and the userid is destroyed in the trusted resource forest, you now have a “ghost” - or more specifically, a “ghost SID.”
Take the following example:
- Roger works at contoso.com.
- Julie works at northwind.com
- Both users are members of companies that were purchased by breadandbutter.com
- breadandbutter.com’s Exchange deployment is stood up in a resource forest known as mail.breadandbutter.com
- Roger and Julie both have disabled accounts in mail.breadandbutter.com that are externally associated to their accounts in contoso.com and northwind.com
- Julie accesses the delegates tab and makes Roger a delegate of her mailbox. This places Roger’s contoso.com SID on her mailbox with the appropriate permissions (NOTE: NOT the mail.breadandbutter.com SID!)
- Brenda sends an invitation to a meeting to Julie. Roger receives a copy of the invitation too because, well, he’s a delegate and things are functioning happily.
- Roger is caught looking at pr0n and fired on the spot. His account is immediately deleted from contoso.com but is NOT deleted from mail.breadandbutter.com.
- Brenda sends another invitation for a meeting to Julie. She receives an NDR, indicating that delivery has failed - but the NDR does not state who the failure against!
Bing, you have a ghost SID problem. Can you tell me which step created the NDR/ghost SID issue? That’s right, class… it’s when Roger was fired for looking at pr0n. When Roger was fired and his SID was destroyed, the SID was still associated in Julie’s mailbox but no longer resolves correctly. Thus, when Brenda sent the invite, Exchange tried to “do the right thing” but it sent a copy of the invite to a ghost.
Now you might see the light with what I mentioned earlier… if you’re going to have this kind of setup, make sure your enterprise has the processes together. What should have happened is that if they deleted Roger’s contoso.com account, they should also delete his mail.breadandbutter.com account. Without that deletion, no one has any clue that Roger is MIA… including Exchange.
How do you resolve it? One of two ways:
1. Open Julie’s delegates settings and remove the SID listed there
2. Open Julie’s mailbox with pfdavadmin and remove the SID listed there on the calendar folder
You will see Roger’s SID instead of his name because the domain is trying to resolve Roger’s SID on the trusted domain, not the resource forest. Since the SID does not resolve, all you see is the SID itself. That indicates a problem.
Now then. Combine this issue with the problem of people putting 30-40 delegates on a mailbox and you have the makings for a real customer management crisis. The devil’s in the design and documentation, folks. Pay attention to what your users are doing and why. Otherwise, they could work themselves into a very confusing scenario much like this and hey, any confusing scenario equals more stress for you.
Write them a spiffy Word document that steers them clear of the delegates tab and you’ll be sitting Tazo tea the rest of the afternoon. Assuming those sheep read documentation, that is. We all know what to think about that, don’t we?
