As an IT admin it will happen to all of us at some point; there will be that problem that seems like you are 10 minutes away from fixing that quickly turns into 10 hours and then 2, 3, even 5+ days.  Before you know it, you have spent a week with nearly zero sleep and a lot of caffeine and then you finally realize that you are not any further along than when you started.  I spent the last week banging my head up against a wall trying to get a clients new Windows Server 2008 and Exchange 2007 SP1 environment up and running, only to find out that Microsoft has a crippling bug in Windows Server 2008 that won’t allow Outlook Anywhere (a.k.a. RPC over HTTP) to run in its default configuration.

The most unfortunate part about this is that Microsoft is still yet to release any information publicly about this problem, which is really sad because they generally do such a great job of at least posting limitations of their products on many of their wonderful blogs.  I had to search the Internet and eventually found articles that led me in the right direction but I was never able to find a blog/article that outlined the exact steps that I used to fix/diagnose Outlook Anywhere which is why I really felt the need to write this post.

The basis of the problem is that Windows Server 2008 (like Windows Vista) gives precedence to IPv6 over IPv4 and this is especially a problem if you have your mailbox and CAS on the same server (the normal default configuration).  Let me start from the beginning though in describing how the bug can be replicated, diagnosed, and then fixed.

Replication:

Normally, if you wanted to start using Outlook Anywhere on an Exchange 2007/Windows 2008 Server, the first command you would enter into a command prompt would be:

ServerManagerCmd -i RPC-over-HTTP-proxy

After this you would wait a few minutes while the server installs the RPC over HTTP proxy into IIS 7.  I generally restart the server at this point even though you don’t have to.

The most important part of this next step is to be patient (specifically, about 15 minutes).  Now you need to actually enable Outlook Anywhere using either the Exchange Management Console or the Exchange Management Shell.  I prefer the shell and it is easier to show on the blog so this is approximately what the command should look like:

[PS] C:\>Enable-OutlookAnywhere -Server host.domain.tld -DefaultAuthenticationMethod:Basic -SSLOffloading:$false

Now you have to wait about 15 minutes for the server to register an Event ID 3006 in the Application log:

Log Name:      Application
Source:        MSExchange RPC Over HTTP Autoconfig
Date:          3/25/2008 1:26:55 AM
Event ID:      3006
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      host.domain.tld
Description:
The Outlook Anywhere feature has been enabled. The ValidPorts registry setting has been modified to reflect this change.
New value:     HOST:6001-6002; HOST:6004;host.domain.tld:6001-6002; host.domain.tld:6004

Now set up an Outlook 2007 client and connect it to the mailbox using the correct settings for Outlook Anywhere access (Autodiscover should take care of this for you if you have it set up properly).  Then at this point everything should be working, right? WRONG! Don’t make the same mistake I did and keep trying to fix something that just can’t be fixed (unless you work and Microsoft and if you do please contact me via the contact page so we can work out a hotfix together).  You can now go to your Outlook icon in the system tray and ctrl+click on it to bring up the “Connection Status” window.  In it you will notice that things aren’t connecting exactly as they should (YMMV from the picture below since I took this after-the-fact just trying to reproduce what you may see):

Directory Disconnect

Diagnosis:

This is the part that drove me crazy and I honestly couldn’t have diagnosed it on my own if it weren’t for some pointers on the Internet which I want to cite here and here.  I’d suggest you read those two links for starters since they are where I learned about the problem from, but to be honest, the reason why it took me so long to find these posts was because I was beyond baffled and was originally looking down the completely wrong paths for a solution.  I could go on and on explaining all of the things that I thought were leading to the problem, but it would be a waste of time since the bug is so obvious now.

The problem we are experiencing here is that the RPC over HTTP proxy isn’t able to communicate over port 6004 with the localhost because there is a bug that is causing the Windows Server 2008 to not listen for connections on port 6004 via IPv6.  This can be confirmed by pulling up a command prompt and typing:

netstat -a -n

The netstat command will return a bunch of source/destination IP addresses and ports, but what is really important to us is the ports relevant to the RPC over HTTP proxy which will be these parts of the output as seen below:

TCP 0.0.0.0:6001 0.0.0.0:0 LISTENING
TCP 0.0.0.0:6002 0.0.0.0:0 LISTENING
TCP 0.0.0.0:6004 0.0.0.0:0 LISTENING
TCP [::]:6001 [::]:0 LISTENING
TCP [::]:6002 [::]:0 LISTENING

As we can see, the server is for some reason not listening on port 6004 via the IPv6 loopback.  This tells a couple of things, but most importantly, someone at Microsoft really screwed up by letting this one out the door without fixing it (especially since it was known about in the RC stage).  This also tells us that we can fix this problem by disabling IPv6 entirely.

You can confirm that the server isn’t listening on port 6004 by telnet’ing to localhost 6004 via (FYI, the telnet client/server are not default features in Windows 2008):

telnet localhost 6004

Fix:

IPv6 is disabled the same way in Windows Server 2008 as it is in Windows Vista, but just for good measure, I recommend that you also uncheck IPv6 TCP/IP on your NIC through the “Manage Network Connections” control panel. But to truly disable IPv6 you need to open regedit and navigate to:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters

Then you will need to add a 32-bit D-WORD with the name DisabledComponents and give it a value of 0xff. This will disable IPv6 on all interfaces and all tunneling interfaces but unfortunately it still doesn’t disable the loopback interface. In order to disable the loopback interface you will need to comment out the following line in your hosts file under %SYSTEMROOT%\System32\drivers\etc\:

::1 localhost

…by changing it to:

# ::1 localhost

…and while you’re at it you may as well add a couple more lines to directly map your HOSTNAME and FQDN to your IPv4 address of the Exchange server.  In the end your hosts file should look something like this:

10.0.0.10 host.domain.tld
10.0.0.10 HOST
127.0.0.1 localhost
# ::1 localhost

I would now recommend rebooting your server so that the registry changes take effect.  Once your server has rebooted you should now be able to run ipconfig without seeing all of the extra IPv6 tunneling interfaces; the only thing that should be visible is the IPv4 network interface. You should also now be able to successfully issue a:

telnet localhost 6004

The final and most important confirmation that this all worked will be to log on to a client workstation again and open up the connection status in Outlook 2007 to make sure that both the Directory and Mail are connected via RPC over HTTPS.

Side Notes:

I have been unsuccessful at setting up NTLM passthrough authentication in Outlook Anywhere on Windows Server 2008. For some reason NTLM continually causes Test-OutlookWebServices to fail the RPC test, but when I Set-OutlookAnywhere to -DefaultAuthentication:Basic I don’t have any problems other than that users complain about having to enter their password every time Outlook opens. If anyone has any advice on this topic, please comment.

Now get off the caffeine and get some sleep.

Be Sociable, Share!