I go into other people's mailboxes all the times as part of my job, so I do end up looking at other people's calendars quite a bit. One thing I've noticed is that the lag between opening up a remote calendar is about the same lag as opening up another mailbox. In our case it's 5-10 seconds when it gets bad, 2-3 more typically, when running in direct mode. It seems to be related to load on the mailstore servers. While we're on Exch2007 now, it did the same back in Exch2003.
One thing it did do if it took a long time was the little Outlook popup, "Trying to connect to [mailbox-server]..." Are you seeing that too? If so, the initial connection is taking too long. If it isn't, then the connection went reasonably fast but getting actual data is taking too long. That would suggest some performance tuning is needed on the mail-stores.