The thing to remember is that the delta algorithm is most useful when the rsync server is running on the remote system and the client is on the local system.
When decides it needs to transfer a large file because the stat() output is different (size, modify dates), then some rsync process will need to read the source and destination files. If you have the remote files mounted, then the process running on your local system will read the entire file in over the network. If you were connecting to a rsync daemon on the remote system, then the remote rsync process will read the files, and the algorithm will be employed to decide which blocks need to be updated, and only the differences will cross the network.
As for why it is slow, there could be many reasons. If your selection of files was basically a huge number of small files, then the amount of time you needed to transfer may be irrelevant, and you are waiting for stat() to complete.
If you are running SMB over a wan connection, you could be running into the standard SMB sucks over high-latency-links issue.
You might want to look at setting up an SSH server on your Windows box that can run rsync. It is no longer free, but I generally use CopSSH/cwRsync.