A file move on a remote share is exactly the same as a local one: Explorer simply instructs the server to move the files/folders into their new destination. It can also be thought of as a full path rename. There's no need to copy the files to your local system and back out.
One of the biggest causes of any slowdown is something on your local system holding the file open: Virus scanners, shell extensions (e.g. compression utilities checking .exe files for SFX stubs, image/movie thumbnail generation, getting title/author information from Word Documents, scanning music/movies for tags, etc.), and so on. Explorer now has to wait for everything to close their open handles on the file before it can finish the file operation, and since many of the things I mentioned involve random small block I/O, SMB latency turns into a much bigger problem than it is on local disks. It's even worse if you're doing this over a wireless connection.
For an example of how fast it can be, try doing the move from a Command Prompt without Explorer open in your source folder. The operation will be very quick.